2024-03-04T17:19:41.391 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-04T17:19:41.391 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi/7581342 --verbose --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} --name suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi --owner scheduled_suriarte@teuthology -- /home/teuthworker/archive/suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi/7581342/orig.config.yaml 2024-03-04T17:19:41.425 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T17:19:41.496 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi/7581342 branch: wip-rgw-d4n description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} email: null first_in_suite: false job_id: '7581342' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-rgw-d4n ceph: conf: client: debug rgw: 20 debug rgw lifecycle: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 3aec390336fdfdc8f927818b718cfb0361bd47c8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 3aec390336fdfdc8f927818b718cfb0361bd47c8 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 rgw-multisite-tests: args: - tests.py workunit: branch: wip-rgw-d4n sha1: 3aec390336fdfdc8f927818b718cfb0361bd47c8 owner: scheduled_suriarte@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 3aec390336fdfdc8f927818b718cfb0361bd47c8 sleep_before_teardown: 0 suite: rgw suite_branch: wip-rgw-d4n suite_path: /home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa suite_relpath: qa suite_repo: https://github.com/pritha-srivastava/ceph.git suite_sha1: 3aec390336fdfdc8f927818b718cfb0361bd47c8 targets: smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAqHjmXINRGS++vEJkLSuQVoCDLD7Q/OTh0RAS4ORRa34gLI/XifPQGnY9NX/bUvqJ8L9VxxHFw5XRlWZkskD6M= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM5+Lf8d9jkIUZDTAHuoCGKmJtNNK7KduqKFetIFOBuUOXs7F9wANcTWVBomJoQHbfTA4oYoHd9PFqtZins8FCs= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-03-04_17:13:34 tube: smithi user: suriarte verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-04T17:19:41.497 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa; will attempt to use it 2024-03-04T17:19:41.497 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks 2024-03-04T17:19:41.497 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-04T17:19:41.498 INFO:teuthology.task.internal:Checking packages... 2024-03-04T17:19:41.519 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '3aec390336fdfdc8f927818b718cfb0361bd47c8' 2024-03-04T17:19:41.519 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-04T17:19:41.519 INFO:teuthology.packaging:ref: None 2024-03-04T17:19:41.519 INFO:teuthology.packaging:tag: None 2024-03-04T17:19:41.519 INFO:teuthology.packaging:branch: wip-rgw-d4n 2024-03-04T17:19:41.519 INFO:teuthology.packaging:sha1: 3aec390336fdfdc8f927818b718cfb0361bd47c8 2024-03-04T17:19:41.519 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-rgw-d4n 2024-03-04T17:19:41.670 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1871-g3aec3903-1jammy 2024-03-04T17:19:41.670 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-04T17:19:41.677 INFO:teuthology.task.internal:no buildpackages task found 2024-03-04T17:19:41.677 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-04T17:19:41.714 INFO:teuthology.task.internal:Saving configuration 2024-03-04T17:19:41.726 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-04T17:19:41.744 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T17:19:41.768 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi/7581342', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 17:13:56.099963', 'locked_by': 'scheduled_suriarte@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINNeXKoFM6Zq/IeTiztAnmFhc9zCXrbEGE1iUNb3lzfp'} 2024-03-04T17:19:41.791 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi/7581342', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 17:13:56.101740', 'locked_by': 'scheduled_suriarte@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL/7FHy3EAT3L0TP67UpOdRlQrb2qfsnZj/i5NjkyJOB'} 2024-03-04T17:19:41.791 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-04T17:19:41.797 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2024-03-04T17:19:41.797 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2024-03-04T17:19:41.797 INFO:teuthology.run_tasks:Running task console_log... 2024-03-04T17:19:41.883 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6f82191dc0>, signals=[15]) 2024-03-04T17:19:41.883 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-04T17:19:41.889 INFO:teuthology.task.internal:Opening connections... 2024-03-04T17:19:41.889 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-03-04T17:19:41.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T17:19:41.968 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-03-04T17:19:41.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T17:19:42.045 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-04T17:19:42.051 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-03-04T17:19:42.056 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-03-04T17:19:42.056 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-03-04T17:19:42.101 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-04T17:19:42.101 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="Ubuntu" 2024-03-04T17:19:42.101 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="22.04" 2024-03-04T17:19:42.101 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-04T17:19:42.101 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_CODENAME=jammy 2024-03-04T17:19:42.101 INFO:teuthology.orchestra.run.smithi136.stdout:ID=ubuntu 2024-03-04T17:19:42.101 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE=debian 2024-03-04T17:19:42.101 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-04T17:19:42.101 INFO:teuthology.orchestra.run.smithi136.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-04T17:19:42.101 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-04T17:19:42.101 INFO:teuthology.orchestra.run.smithi136.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-04T17:19:42.101 INFO:teuthology.orchestra.run.smithi136.stdout:UBUNTU_CODENAME=jammy 2024-03-04T17:19:42.102 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-03-04T17:19:42.126 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-03-04T17:19:42.131 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-03-04T17:19:42.131 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-03-04T17:19:42.178 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-04T17:19:42.178 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="Ubuntu" 2024-03-04T17:19:42.178 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="22.04" 2024-03-04T17:19:42.178 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-04T17:19:42.178 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_CODENAME=jammy 2024-03-04T17:19:42.178 INFO:teuthology.orchestra.run.smithi153.stdout:ID=ubuntu 2024-03-04T17:19:42.178 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE=debian 2024-03-04T17:19:42.178 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-04T17:19:42.179 INFO:teuthology.orchestra.run.smithi153.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-04T17:19:42.179 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-04T17:19:42.179 INFO:teuthology.orchestra.run.smithi153.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-04T17:19:42.179 INFO:teuthology.orchestra.run.smithi153.stdout:UBUNTU_CODENAME=jammy 2024-03-04T17:19:42.179 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-03-04T17:19:42.200 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-04T17:19:42.212 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-04T17:19:42.238 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-04T17:19:42.239 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-03-04T17:19:42.241 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-03-04T17:19:42.246 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-04T17:19:42.263 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-04T17:19:42.263 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-03-04T17:19:42.287 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-03-04T17:19:42.296 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-04T17:19:42.361 INFO:teuthology.run_tasks:Running task kernel... 2024-03-04T17:19:42.375 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-04T17:19:42.375 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-04T17:19:42.375 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T17:19:42.375 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T17:19:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T17:19:42.380 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-03-04T17:19:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T17:19:42.381 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-04T17:19:42.426 INFO:teuthology.orchestra.run.smithi153.stdout:5.15.0-56-generic 2024-03-04T17:19:42.427 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.15.0-56-generic 2024-03-04T17:19:42.427 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get clean 2024-03-04T17:19:42.429 INFO:teuthology.orchestra.run.smithi136.stdout:5.15.0-56-generic 2024-03-04T17:19:42.430 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.15.0-56-generic 2024-03-04T17:19:42.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get clean 2024-03-04T17:19:42.558 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2024-03-04T17:19:42.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2024-03-04T17:19:42.701 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-04T17:19:42.710 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-04T17:19:42.710 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-04T17:19:42.793 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-04T17:19:42.860 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-04T17:19:42.937 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,205 kB] 2024-03-04T17:19:42.957 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-04T17:19:43.010 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [579 kB] 2024-03-04T17:19:43.050 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [413 kB] 2024-03-04T17:19:43.062 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [219 kB] 2024-03-04T17:19:43.064 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-04T17:19:43.065 INFO:teuthology.orchestra.run.smithi153.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-04T17:19:43.065 INFO:teuthology.orchestra.run.smithi153.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,476 kB] 2024-03-04T17:19:43.101 INFO:teuthology.orchestra.run.smithi153.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [244 kB] 2024-03-04T17:19:43.106 INFO:teuthology.orchestra.run.smithi153.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-04T17:19:43.106 INFO:teuthology.orchestra.run.smithi153.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [846 kB] 2024-03-04T17:19:43.121 INFO:teuthology.orchestra.run.smithi153.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,422 kB] 2024-03-04T17:19:43.131 INFO:teuthology.orchestra.run.smithi153.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [596 kB] 2024-03-04T17:19:43.143 INFO:teuthology.orchestra.run.smithi153.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [161 kB] 2024-03-04T17:19:43.145 INFO:teuthology.orchestra.run.smithi153.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-04T17:19:43.165 INFO:teuthology.orchestra.run.smithi153.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-04T17:19:43.166 INFO:teuthology.orchestra.run.smithi153.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-04T17:19:43.166 INFO:teuthology.orchestra.run.smithi153.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-04T17:19:43.166 INFO:teuthology.orchestra.run.smithi153.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-04T17:19:43.197 INFO:teuthology.orchestra.run.smithi153.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [279 kB] 2024-03-04T17:19:43.207 INFO:teuthology.orchestra.run.smithi153.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-04T17:19:43.207 INFO:teuthology.orchestra.run.smithi153.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-04T17:19:43.209 INFO:teuthology.orchestra.run.smithi153.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,504 kB] 2024-03-04T17:19:43.249 INFO:teuthology.orchestra.run.smithi153.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [247 kB] 2024-03-04T17:19:43.254 INFO:teuthology.orchestra.run.smithi153.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-04T17:19:43.254 INFO:teuthology.orchestra.run.smithi153.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,052 kB] 2024-03-04T17:19:43.280 INFO:teuthology.orchestra.run.smithi153.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [691 kB] 2024-03-04T17:19:43.292 INFO:teuthology.orchestra.run.smithi153.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [237 kB] 2024-03-04T17:19:43.296 INFO:teuthology.orchestra.run.smithi153.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-04T17:19:43.296 INFO:teuthology.orchestra.run.smithi153.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-04T17:19:43.296 INFO:teuthology.orchestra.run.smithi153.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-04T17:19:43.297 INFO:teuthology.orchestra.run.smithi153.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-04T17:19:43.299 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-04T17:19:43.312 INFO:teuthology.orchestra.run.smithi153.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-04T17:19:43.312 INFO:teuthology.orchestra.run.smithi153.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-03-04T17:19:43.313 INFO:teuthology.orchestra.run.smithi153.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-03-04T17:19:43.322 INFO:teuthology.orchestra.run.smithi153.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-03-04T17:19:43.323 INFO:teuthology.orchestra.run.smithi153.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-04T17:19:43.323 INFO:teuthology.orchestra.run.smithi153.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2024-03-04T17:19:43.324 INFO:teuthology.orchestra.run.smithi153.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-03-04T17:19:43.324 INFO:teuthology.orchestra.run.smithi153.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-03-04T17:19:43.336 INFO:teuthology.orchestra.run.smithi153.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-04T17:19:43.342 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-04T17:19:43.432 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,422 kB] 2024-03-04T17:19:43.739 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [579 kB] 2024-03-04T17:19:43.768 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [279 kB] 2024-03-04T17:19:43.777 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,205 kB] 2024-03-04T17:19:43.784 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-04T17:19:43.784 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-04T17:19:43.786 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,504 kB] 2024-03-04T17:19:43.844 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [247 kB] 2024-03-04T17:19:43.851 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-04T17:19:44.015 INFO:teuthology.orchestra.run.smithi136.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,052 kB] 2024-03-04T17:19:44.172 INFO:teuthology.orchestra.run.smithi136.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [413 kB] 2024-03-04T17:19:44.191 INFO:teuthology.orchestra.run.smithi136.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [219 kB] 2024-03-04T17:19:44.201 INFO:teuthology.orchestra.run.smithi136.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-04T17:19:44.201 INFO:teuthology.orchestra.run.smithi136.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-04T17:19:44.203 INFO:teuthology.orchestra.run.smithi136.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,476 kB] 2024-03-04T17:19:44.294 INFO:teuthology.orchestra.run.smithi136.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [244 kB] 2024-03-04T17:19:44.304 INFO:teuthology.orchestra.run.smithi136.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-04T17:19:44.305 INFO:teuthology.orchestra.run.smithi136.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [596 kB] 2024-03-04T17:19:44.328 INFO:teuthology.orchestra.run.smithi136.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [846 kB] 2024-03-04T17:19:44.514 INFO:teuthology.orchestra.run.smithi136.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [161 kB] 2024-03-04T17:19:44.640 INFO:teuthology.orchestra.run.smithi136.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [691 kB] 2024-03-04T17:19:44.678 INFO:teuthology.orchestra.run.smithi136.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [237 kB] 2024-03-04T17:19:44.691 INFO:teuthology.orchestra.run.smithi136.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-04T17:19:44.692 INFO:teuthology.orchestra.run.smithi136.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-04T17:19:44.694 INFO:teuthology.orchestra.run.smithi136.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-04T17:19:44.694 INFO:teuthology.orchestra.run.smithi136.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-04T17:19:44.695 INFO:teuthology.orchestra.run.smithi136.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-04T17:19:44.695 INFO:teuthology.orchestra.run.smithi136.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-03-04T17:19:44.697 INFO:teuthology.orchestra.run.smithi136.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-03-04T17:19:44.722 INFO:teuthology.orchestra.run.smithi136.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-03-04T17:19:44.886 INFO:teuthology.orchestra.run.smithi136.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-04T17:19:44.910 INFO:teuthology.orchestra.run.smithi136.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-04T17:19:44.916 INFO:teuthology.orchestra.run.smithi136.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-04T17:19:44.929 INFO:teuthology.orchestra.run.smithi136.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-04T17:19:44.931 INFO:teuthology.orchestra.run.smithi136.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-04T17:19:44.933 INFO:teuthology.orchestra.run.smithi136.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-04T17:19:44.969 INFO:teuthology.orchestra.run.smithi136.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2024-03-04T17:19:45.065 INFO:teuthology.orchestra.run.smithi136.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-03-04T17:19:45.093 INFO:teuthology.orchestra.run.smithi136.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-03-04T17:19:45.145 INFO:teuthology.orchestra.run.smithi136.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-04T17:19:48.443 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 11.9 MB in 2s (7,015 kB/s) 2024-03-04T17:19:49.520 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 11.9 MB in 2s (4,766 kB/s) 2024-03-04T17:19:49.831 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-03-04T17:19:49.849 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-04T17:19:49.903 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-03-04T17:19:50.094 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2024-03-04T17:19:50.095 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2024-03-04T17:19:50.253 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2024-03-04T17:19:50.253 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-04T17:19:50.253 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-04T17:19:50.253 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-04T17:19:50.254 INFO:teuthology.orchestra.run.smithi153.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-04T17:19:50.254 INFO:teuthology.orchestra.run.smithi153.stdout: python2.7-minimal 2024-03-04T17:19:50.254 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-04T17:19:50.255 INFO:teuthology.orchestra.run.smithi153.stdout:The following additional packages will be installed: 2024-03-04T17:19:50.255 INFO:teuthology.orchestra.run.smithi153.stdout: linux-generic linux-headers-5.15.0-97 linux-headers-5.15.0-97-generic 2024-03-04T17:19:50.255 INFO:teuthology.orchestra.run.smithi153.stdout: linux-headers-generic linux-image-5.15.0-97-generic 2024-03-04T17:19:50.255 INFO:teuthology.orchestra.run.smithi153.stdout: linux-modules-5.15.0-97-generic linux-modules-extra-5.15.0-97-generic 2024-03-04T17:19:50.256 INFO:teuthology.orchestra.run.smithi153.stdout:Suggested packages: 2024-03-04T17:19:50.256 INFO:teuthology.orchestra.run.smithi153.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-04T17:19:50.256 INFO:teuthology.orchestra.run.smithi153.stdout:Recommended packages: 2024-03-04T17:19:50.256 INFO:teuthology.orchestra.run.smithi153.stdout: thermald 2024-03-04T17:19:50.301 INFO:teuthology.orchestra.run.smithi153.stdout:The following NEW packages will be installed: 2024-03-04T17:19:50.302 INFO:teuthology.orchestra.run.smithi153.stdout: linux-headers-5.15.0-97 linux-headers-5.15.0-97-generic 2024-03-04T17:19:50.302 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.15.0-97-generic linux-modules-5.15.0-97-generic 2024-03-04T17:19:50.302 INFO:teuthology.orchestra.run.smithi153.stdout: linux-modules-extra-5.15.0-97-generic 2024-03-04T17:19:50.303 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages will be upgraded: 2024-03-04T17:19:50.303 INFO:teuthology.orchestra.run.smithi153.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-04T17:19:50.482 INFO:teuthology.orchestra.run.smithi153.stdout:3 upgraded, 5 newly installed, 0 to remove and 288 not upgraded. 2024-03-04T17:19:50.482 INFO:teuthology.orchestra.run.smithi153.stdout:Need to get 113 MB of archives. 2024-03-04T17:19:50.482 INFO:teuthology.orchestra.run.smithi153.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-04T17:19:50.482 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-97-generic amd64 5.15.0-97.107 [22.6 MB] 2024-03-04T17:19:50.925 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-03-04T17:19:50.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-04T17:19:50.994 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-03-04T17:19:51.167 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2024-03-04T17:19:51.168 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2024-03-04T17:19:51.317 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2024-03-04T17:19:51.317 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-04T17:19:51.317 INFO:teuthology.orchestra.run.smithi136.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-04T17:19:51.318 INFO:teuthology.orchestra.run.smithi136.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-04T17:19:51.318 INFO:teuthology.orchestra.run.smithi136.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-04T17:19:51.318 INFO:teuthology.orchestra.run.smithi136.stdout: python2.7-minimal 2024-03-04T17:19:51.318 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-04T17:19:51.319 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2024-03-04T17:19:51.319 INFO:teuthology.orchestra.run.smithi136.stdout: linux-generic linux-headers-5.15.0-97 linux-headers-5.15.0-97-generic 2024-03-04T17:19:51.319 INFO:teuthology.orchestra.run.smithi136.stdout: linux-headers-generic linux-image-5.15.0-97-generic 2024-03-04T17:19:51.320 INFO:teuthology.orchestra.run.smithi136.stdout: linux-modules-5.15.0-97-generic linux-modules-extra-5.15.0-97-generic 2024-03-04T17:19:51.321 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2024-03-04T17:19:51.321 INFO:teuthology.orchestra.run.smithi136.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-04T17:19:51.321 INFO:teuthology.orchestra.run.smithi136.stdout:Recommended packages: 2024-03-04T17:19:51.321 INFO:teuthology.orchestra.run.smithi136.stdout: thermald 2024-03-04T17:19:51.373 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2024-03-04T17:19:51.374 INFO:teuthology.orchestra.run.smithi136.stdout: linux-headers-5.15.0-97 linux-headers-5.15.0-97-generic 2024-03-04T17:19:51.374 INFO:teuthology.orchestra.run.smithi136.stdout: linux-image-5.15.0-97-generic linux-modules-5.15.0-97-generic 2024-03-04T17:19:51.374 INFO:teuthology.orchestra.run.smithi136.stdout: linux-modules-extra-5.15.0-97-generic 2024-03-04T17:19:51.375 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages will be upgraded: 2024-03-04T17:19:51.375 INFO:teuthology.orchestra.run.smithi136.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-04T17:19:51.553 INFO:teuthology.orchestra.run.smithi136.stdout:3 upgraded, 5 newly installed, 0 to remove and 288 not upgraded. 2024-03-04T17:19:51.553 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 113 MB of archives. 2024-03-04T17:19:51.553 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-04T17:19:51.553 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-97-generic amd64 5.15.0-97.107 [22.6 MB] 2024-03-04T17:19:51.891 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-97-generic amd64 5.15.0-97.107 [11.5 MB] 2024-03-04T17:19:52.249 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-97-generic amd64 5.15.0-97.107 [63.8 MB] 2024-03-04T17:19:52.947 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-97-generic amd64 5.15.0-97.107 [11.5 MB] 2024-03-04T17:19:53.296 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-97-generic amd64 5.15.0-97.107 [63.8 MB] 2024-03-04T17:19:54.481 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.97.92 [1,696 B] 2024-03-04T17:19:54.481 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.97.92 [2,494 B] 2024-03-04T17:19:54.482 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-97 all 5.15.0-97.107 [12.3 MB] 2024-03-04T17:19:55.544 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-97-generic amd64 5.15.0-97.107 [2,892 kB] 2024-03-04T17:19:55.642 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.97.92 [1,696 B] 2024-03-04T17:19:55.642 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.97.92 [2,494 B] 2024-03-04T17:19:55.643 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-97 all 5.15.0-97.107 [12.3 MB] 2024-03-04T17:19:55.684 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.97.92 [2,342 B] 2024-03-04T17:19:55.998 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-97-generic amd64 5.15.0-97.107 [2,892 kB] 2024-03-04T17:19:56.078 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 113 MB in 5s (21.0 MB/s) 2024-03-04T17:19:56.082 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.97.92 [2,342 B] 2024-03-04T17:19:56.263 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-modules-5.15.0-97-generic. 2024-03-04T17:19:56.484 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 113 MB in 5s (24.0 MB/s) 2024-03-04T17:19:56.653 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-modules-5.15.0-97-generic. 2024-03-04T17:19:58.269 INFO:teuthology.orchestra.run.smithi153.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-04T17:19:58.274 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../0-linux-modules-5.15.0-97-generic_5.15.0-97.107_amd64.deb ... 2024-03-04T17:19:58.369 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-modules-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:19:58.637 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-04T17:19:58.641 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../0-linux-modules-5.15.0-97-generic_5.15.0-97.107_amd64.deb ... 2024-03-04T17:19:58.760 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-modules-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:00.762 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-image-5.15.0-97-generic. 2024-03-04T17:20:00.784 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../1-linux-image-5.15.0-97-generic_5.15.0-97.107_amd64.deb ... 2024-03-04T17:20:00.869 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-image-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:01.315 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-97-generic. 2024-03-04T17:20:01.335 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-97-generic_5.15.0-97.107_amd64.deb ... 2024-03-04T17:20:01.373 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-modules-extra-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:01.512 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-image-5.15.0-97-generic. 2024-03-04T17:20:01.533 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../1-linux-image-5.15.0-97-generic_5.15.0-97.107_amd64.deb ... 2024-03-04T17:20:01.622 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-image-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:02.091 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-97-generic. 2024-03-04T17:20:02.111 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-97-generic_5.15.0-97.107_amd64.deb ... 2024-03-04T17:20:02.157 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-modules-extra-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:07.731 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../3-linux-generic_5.15.0.97.92_amd64.deb ... 2024-03-04T17:20:07.833 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-generic (5.15.0.97.92) over (5.15.0.56.54) ... 2024-03-04T17:20:08.184 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.97.92_amd64.deb ... 2024-03-04T17:20:08.305 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-image-generic (5.15.0.97.92) over (5.15.0.56.54) ... 2024-03-04T17:20:08.625 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-headers-5.15.0-97. 2024-03-04T17:20:08.636 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../5-linux-headers-5.15.0-97_5.15.0-97.107_all.deb ... 2024-03-04T17:20:08.675 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-5.15.0-97 (5.15.0-97.107) ... 2024-03-04T17:20:08.695 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../3-linux-generic_5.15.0.97.92_amd64.deb ... 2024-03-04T17:20:08.846 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-generic (5.15.0.97.92) over (5.15.0.56.54) ... 2024-03-04T17:20:09.205 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.97.92_amd64.deb ... 2024-03-04T17:20:09.316 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-image-generic (5.15.0.97.92) over (5.15.0.56.54) ... 2024-03-04T17:20:09.586 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-headers-5.15.0-97. 2024-03-04T17:20:09.609 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../5-linux-headers-5.15.0-97_5.15.0-97.107_all.deb ... 2024-03-04T17:20:09.644 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-headers-5.15.0-97 (5.15.0-97.107) ... 2024-03-04T17:20:14.133 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-headers-5.15.0-97-generic. 2024-03-04T17:20:14.170 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../6-linux-headers-5.15.0-97-generic_5.15.0-97.107_amd64.deb ... 2024-03-04T17:20:14.208 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:16.143 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-headers-5.15.0-97-generic. 2024-03-04T17:20:16.179 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../6-linux-headers-5.15.0-97-generic_5.15.0-97.107_amd64.deb ... 2024-03-04T17:20:16.218 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-headers-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:17.328 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.97.92_amd64.deb ... 2024-03-04T17:20:17.459 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-generic (5.15.0.97.92) over (5.15.0.56.54) ... 2024-03-04T17:20:17.826 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-5.15.0-97 (5.15.0-97.107) ... 2024-03-04T17:20:17.947 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:18.188 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-generic (5.15.0.97.92) ... 2024-03-04T17:20:18.861 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-image-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:19.726 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.97.92_amd64.deb ... 2024-03-04T17:20:19.848 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-headers-generic (5.15.0.97.92) over (5.15.0.56.54) ... 2024-03-04T17:20:20.171 INFO:teuthology.orchestra.run.smithi153.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-04T17:20:20.171 INFO:teuthology.orchestra.run.smithi153.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-04T17:20:20.171 INFO:teuthology.orchestra.run.smithi153.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-97-generic 2024-03-04T17:20:20.171 INFO:teuthology.orchestra.run.smithi153.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-97-generic 2024-03-04T17:20:20.198 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-headers-5.15.0-97 (5.15.0-97.107) ... 2024-03-04T17:20:20.345 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-headers-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:20.347 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-modules-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:20.471 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-headers-generic (5.15.0.97.92) ... 2024-03-04T17:20:20.616 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-image-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:21.732 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-modules-extra-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:21.935 INFO:teuthology.orchestra.run.smithi136.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-04T17:20:21.935 INFO:teuthology.orchestra.run.smithi136.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-04T17:20:21.935 INFO:teuthology.orchestra.run.smithi136.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-97-generic 2024-03-04T17:20:21.935 INFO:teuthology.orchestra.run.smithi136.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-97-generic 2024-03-04T17:20:22.094 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-modules-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:23.150 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-image-generic (5.15.0.97.92) ... 2024-03-04T17:20:23.301 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-generic (5.15.0.97.92) ... 2024-03-04T17:20:23.427 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for linux-image-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:23.494 INFO:teuthology.orchestra.run.smithi153.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-04T17:20:23.494 INFO:teuthology.orchestra.run.smithi153.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-97-generic 2024-03-04T17:20:23.504 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-modules-extra-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:25.399 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-image-generic (5.15.0.97.92) ... 2024-03-04T17:20:25.525 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-generic (5.15.0.97.92) ... 2024-03-04T17:20:25.651 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for linux-image-5.15.0-97-generic (5.15.0-97.107) ... 2024-03-04T17:20:25.714 INFO:teuthology.orchestra.run.smithi136.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-04T17:20:25.715 INFO:teuthology.orchestra.run.smithi136.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-97-generic 2024-03-04T17:20:37.244 INFO:teuthology.orchestra.run.smithi153.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-04T17:20:37.244 INFO:teuthology.orchestra.run.smithi153.stdout:Sourcing file `/etc/default/grub' 2024-03-04T17:20:37.270 INFO:teuthology.orchestra.run.smithi153.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-04T17:20:37.297 INFO:teuthology.orchestra.run.smithi153.stdout:Generating grub configuration file ... 2024-03-04T17:20:37.548 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.15.0-97-generic 2024-03-04T17:20:37.583 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.15.0-97-generic 2024-03-04T17:20:37.874 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-04T17:20:37.880 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-04T17:20:37.961 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-04T17:20:37.967 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-04T17:20:38.123 INFO:teuthology.orchestra.run.smithi153.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-04T17:20:38.123 INFO:teuthology.orchestra.run.smithi153.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-04T17:20:38.123 INFO:teuthology.orchestra.run.smithi153.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-04T17:20:38.177 INFO:teuthology.orchestra.run.smithi153.stdout:done 2024-03-04T17:20:38.660 DEBUG:teuthology.orchestra.run.smithi153:> dpkg -s linux-image-generic 2024-03-04T17:20:38.676 INFO:teuthology.orchestra.run.smithi153.stdout:Package: linux-image-generic 2024-03-04T17:20:38.677 INFO:teuthology.orchestra.run.smithi153.stdout:Status: install ok installed 2024-03-04T17:20:38.677 INFO:teuthology.orchestra.run.smithi153.stdout:Priority: optional 2024-03-04T17:20:38.677 INFO:teuthology.orchestra.run.smithi153.stdout:Section: kernel 2024-03-04T17:20:38.677 INFO:teuthology.orchestra.run.smithi153.stdout:Installed-Size: 21 2024-03-04T17:20:38.677 INFO:teuthology.orchestra.run.smithi153.stdout:Maintainer: Ubuntu Kernel Team 2024-03-04T17:20:38.677 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: amd64 2024-03-04T17:20:38.677 INFO:teuthology.orchestra.run.smithi153.stdout:Source: linux-meta 2024-03-04T17:20:38.677 INFO:teuthology.orchestra.run.smithi153.stdout:Version: 5.15.0.97.92 2024-03-04T17:20:38.677 INFO:teuthology.orchestra.run.smithi153.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-97), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-04T17:20:38.677 INFO:teuthology.orchestra.run.smithi153.stdout:Depends: linux-image-5.15.0-97-generic, linux-modules-extra-5.15.0-97-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-04T17:20:38.677 INFO:teuthology.orchestra.run.smithi153.stdout:Recommends: thermald 2024-03-04T17:20:38.677 INFO:teuthology.orchestra.run.smithi153.stdout:Description: Generic Linux kernel image 2024-03-04T17:20:38.677 INFO:teuthology.orchestra.run.smithi153.stdout: This package will always depend on the latest generic kernel image 2024-03-04T17:20:38.678 INFO:teuthology.orchestra.run.smithi153.stdout: available. 2024-03-04T17:20:38.678 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-97-generic 2024-03-04T17:20:38.678 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T17:20:38.678 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T17:20:38.678 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2024-03-04T17:20:38.678 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T17:20:38.678 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get clean 2024-03-04T17:20:38.852 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2024-03-04T17:20:39.110 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-04T17:20:39.110 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-04T17:20:39.180 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-04T17:20:39.262 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-04T17:20:39.809 INFO:teuthology.orchestra.run.smithi136.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-04T17:20:39.809 INFO:teuthology.orchestra.run.smithi136.stdout:Sourcing file `/etc/default/grub' 2024-03-04T17:20:39.827 INFO:teuthology.orchestra.run.smithi136.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-04T17:20:39.849 INFO:teuthology.orchestra.run.smithi136.stdout:Generating grub configuration file ... 2024-03-04T17:20:40.162 INFO:teuthology.orchestra.run.smithi136.stdout:Found linux image: /boot/vmlinuz-5.15.0-97-generic 2024-03-04T17:20:40.181 INFO:teuthology.orchestra.run.smithi136.stdout:Found initrd image: /boot/initrd.img-5.15.0-97-generic 2024-03-04T17:20:40.469 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-03-04T17:20:40.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-04T17:20:40.502 INFO:teuthology.orchestra.run.smithi136.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-04T17:20:40.510 INFO:teuthology.orchestra.run.smithi136.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-04T17:20:40.548 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-03-04T17:20:40.576 INFO:teuthology.orchestra.run.smithi136.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-04T17:20:40.582 INFO:teuthology.orchestra.run.smithi136.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-04T17:20:40.740 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2024-03-04T17:20:40.740 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2024-03-04T17:20:40.763 INFO:teuthology.orchestra.run.smithi136.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-04T17:20:40.763 INFO:teuthology.orchestra.run.smithi136.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-04T17:20:40.763 INFO:teuthology.orchestra.run.smithi136.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-04T17:20:40.811 INFO:teuthology.orchestra.run.smithi136.stdout:done 2024-03-04T17:20:40.910 INFO:teuthology.orchestra.run.smithi153.stdout:linux-image-generic is already the newest version (5.15.0.97.92). 2024-03-04T17:20:40.911 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2024-03-04T17:20:40.911 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-04T17:20:40.911 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-04T17:20:40.911 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-04T17:20:40.911 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-04T17:20:40.912 INFO:teuthology.orchestra.run.smithi153.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-04T17:20:40.912 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-04T17:20:40.953 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 0 newly installed, 0 to remove and 288 not upgraded. 2024-03-04T17:20:40.955 DEBUG:teuthology.orchestra.run.smithi153:> dpkg -s linux-image-generic 2024-03-04T17:20:40.967 INFO:teuthology.orchestra.run.smithi153.stdout:Package: linux-image-generic 2024-03-04T17:20:40.967 INFO:teuthology.orchestra.run.smithi153.stdout:Status: install ok installed 2024-03-04T17:20:40.967 INFO:teuthology.orchestra.run.smithi153.stdout:Priority: optional 2024-03-04T17:20:40.967 INFO:teuthology.orchestra.run.smithi153.stdout:Section: kernel 2024-03-04T17:20:40.967 INFO:teuthology.orchestra.run.smithi153.stdout:Installed-Size: 21 2024-03-04T17:20:40.967 INFO:teuthology.orchestra.run.smithi153.stdout:Maintainer: Ubuntu Kernel Team 2024-03-04T17:20:40.967 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: amd64 2024-03-04T17:20:40.968 INFO:teuthology.orchestra.run.smithi153.stdout:Source: linux-meta 2024-03-04T17:20:40.968 INFO:teuthology.orchestra.run.smithi153.stdout:Version: 5.15.0.97.92 2024-03-04T17:20:40.968 INFO:teuthology.orchestra.run.smithi153.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-97), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-04T17:20:40.968 INFO:teuthology.orchestra.run.smithi153.stdout:Depends: linux-image-5.15.0-97-generic, linux-modules-extra-5.15.0-97-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-04T17:20:40.968 INFO:teuthology.orchestra.run.smithi153.stdout:Recommends: thermald 2024-03-04T17:20:40.968 INFO:teuthology.orchestra.run.smithi153.stdout:Description: Generic Linux kernel image 2024-03-04T17:20:40.968 INFO:teuthology.orchestra.run.smithi153.stdout: This package will always depend on the latest generic kernel image 2024-03-04T17:20:40.968 INFO:teuthology.orchestra.run.smithi153.stdout: available. 2024-03-04T17:20:40.968 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-03-04T17:20:41.014 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.WBbJpNPLLf 2024-03-04T17:20:41.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub/grub.cfg /tmp/tmp.WBbJpNPLLf 2024-03-04T17:20:41.070 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.WBbJpNPLLf 2024-03-04T17:20:41.196 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.WBbJpNPLLf is 10KB 2024-03-04T17:20:41.209 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.WBbJpNPLLf 2024-03-04T17:20:41.214 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-04T17:20:41.270 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:20:41.270 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-04T17:20:41.271 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-04T17:20:41.301 DEBUG:teuthology.orchestra.run.smithi136:> dpkg -s linux-image-generic 2024-03-04T17:20:41.322 INFO:teuthology.orchestra.run.smithi136.stdout:Package: linux-image-generic 2024-03-04T17:20:41.322 INFO:teuthology.orchestra.run.smithi136.stdout:Status: install ok installed 2024-03-04T17:20:41.322 INFO:teuthology.orchestra.run.smithi136.stdout:Priority: optional 2024-03-04T17:20:41.322 INFO:teuthology.orchestra.run.smithi136.stdout:Section: kernel 2024-03-04T17:20:41.322 INFO:teuthology.orchestra.run.smithi136.stdout:Installed-Size: 21 2024-03-04T17:20:41.322 INFO:teuthology.orchestra.run.smithi136.stdout:Maintainer: Ubuntu Kernel Team 2024-03-04T17:20:41.322 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: amd64 2024-03-04T17:20:41.322 INFO:teuthology.orchestra.run.smithi136.stdout:Source: linux-meta 2024-03-04T17:20:41.322 INFO:teuthology.orchestra.run.smithi136.stdout:Version: 5.15.0.97.92 2024-03-04T17:20:41.322 INFO:teuthology.orchestra.run.smithi136.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-97), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-04T17:20:41.322 INFO:teuthology.orchestra.run.smithi136.stdout:Depends: linux-image-5.15.0-97-generic, linux-modules-extra-5.15.0-97-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-04T17:20:41.322 INFO:teuthology.orchestra.run.smithi136.stdout:Recommends: thermald 2024-03-04T17:20:41.323 INFO:teuthology.orchestra.run.smithi136.stdout:Description: Generic Linux kernel image 2024-03-04T17:20:41.323 INFO:teuthology.orchestra.run.smithi136.stdout: This package will always depend on the latest generic kernel image 2024-03-04T17:20:41.323 INFO:teuthology.orchestra.run.smithi136.stdout: available. 2024-03-04T17:20:41.323 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-97-generic 2024-03-04T17:20:41.323 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T17:20:41.323 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T17:20:41.323 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2024-03-04T17:20:41.323 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T17:20:41.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get clean 2024-03-04T17:20:41.336 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-97-generic 2024-03-04T17:20:41.336 DEBUG:teuthology.orchestra.run.smithi153:> sudo update-grub 2024-03-04T17:20:41.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2024-03-04T17:20:41.651 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-04T17:20:41.651 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-04T17:20:41.665 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-04T17:20:41.686 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-04T17:20:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:Sourcing file `/etc/default/grub' 2024-03-04T17:20:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-04T17:20:42.619 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-03-04T17:20:42.788 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.15.0-97-generic 2024-03-04T17:20:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.15.0-97-generic 2024-03-04T17:20:42.989 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-03-04T17:20:43.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-04T17:20:43.055 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-04T17:20:43.059 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-03-04T17:20:43.060 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-04T17:20:43.101 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-04T17:20:43.106 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-04T17:20:43.227 INFO:teuthology.orchestra.run.smithi153.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-04T17:20:43.227 INFO:teuthology.orchestra.run.smithi153.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-04T17:20:43.227 INFO:teuthology.orchestra.run.smithi153.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-04T17:20:43.230 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2024-03-04T17:20:43.231 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2024-03-04T17:20:43.237 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-03-04T17:20:43.239 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-03-04T17:20:43.381 INFO:teuthology.orchestra.run.smithi136.stdout:linux-image-generic is already the newest version (5.15.0.97.92). 2024-03-04T17:20:43.381 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2024-03-04T17:20:43.381 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-04T17:20:43.381 INFO:teuthology.orchestra.run.smithi136.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-04T17:20:43.381 INFO:teuthology.orchestra.run.smithi136.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-04T17:20:43.382 INFO:teuthology.orchestra.run.smithi136.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-04T17:20:43.382 INFO:teuthology.orchestra.run.smithi136.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-04T17:20:43.382 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-04T17:20:43.420 INFO:teuthology.orchestra.run.smithi136.stdout:0 upgraded, 0 newly installed, 0 to remove and 288 not upgraded. 2024-03-04T17:20:43.422 DEBUG:teuthology.orchestra.run.smithi136:> dpkg -s linux-image-generic 2024-03-04T17:20:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:Package: linux-image-generic 2024-03-04T17:20:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:Status: install ok installed 2024-03-04T17:20:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:Priority: optional 2024-03-04T17:20:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:Section: kernel 2024-03-04T17:20:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:Installed-Size: 21 2024-03-04T17:20:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:Maintainer: Ubuntu Kernel Team 2024-03-04T17:20:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: amd64 2024-03-04T17:20:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:Source: linux-meta 2024-03-04T17:20:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:Version: 5.15.0.97.92 2024-03-04T17:20:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-97), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-04T17:20:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:Depends: linux-image-5.15.0-97-generic, linux-modules-extra-5.15.0-97-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-04T17:20:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:Recommends: thermald 2024-03-04T17:20:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:Description: Generic Linux kernel image 2024-03-04T17:20:43.481 INFO:teuthology.orchestra.run.smithi136.stdout: This package will always depend on the latest generic kernel image 2024-03-04T17:20:43.481 INFO:teuthology.orchestra.run.smithi136.stdout: available. 2024-03-04T17:20:43.481 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-03-04T17:20:43.525 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.of8y0REfqY 2024-03-04T17:20:43.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub/grub.cfg /tmp/tmp.of8y0REfqY 2024-03-04T17:20:43.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.of8y0REfqY 2024-03-04T17:20:43.691 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.of8y0REfqY is 10KB 2024-03-04T17:20:43.703 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.of8y0REfqY 2024-03-04T17:20:43.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-04T17:20:43.764 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:20:43.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-04T17:20:43.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-04T17:20:43.831 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-97-generic 2024-03-04T17:20:43.831 DEBUG:teuthology.orchestra.run.smithi136:> sudo update-grub 2024-03-04T17:20:45.106 INFO:teuthology.orchestra.run.smithi136.stderr:Sourcing file `/etc/default/grub' 2024-03-04T17:20:45.107 INFO:teuthology.orchestra.run.smithi136.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-04T17:20:45.110 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2024-03-04T17:20:45.318 INFO:teuthology.orchestra.run.smithi136.stderr:Found linux image: /boot/vmlinuz-5.15.0-97-generic 2024-03-04T17:20:45.324 INFO:teuthology.orchestra.run.smithi136.stderr:Found initrd image: /boot/initrd.img-5.15.0-97-generic 2024-03-04T17:20:45.682 INFO:teuthology.orchestra.run.smithi136.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-04T17:20:45.687 INFO:teuthology.orchestra.run.smithi136.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-04T17:20:45.731 INFO:teuthology.orchestra.run.smithi136.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-04T17:20:45.737 INFO:teuthology.orchestra.run.smithi136.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-04T17:20:45.871 INFO:teuthology.orchestra.run.smithi136.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-04T17:20:45.871 INFO:teuthology.orchestra.run.smithi136.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-04T17:20:45.871 INFO:teuthology.orchestra.run.smithi136.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-04T17:20:45.891 INFO:teuthology.orchestra.run.smithi136.stderr:done 2024-03-04T17:20:45.893 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2024-03-04T17:21:13.268 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2024-03-04T17:21:13.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T17:21:13.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T17:21:15.899 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2024-03-04T17:21:15.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T17:21:15.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T17:21:31.708 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T17:21:40.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T17:21:40.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T17:21:43.776 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T17:21:50.397 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T17:21:55.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T17:21:55.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T17:21:59.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T17:21:59.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T17:22:14.204 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T17:22:17.792 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T17:22:29.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T17:22:29.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T17:22:29.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T17:22:29.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T17:22:30.097 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-04T17:22:30.268 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-03-04T17:22:31.040 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T17:22:31.041 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "5.15.0-97-generic"... 2024-03-04T17:22:31.041 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-04T17:22:31.086 INFO:teuthology.orchestra.run.smithi153.stdout:5.15.0-97-generic 2024-03-04T17:22:31.087 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-97-generic vs 5.15.0-97-generic 2024-03-04T17:22:31.087 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T17:22:31.087 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-04T17:22:31.172 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T17:22:31.172 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "5.15.0-97-generic"... 2024-03-04T17:22:31.173 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-03-04T17:22:31.221 INFO:teuthology.orchestra.run.smithi136.stdout:5.15.0-97-generic 2024-03-04T17:22:31.221 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-97-generic vs 5.15.0-97-generic 2024-03-04T17:22:31.221 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T17:22:31.221 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-04T17:22:32.088 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2024-03-04T17:22:32.088 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T17:22:32.221 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2024-03-04T17:22:32.222 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T17:22:32.222 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-03-04T17:22:32.248 DEBUG:teuthology.parallel:result is None 2024-03-04T17:22:32.746 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-03-04T17:22:32.770 DEBUG:teuthology.parallel:result is None 2024-03-04T17:22:32.770 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-04T17:22:32.778 INFO:teuthology.task.internal:Creating test directory... 2024-03-04T17:22:32.778 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T17:22:32.780 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T17:22:32.786 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-04T17:22:32.825 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-04T17:22:32.876 INFO:teuthology.task.internal:Creating archive directory... 2024-03-04T17:22:32.876 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T17:22:32.878 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T17:22:32.931 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-04T17:22:32.939 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-04T17:22:32.939 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T17:22:32.941 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T17:22:32.965 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T17:22:32.973 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T17:22:33.030 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T17:22:33.039 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T17:22:33.041 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-04T17:22:33.052 INFO:teuthology.task.internal:Configuring sudo... 2024-03-04T17:22:33.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T17:22:33.055 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T17:22:33.068 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-04T17:22:33.082 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-04T17:22:33.082 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T17:22:33.113 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T17:22:33.118 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T17:22:33.164 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T17:22:33.212 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:22:33.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T17:22:33.276 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T17:22:33.282 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T17:22:33.335 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:22:33.335 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T17:22:33.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-03-04T17:22:33.413 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-03-04T17:22:33.564 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-04T17:22:33.571 INFO:teuthology.task.internal:Starting timer... 2024-03-04T17:22:33.571 INFO:teuthology.run_tasks:Running task pcp... 2024-03-04T17:22:33.603 INFO:teuthology.run_tasks:Running task selinux... 2024-03-04T17:22:33.642 DEBUG:teuthology.task.selinux:Excluding smithi136: OS 'ubuntu' does not support SELinux 2024-03-04T17:22:33.643 DEBUG:teuthology.task.selinux:Excluding smithi153: OS 'ubuntu' does not support SELinux 2024-03-04T17:22:33.643 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-04T17:22:33.643 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-04T17:22:33.643 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-04T17:22:33.643 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-04T17:22:33.693 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-04T17:22:33.708 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-04T17:22:33.710 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi136.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-04T17:29:37.726 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi136.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-03-04T17:29:37.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T17:29:37.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T17:29:37.894 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-03-04T17:29:37.976 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T17:29:37.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T17:29:37.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T17:29:38.126 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-04T17:29:38.204 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T17:29:38.204 INFO:teuthology.run_tasks:Running task clock... 2024-03-04T17:29:38.221 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-04T17:29:38.221 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T17:29:38.222 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T17:29:38.226 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T17:29:38.226 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T17:29:38.264 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-04T17:29:38.264 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: Command line: ntpd -gq 2024-03-04T17:29:38.264 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: ---------------------------------------------------- 2024-03-04T17:29:38.264 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: ntp-4 is maintained by Network Time Foundation, 2024-03-04T17:29:38.264 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-04T17:29:38.264 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: corporation. Support and training for ntp-4 are 2024-03-04T17:29:38.264 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: available at https://www.nwtime.org/support 2024-03-04T17:29:38.264 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: ---------------------------------------------------- 2024-03-04T17:29:38.264 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: proto: precision = 0.057 usec (-24) 2024-03-04T17:29:38.265 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: basedate set to 2022-02-04 2024-03-04T17:29:38.265 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: gps base set to 2022-02-06 (week 2196) 2024-03-04T17:29:38.265 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-04T17:29:38.265 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: restrict ::: KOD does nothing without LIMITED. 2024-03-04T17:29:38.266 INFO:teuthology.orchestra.run.smithi136.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-04T17:29:38.266 INFO:teuthology.orchestra.run.smithi136.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-04T17:29:38.267 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: Listen and drop on 0 v6wildcard [::]:123 2024-03-04T17:29:38.268 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-04T17:29:38.268 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: Listen normally on 2 lo 127.0.0.1:123 2024-03-04T17:29:38.269 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: Listen normally on 3 enp3s0f1 172.21.15.136:123 2024-03-04T17:29:38.269 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: Listen normally on 4 lo [::1]:123 2024-03-04T17:29:38.269 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e43%5]:123 2024-03-04T17:29:38.269 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:38 ntpd[17165]: Listening on routing socket on fd #22 for interface updates 2024-03-04T17:29:38.277 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-04T17:29:38.277 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: Command line: ntpd -gq 2024-03-04T17:29:38.277 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: ---------------------------------------------------- 2024-03-04T17:29:38.277 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: ntp-4 is maintained by Network Time Foundation, 2024-03-04T17:29:38.277 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-04T17:29:38.277 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: corporation. Support and training for ntp-4 are 2024-03-04T17:29:38.278 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: available at https://www.nwtime.org/support 2024-03-04T17:29:38.278 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: ---------------------------------------------------- 2024-03-04T17:29:38.278 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: proto: precision = 0.046 usec (-24) 2024-03-04T17:29:38.278 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: basedate set to 2022-02-04 2024-03-04T17:29:38.278 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: gps base set to 2022-02-06 (week 2196) 2024-03-04T17:29:38.278 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-04T17:29:38.279 INFO:teuthology.orchestra.run.smithi153.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-04T17:29:38.279 INFO:teuthology.orchestra.run.smithi153.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-04T17:29:38.280 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: restrict ::: KOD does nothing without LIMITED. 2024-03-04T17:29:38.280 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: Listen and drop on 0 v6wildcard [::]:123 2024-03-04T17:29:38.280 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-04T17:29:38.281 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: Listen normally on 2 lo 127.0.0.1:123 2024-03-04T17:29:38.281 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: Listen normally on 3 enp3s0f1 172.21.15.153:123 2024-03-04T17:29:38.281 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: Listen normally on 4 lo [::1]:123 2024-03-04T17:29:38.281 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6deb%5]:123 2024-03-04T17:29:38.281 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:38 ntpd[17164]: Listening on routing socket on fd #22 for interface updates 2024-03-04T17:29:39.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:39.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:39.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:39.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:39.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:39.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:39.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:39.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:39.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:39.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:39.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:39.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:39.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:39.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:39 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:40.277 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:40 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:40.277 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:40 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:40.277 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:40 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:40.277 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:40 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:40.277 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:40 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:40.277 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:40 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:40.277 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:40 ntpd[17164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:40.277 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:40 ntpd[17164]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:40.277 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:40 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:40.277 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:40 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:40.277 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:40 ntpd[17164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:40.277 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:40 ntpd[17164]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:41.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:41 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:41.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:41 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:41.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:41 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:41.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:41 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:41.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:41 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:41.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:41 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:41.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:41 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:41.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:41 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:41.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:41 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:41.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:41 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:41.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:41 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:41.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:41 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:42.281 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:42.281 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:42.281 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:42.282 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:42.282 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:42.282 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:42.282 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:42.282 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:42.282 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:42.282 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:42.282 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:42.282 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:42.282 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:42.282 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:42.282 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:42.282 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:42.283 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:42.283 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:42 ntpd[17164]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:43.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:43.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:43.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:43.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:43.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:43.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:43.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:43.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:43.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:43.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:43.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:43.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:43.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:43.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:43.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:43.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:43.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:43.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:43 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:43.276 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:43 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:43.276 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:43 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:44.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:44.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:44.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:44.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:44.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:44.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:44.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:44.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:44.280 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:44.280 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:44.280 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:44.280 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:44.280 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:44.280 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:44.280 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:44.280 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:44.280 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:44.280 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:44 ntpd[17164]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:45.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:45 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:45.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:45 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:45.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:45 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:45.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:45 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:45.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:45 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:45.253 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:45 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:45.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:45 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:45.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:45 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:45.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:45 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:45.254 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:45 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:45.254 INFO:teuthology.orchestra.run.smithi136.stdout: 4 Mar 17:29:45 ntpd[17165]: ntpd: time slew -0.002014 s 2024-03-04T17:29:45.254 INFO:teuthology.orchestra.run.smithi136.stdout:ntpd: time slew -0.002014s 2024-03-04T17:29:45.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:45 ntpd[17165]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-04T17:29:45.255 INFO:teuthology.orchestra.run.smithi136.stderr: 4 Mar 17:29:45 ntpd[17165]: can't open /var/log/ntpstats/loopstats.20240304: Permission denied 2024-03-04T17:29:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: remote refid st t when poll reach delay offset jitter 2024-03-04T17:29:45.301 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================== 2024-03-04T17:29:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-04T17:29:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-04T17:29:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-04T17:29:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-04T17:29:46.278 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:46 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:46.278 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:46 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:46.278 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:46 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:46.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:46 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:46.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:46 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:46.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:46 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:46.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:46 ntpd[17164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-04T17:29:46.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:46 ntpd[17164]: can't open /var/log/ntpstats/rawstats.20240304: Permission denied 2024-03-04T17:29:46.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:46 ntpd[17164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-04T17:29:46.279 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:46 ntpd[17164]: can't open /var/log/ntpstats/peerstats.20240304: Permission denied 2024-03-04T17:29:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: 4 Mar 17:29:46 ntpd[17164]: ntpd: time slew -0.001123 s 2024-03-04T17:29:46.284 INFO:teuthology.orchestra.run.smithi153.stdout:ntpd: time slew -0.001123s 2024-03-04T17:29:46.285 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:46 ntpd[17164]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-04T17:29:46.285 INFO:teuthology.orchestra.run.smithi153.stderr: 4 Mar 17:29:46 ntpd[17164]: can't open /var/log/ntpstats/loopstats.20240304: Permission denied 2024-03-04T17:29:46.335 INFO:teuthology.orchestra.run.smithi153.stdout: remote refid st t when poll reach delay offset jitter 2024-03-04T17:29:46.343 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================== 2024-03-04T17:29:46.343 INFO:teuthology.orchestra.run.smithi153.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-04T17:29:46.343 INFO:teuthology.orchestra.run.smithi153.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-04T17:29:46.343 INFO:teuthology.orchestra.run.smithi153.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-04T17:29:46.343 INFO:teuthology.orchestra.run.smithi153.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-04T17:29:46.344 INFO:teuthology.run_tasks:Running task install... 2024-03-04T17:29:46.361 DEBUG:teuthology.task.install:project ceph 2024-03-04T17:29:46.361 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8'}} 2024-03-04T17:29:46.361 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8'} 2024-03-04T17:29:46.362 INFO:teuthology.task.install:Using flavor: default 2024-03-04T17:29:46.367 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-04T17:29:46.367 INFO:teuthology.task.install:extra packages: [] 2024-03-04T17:29:46.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-key list | grep Ceph 2024-03-04T17:29:46.378 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-key list | grep Ceph 2024-03-04T17:29:46.437 INFO:teuthology.orchestra.run.smithi153.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-04T17:29:46.442 INFO:teuthology.orchestra.run.smithi136.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-04T17:29:46.467 INFO:teuthology.orchestra.run.smithi153.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-04T17:29:46.467 INFO:teuthology.orchestra.run.smithi153.stdout:uid [ unknown] Ceph.com (release key) 2024-03-04T17:29:46.467 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-04T17:29:46.467 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3aec390336fdfdc8f927818b718cfb0361bd47c8 2024-03-04T17:29:46.474 INFO:teuthology.orchestra.run.smithi136.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-04T17:29:46.474 INFO:teuthology.orchestra.run.smithi136.stdout:uid [ unknown] Ceph.com (release key) 2024-03-04T17:29:46.474 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-04T17:29:46.477 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3aec390336fdfdc8f927818b718cfb0361bd47c8 2024-03-04T17:29:46.739 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default/ 2024-03-04T17:29:46.742 INFO:teuthology.task.install.deb:Package version is 19.0.0-1871-g3aec3903-1jammy 2024-03-04T17:29:46.756 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default/ 2024-03-04T17:29:46.756 INFO:teuthology.task.install.deb:Package version is 19.0.0-1871-g3aec3903-1jammy 2024-03-04T17:29:47.026 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:29:47.027 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-04T17:29:47.049 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2024-03-04T17:29:47.240 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-04T17:29:47.265 INFO:teuthology.orchestra.run.smithi153.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy InRelease 2024-03-04T17:29:47.285 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-04T17:29:47.307 INFO:teuthology.orchestra.run.smithi153.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-04T17:29:47.329 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,395 B] 2024-03-04T17:29:47.335 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-04T17:29:47.364 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-04T17:29:47.415 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-04T17:29:47.417 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:29:47.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-04T17:29:47.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2024-03-04T17:29:47.638 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-04T17:29:47.638 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-04T17:29:47.653 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-04T17:29:47.676 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-04T17:29:47.770 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-04T17:29:47.926 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 148 kB in 1s (212 kB/s) 2024-03-04T17:29:47.957 INFO:teuthology.orchestra.run.smithi136.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy InRelease 2024-03-04T17:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-04T17:29:48.147 INFO:teuthology.orchestra.run.smithi136.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-04T17:29:48.235 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-04T17:29:48.332 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,395 B] 2024-03-04T17:29:48.419 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 28.9 kB in 1s (38.3 kB/s) 2024-03-04T17:29:48.859 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-03-04T17:29:48.872 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1871-g3aec3903-1jammy cephadm=19.0.0-1871-g3aec3903-1jammy ceph-mds=19.0.0-1871-g3aec3903-1jammy ceph-mgr=19.0.0-1871-g3aec3903-1jammy ceph-common=19.0.0-1871-g3aec3903-1jammy ceph-fuse=19.0.0-1871-g3aec3903-1jammy ceph-test=19.0.0-1871-g3aec3903-1jammy ceph-volume=19.0.0-1871-g3aec3903-1jammy radosgw=19.0.0-1871-g3aec3903-1jammy python3-rados=19.0.0-1871-g3aec3903-1jammy python3-rgw=19.0.0-1871-g3aec3903-1jammy python3-cephfs=19.0.0-1871-g3aec3903-1jammy python3-rbd=19.0.0-1871-g3aec3903-1jammy libcephfs2=19.0.0-1871-g3aec3903-1jammy libcephfs-dev=19.0.0-1871-g3aec3903-1jammy librados2=19.0.0-1871-g3aec3903-1jammy librbd1=19.0.0-1871-g3aec3903-1jammy rbd-fuse=19.0.0-1871-g3aec3903-1jammy 2024-03-04T17:29:48.933 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-03-04T17:29:49.128 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2024-03-04T17:29:49.128 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2024-03-04T17:29:49.302 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2024-03-04T17:29:49.303 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-04T17:29:49.303 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-04T17:29:49.303 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-04T17:29:49.303 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-04T17:29:49.303 INFO:teuthology.orchestra.run.smithi153.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-04T17:29:49.303 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-04T17:29:49.304 INFO:teuthology.orchestra.run.smithi153.stdout:The following additional packages will be installed: 2024-03-04T17:29:49.304 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-04T17:29:49.304 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-04T17:29:49.304 INFO:teuthology.orchestra.run.smithi153.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-04T17:29:49.304 INFO:teuthology.orchestra.run.smithi153.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-04T17:29:49.305 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-04T17:29:49.306 INFO:teuthology.orchestra.run.smithi153.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-04T17:29:49.306 INFO:teuthology.orchestra.run.smithi153.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-04T17:29:49.306 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-04T17:29:49.306 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-04T17:29:49.306 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-04T17:29:49.306 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-04T17:29:49.306 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-04T17:29:49.306 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-04T17:29:49.306 INFO:teuthology.orchestra.run.smithi153.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-04T17:29:49.306 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-04T17:29:49.306 INFO:teuthology.orchestra.run.smithi153.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-04T17:29:49.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-04T17:29:49.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-04T17:29:49.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-04T17:29:49.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-04T17:29:49.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-04T17:29:49.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-04T17:29:49.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-04T17:29:49.307 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet zip 2024-03-04T17:29:49.307 INFO:teuthology.orchestra.run.smithi153.stdout:Suggested packages: 2024-03-04T17:29:49.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-04T17:29:49.308 INFO:teuthology.orchestra.run.smithi153.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-04T17:29:49.308 INFO:teuthology.orchestra.run.smithi153.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-04T17:29:49.308 INFO:teuthology.orchestra.run.smithi153.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-04T17:29:49.308 INFO:teuthology.orchestra.run.smithi153.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-04T17:29:49.308 INFO:teuthology.orchestra.run.smithi153.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-04T17:29:49.308 INFO:teuthology.orchestra.run.smithi153.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-04T17:29:49.308 INFO:teuthology.orchestra.run.smithi153.stdout:Recommended packages: 2024-03-04T17:29:49.308 INFO:teuthology.orchestra.run.smithi153.stdout: btrfs-tools 2024-03-04T17:29:49.355 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-03-04T17:29:49.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1871-g3aec3903-1jammy cephadm=19.0.0-1871-g3aec3903-1jammy ceph-mds=19.0.0-1871-g3aec3903-1jammy ceph-mgr=19.0.0-1871-g3aec3903-1jammy ceph-common=19.0.0-1871-g3aec3903-1jammy ceph-fuse=19.0.0-1871-g3aec3903-1jammy ceph-test=19.0.0-1871-g3aec3903-1jammy ceph-volume=19.0.0-1871-g3aec3903-1jammy radosgw=19.0.0-1871-g3aec3903-1jammy python3-rados=19.0.0-1871-g3aec3903-1jammy python3-rgw=19.0.0-1871-g3aec3903-1jammy python3-cephfs=19.0.0-1871-g3aec3903-1jammy python3-rbd=19.0.0-1871-g3aec3903-1jammy libcephfs2=19.0.0-1871-g3aec3903-1jammy libcephfs-dev=19.0.0-1871-g3aec3903-1jammy librados2=19.0.0-1871-g3aec3903-1jammy librbd1=19.0.0-1871-g3aec3903-1jammy rbd-fuse=19.0.0-1871-g3aec3903-1jammy 2024-03-04T17:29:49.414 INFO:teuthology.orchestra.run.smithi153.stdout:The following NEW packages will be installed: 2024-03-04T17:29:49.414 INFO:teuthology.orchestra.run.smithi153.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-04T17:29:49.414 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-04T17:29:49.414 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-04T17:29:49.414 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-04T17:29:49.414 INFO:teuthology.orchestra.run.smithi153.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-04T17:29:49.414 INFO:teuthology.orchestra.run.smithi153.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-04T17:29:49.415 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-04T17:29:49.416 INFO:teuthology.orchestra.run.smithi153.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-04T17:29:49.416 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-04T17:29:49.416 INFO:teuthology.orchestra.run.smithi153.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-04T17:29:49.416 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-04T17:29:49.416 INFO:teuthology.orchestra.run.smithi153.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-04T17:29:49.422 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-03-04T17:29:49.599 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2024-03-04T17:29:49.599 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 118 newly installed, 0 to remove and 275 not upgraded. 2024-03-04T17:29:49.600 INFO:teuthology.orchestra.run.smithi153.stdout:Need to get 177 MB of archives. 2024-03-04T17:29:49.600 INFO:teuthology.orchestra.run.smithi153.stdout:After this operation, 734 MB of additional disk space will be used. 2024-03-04T17:29:49.600 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-04T17:29:49.600 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2024-03-04T17:29:49.759 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2024-03-04T17:29:49.759 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-04T17:29:49.759 INFO:teuthology.orchestra.run.smithi136.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-04T17:29:49.759 INFO:teuthology.orchestra.run.smithi136.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-04T17:29:49.759 INFO:teuthology.orchestra.run.smithi136.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-04T17:29:49.760 INFO:teuthology.orchestra.run.smithi136.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-04T17:29:49.760 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-04T17:29:49.760 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2024-03-04T17:29:49.760 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-04T17:29:49.760 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-04T17:29:49.761 INFO:teuthology.orchestra.run.smithi136.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-04T17:29:49.761 INFO:teuthology.orchestra.run.smithi136.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-04T17:29:49.761 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-04T17:29:49.761 INFO:teuthology.orchestra.run.smithi136.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-04T17:29:49.761 INFO:teuthology.orchestra.run.smithi136.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-04T17:29:49.762 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-04T17:29:49.762 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-04T17:29:49.762 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-04T17:29:49.762 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-04T17:29:49.762 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-04T17:29:49.762 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-04T17:29:49.764 INFO:teuthology.orchestra.run.smithi136.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-04T17:29:49.764 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-04T17:29:49.764 INFO:teuthology.orchestra.run.smithi136.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-04T17:29:49.764 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-04T17:29:49.765 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-04T17:29:49.765 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-04T17:29:49.765 INFO:teuthology.orchestra.run.smithi136.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-04T17:29:49.765 INFO:teuthology.orchestra.run.smithi136.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-04T17:29:49.765 INFO:teuthology.orchestra.run.smithi136.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-04T17:29:49.765 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-04T17:29:49.765 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet zip 2024-03-04T17:29:49.765 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2024-03-04T17:29:49.765 INFO:teuthology.orchestra.run.smithi136.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-04T17:29:49.766 INFO:teuthology.orchestra.run.smithi136.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-04T17:29:49.766 INFO:teuthology.orchestra.run.smithi136.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-04T17:29:49.766 INFO:teuthology.orchestra.run.smithi136.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-04T17:29:49.766 INFO:teuthology.orchestra.run.smithi136.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-04T17:29:49.766 INFO:teuthology.orchestra.run.smithi136.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-04T17:29:49.766 INFO:teuthology.orchestra.run.smithi136.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-04T17:29:49.766 INFO:teuthology.orchestra.run.smithi136.stdout:Recommended packages: 2024-03-04T17:29:49.766 INFO:teuthology.orchestra.run.smithi136.stdout: btrfs-tools 2024-03-04T17:29:49.861 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2024-03-04T17:29:49.861 INFO:teuthology.orchestra.run.smithi136.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-04T17:29:49.861 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-04T17:29:49.861 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-04T17:29:49.861 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-04T17:29:49.861 INFO:teuthology.orchestra.run.smithi136.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-04T17:29:49.861 INFO:teuthology.orchestra.run.smithi136.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-04T17:29:49.862 INFO:teuthology.orchestra.run.smithi136.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-04T17:29:49.862 INFO:teuthology.orchestra.run.smithi136.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-04T17:29:49.862 INFO:teuthology.orchestra.run.smithi136.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-04T17:29:49.863 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-04T17:29:49.864 INFO:teuthology.orchestra.run.smithi136.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-04T17:29:49.864 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-04T17:29:49.864 INFO:teuthology.orchestra.run.smithi136.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-04T17:29:49.900 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1871-g3aec3903-1jammy [3,547 kB] 2024-03-04T17:29:50.052 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-04T17:29:50.052 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-04T17:29:50.055 INFO:teuthology.orchestra.run.smithi136.stdout:0 upgraded, 118 newly installed, 0 to remove and 275 not upgraded. 2024-03-04T17:29:50.055 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 177 MB of archives. 2024-03-04T17:29:50.055 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 734 MB of additional disk space will be used. 2024-03-04T17:29:50.055 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-04T17:29:50.123 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-04T17:29:50.317 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1871-g3aec3903-1jammy [3,547 kB] 2024-03-04T17:29:50.355 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-04T17:29:50.366 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-04T17:29:50.396 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-04T17:29:50.406 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-04T17:29:50.406 INFO:teuthology.orchestra.run.smithi153.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-04T17:29:50.408 INFO:teuthology.orchestra.run.smithi153.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-04T17:29:50.429 INFO:teuthology.orchestra.run.smithi153.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-04T17:29:50.473 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-04T17:29:50.484 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-04T17:29:50.569 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-04T17:29:50.596 INFO:teuthology.orchestra.run.smithi153.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-04T17:29:50.795 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-04T17:29:50.807 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-04T17:29:50.837 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-04T17:29:50.846 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-04T17:29:50.847 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-04T17:29:50.852 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-04T17:29:50.873 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-04T17:29:51.003 INFO:teuthology.orchestra.run.smithi153.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-04T17:29:51.036 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-04T17:29:51.037 INFO:teuthology.orchestra.run.smithi153.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-04T17:29:51.037 INFO:teuthology.orchestra.run.smithi153.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-04T17:29:51.115 INFO:teuthology.orchestra.run.smithi153.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-04T17:29:51.167 INFO:teuthology.orchestra.run.smithi153.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-04T17:29:51.188 INFO:teuthology.orchestra.run.smithi153.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-04T17:29:51.194 INFO:teuthology.orchestra.run.smithi153.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-04T17:29:51.196 INFO:teuthology.orchestra.run.smithi153.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-04T17:29:51.207 INFO:teuthology.orchestra.run.smithi153.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-04T17:29:51.207 INFO:teuthology.orchestra.run.smithi153.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-04T17:29:51.373 INFO:teuthology.orchestra.run.smithi153.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-04T17:29:51.436 INFO:teuthology.orchestra.run.smithi136.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-04T17:29:51.456 INFO:teuthology.orchestra.run.smithi153.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-04T17:29:51.471 INFO:teuthology.orchestra.run.smithi136.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-04T17:29:51.472 INFO:teuthology.orchestra.run.smithi136.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-04T17:29:51.484 INFO:teuthology.orchestra.run.smithi153.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-04T17:29:51.502 INFO:teuthology.orchestra.run.smithi153.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-04T17:29:51.546 INFO:teuthology.orchestra.run.smithi153.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-04T17:29:51.551 INFO:teuthology.orchestra.run.smithi136.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-04T17:29:51.603 INFO:teuthology.orchestra.run.smithi136.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-04T17:29:51.626 INFO:teuthology.orchestra.run.smithi136.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-04T17:29:51.630 INFO:teuthology.orchestra.run.smithi136.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-04T17:29:51.632 INFO:teuthology.orchestra.run.smithi136.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-04T17:29:51.643 INFO:teuthology.orchestra.run.smithi136.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-04T17:29:51.644 INFO:teuthology.orchestra.run.smithi136.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-04T17:29:51.761 INFO:teuthology.orchestra.run.smithi153.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-04T17:29:51.776 INFO:teuthology.orchestra.run.smithi153.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-04T17:29:51.783 INFO:teuthology.orchestra.run.smithi153.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-04T17:29:51.788 INFO:teuthology.orchestra.run.smithi153.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-04T17:29:51.809 INFO:teuthology.orchestra.run.smithi153.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-04T17:29:51.809 INFO:teuthology.orchestra.run.smithi136.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-04T17:29:51.813 INFO:teuthology.orchestra.run.smithi153.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-04T17:29:51.892 INFO:teuthology.orchestra.run.smithi136.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-04T17:29:51.920 INFO:teuthology.orchestra.run.smithi136.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-04T17:29:51.920 INFO:teuthology.orchestra.run.smithi136.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1871-g3aec3903-1jammy [3,217 kB] 2024-03-04T17:29:51.940 INFO:teuthology.orchestra.run.smithi136.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-04T17:29:51.978 INFO:teuthology.orchestra.run.smithi153.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-04T17:29:51.982 INFO:teuthology.orchestra.run.smithi136.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-04T17:29:52.154 INFO:teuthology.orchestra.run.smithi153.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1871-g3aec3903-1jammy [3,217 kB] 2024-03-04T17:29:52.197 INFO:teuthology.orchestra.run.smithi136.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-04T17:29:52.211 INFO:teuthology.orchestra.run.smithi136.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-04T17:29:52.220 INFO:teuthology.orchestra.run.smithi136.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-04T17:29:52.223 INFO:teuthology.orchestra.run.smithi136.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-04T17:29:52.245 INFO:teuthology.orchestra.run.smithi136.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-04T17:29:52.249 INFO:teuthology.orchestra.run.smithi136.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-04T17:29:52.310 INFO:teuthology.orchestra.run.smithi153.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-04T17:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-04T17:29:52.351 INFO:teuthology.orchestra.run.smithi153.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-04T17:29:52.416 INFO:teuthology.orchestra.run.smithi136.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-04T17:29:52.557 INFO:teuthology.orchestra.run.smithi153.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-04T17:29:52.557 INFO:teuthology.orchestra.run.smithi153.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-04T17:29:52.561 INFO:teuthology.orchestra.run.smithi153.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-04T17:29:52.567 INFO:teuthology.orchestra.run.smithi153.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-04T17:29:52.582 INFO:teuthology.orchestra.run.smithi153.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-04T17:29:52.603 INFO:teuthology.orchestra.run.smithi153.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-04T17:29:52.609 INFO:teuthology.orchestra.run.smithi153.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-04T17:29:52.679 INFO:teuthology.orchestra.run.smithi136.stdout:Get:37 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1871-g3aec3903-1jammy [710 kB] 2024-03-04T17:29:52.730 INFO:teuthology.orchestra.run.smithi136.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-04T17:29:52.768 INFO:teuthology.orchestra.run.smithi136.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-04T17:29:52.774 INFO:teuthology.orchestra.run.smithi136.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1871-g3aec3903-1jammy [356 kB] 2024-03-04T17:29:52.780 INFO:teuthology.orchestra.run.smithi136.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-04T17:29:52.784 INFO:teuthology.orchestra.run.smithi153.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-04T17:29:52.865 INFO:teuthology.orchestra.run.smithi136.stdout:Get:42 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1871-g3aec3903-1jammy [32.8 kB] 2024-03-04T17:29:52.865 INFO:teuthology.orchestra.run.smithi136.stdout:Get:43 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1871-g3aec3903-1jammy [180 kB] 2024-03-04T17:29:52.941 INFO:teuthology.orchestra.run.smithi136.stdout:Get:44 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1871-g3aec3903-1jammy [64.4 kB] 2024-03-04T17:29:52.943 INFO:teuthology.orchestra.run.smithi136.stdout:Get:45 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1871-g3aec3903-1jammy [330 kB] 2024-03-04T17:29:52.956 INFO:teuthology.orchestra.run.smithi136.stdout:Get:46 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1871-g3aec3903-1jammy [6,634 kB] 2024-03-04T17:29:52.984 INFO:teuthology.orchestra.run.smithi136.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-04T17:29:52.985 INFO:teuthology.orchestra.run.smithi136.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-04T17:29:52.991 INFO:teuthology.orchestra.run.smithi136.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-04T17:29:52.997 INFO:teuthology.orchestra.run.smithi136.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-04T17:29:53.011 INFO:teuthology.orchestra.run.smithi136.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-04T17:29:53.033 INFO:teuthology.orchestra.run.smithi136.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-04T17:29:53.038 INFO:teuthology.orchestra.run.smithi136.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-04T17:29:53.041 INFO:teuthology.orchestra.run.smithi153.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-04T17:29:53.055 INFO:teuthology.orchestra.run.smithi153.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1871-g3aec3903-1jammy [710 kB] 2024-03-04T17:29:53.174 INFO:teuthology.orchestra.run.smithi153.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1871-g3aec3903-1jammy [356 kB] 2024-03-04T17:29:53.213 INFO:teuthology.orchestra.run.smithi136.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-04T17:29:53.218 INFO:teuthology.orchestra.run.smithi153.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-04T17:29:53.233 INFO:teuthology.orchestra.run.smithi153.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-04T17:29:53.245 INFO:teuthology.orchestra.run.smithi153.stdout:Get:53 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1871-g3aec3903-1jammy [32.8 kB] 2024-03-04T17:29:53.245 INFO:teuthology.orchestra.run.smithi153.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1871-g3aec3903-1jammy [180 kB] 2024-03-04T17:29:53.253 INFO:teuthology.orchestra.run.smithi153.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-04T17:29:53.300 INFO:teuthology.orchestra.run.smithi153.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1871-g3aec3903-1jammy [64.4 kB] 2024-03-04T17:29:53.300 INFO:teuthology.orchestra.run.smithi153.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1871-g3aec3903-1jammy [330 kB] 2024-03-04T17:29:53.318 INFO:teuthology.orchestra.run.smithi153.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-04T17:29:53.354 INFO:teuthology.orchestra.run.smithi153.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1871-g3aec3903-1jammy [6,634 kB] 2024-03-04T17:29:53.466 INFO:teuthology.orchestra.run.smithi136.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-04T17:29:53.608 INFO:teuthology.orchestra.run.smithi153.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-04T17:29:53.610 INFO:teuthology.orchestra.run.smithi153.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-04T17:29:53.612 INFO:teuthology.orchestra.run.smithi153.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-04T17:29:53.613 INFO:teuthology.orchestra.run.smithi153.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-04T17:29:53.615 INFO:teuthology.orchestra.run.smithi153.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-04T17:29:53.643 INFO:teuthology.orchestra.run.smithi136.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-04T17:29:53.657 INFO:teuthology.orchestra.run.smithi136.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-04T17:29:53.678 INFO:teuthology.orchestra.run.smithi136.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-04T17:29:53.741 INFO:teuthology.orchestra.run.smithi136.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-04T17:29:53.791 INFO:teuthology.orchestra.run.smithi136.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1871-g3aec3903-1jammy [112 kB] 2024-03-04T17:29:53.793 INFO:teuthology.orchestra.run.smithi136.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1871-g3aec3903-1jammy [466 kB] 2024-03-04T17:29:53.843 INFO:teuthology.orchestra.run.smithi153.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-04T17:29:53.846 INFO:teuthology.orchestra.run.smithi136.stdout:Get:62 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1871-g3aec3903-1jammy [24.2 MB] 2024-03-04T17:29:54.028 INFO:teuthology.orchestra.run.smithi136.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-04T17:29:54.031 INFO:teuthology.orchestra.run.smithi136.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-04T17:29:54.033 INFO:teuthology.orchestra.run.smithi136.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-04T17:29:54.034 INFO:teuthology.orchestra.run.smithi136.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-04T17:29:54.036 INFO:teuthology.orchestra.run.smithi136.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-04T17:29:54.248 INFO:teuthology.orchestra.run.smithi153.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1871-g3aec3903-1jammy [112 kB] 2024-03-04T17:29:54.248 INFO:teuthology.orchestra.run.smithi153.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1871-g3aec3903-1jammy [466 kB] 2024-03-04T17:29:54.260 INFO:teuthology.orchestra.run.smithi153.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1871-g3aec3903-1jammy [24.2 MB] 2024-03-04T17:29:54.265 INFO:teuthology.orchestra.run.smithi136.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-04T17:29:54.282 INFO:teuthology.orchestra.run.smithi153.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-04T17:29:54.292 INFO:teuthology.orchestra.run.smithi153.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-04T17:29:54.580 INFO:teuthology.orchestra.run.smithi153.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-04T17:29:54.581 INFO:teuthology.orchestra.run.smithi153.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-04T17:29:54.582 INFO:teuthology.orchestra.run.smithi153.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-04T17:29:54.587 INFO:teuthology.orchestra.run.smithi153.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-04T17:29:54.590 INFO:teuthology.orchestra.run.smithi153.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-04T17:29:54.590 INFO:teuthology.orchestra.run.smithi153.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-04T17:29:54.591 INFO:teuthology.orchestra.run.smithi153.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-04T17:29:54.592 INFO:teuthology.orchestra.run.smithi153.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-04T17:29:54.702 INFO:teuthology.orchestra.run.smithi136.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-04T17:29:54.708 INFO:teuthology.orchestra.run.smithi136.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-04T17:29:54.772 INFO:teuthology.orchestra.run.smithi153.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-04T17:29:55.000 INFO:teuthology.orchestra.run.smithi136.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-04T17:29:55.001 INFO:teuthology.orchestra.run.smithi136.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-04T17:29:55.001 INFO:teuthology.orchestra.run.smithi136.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-04T17:29:55.003 INFO:teuthology.orchestra.run.smithi136.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-04T17:29:55.004 INFO:teuthology.orchestra.run.smithi136.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-04T17:29:55.006 INFO:teuthology.orchestra.run.smithi136.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-04T17:29:55.007 INFO:teuthology.orchestra.run.smithi136.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-04T17:29:55.010 INFO:teuthology.orchestra.run.smithi136.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-04T17:29:55.181 INFO:teuthology.orchestra.run.smithi153.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-04T17:29:55.190 INFO:teuthology.orchestra.run.smithi136.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-04T17:29:55.225 INFO:teuthology.orchestra.run.smithi153.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-04T17:29:55.260 INFO:teuthology.orchestra.run.smithi153.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-04T17:29:55.358 INFO:teuthology.orchestra.run.smithi153.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-04T17:29:55.398 INFO:teuthology.orchestra.run.smithi153.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-04T17:29:55.410 INFO:teuthology.orchestra.run.smithi153.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-04T17:29:55.416 INFO:teuthology.orchestra.run.smithi153.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-04T17:29:55.463 INFO:teuthology.orchestra.run.smithi153.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-04T17:29:55.466 INFO:teuthology.orchestra.run.smithi153.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-04T17:29:55.489 INFO:teuthology.orchestra.run.smithi153.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-04T17:29:55.603 INFO:teuthology.orchestra.run.smithi136.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-04T17:29:55.648 INFO:teuthology.orchestra.run.smithi136.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-04T17:29:55.651 INFO:teuthology.orchestra.run.smithi153.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-04T17:29:55.685 INFO:teuthology.orchestra.run.smithi136.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-04T17:29:55.781 INFO:teuthology.orchestra.run.smithi136.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-04T17:29:55.820 INFO:teuthology.orchestra.run.smithi136.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-04T17:29:55.832 INFO:teuthology.orchestra.run.smithi136.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-04T17:29:55.837 INFO:teuthology.orchestra.run.smithi136.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-04T17:29:55.884 INFO:teuthology.orchestra.run.smithi136.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-04T17:29:55.887 INFO:teuthology.orchestra.run.smithi136.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-04T17:29:55.909 INFO:teuthology.orchestra.run.smithi136.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-04T17:29:55.928 INFO:teuthology.orchestra.run.smithi153.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-04T17:29:55.943 INFO:teuthology.orchestra.run.smithi153.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-04T17:29:56.072 INFO:teuthology.orchestra.run.smithi136.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-04T17:29:56.097 INFO:teuthology.orchestra.run.smithi153.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-04T17:29:56.125 INFO:teuthology.orchestra.run.smithi153.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-04T17:29:56.283 INFO:teuthology.orchestra.run.smithi153.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-04T17:29:56.286 INFO:teuthology.orchestra.run.smithi153.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-04T17:29:56.288 INFO:teuthology.orchestra.run.smithi153.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-04T17:29:56.311 INFO:teuthology.orchestra.run.smithi153.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-04T17:29:56.315 INFO:teuthology.orchestra.run.smithi153.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-04T17:29:56.345 INFO:teuthology.orchestra.run.smithi136.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-04T17:29:56.367 INFO:teuthology.orchestra.run.smithi136.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-04T17:29:56.502 INFO:teuthology.orchestra.run.smithi136.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1871-g3aec3903-1jammy [5,077 kB] 2024-03-04T17:29:56.517 INFO:teuthology.orchestra.run.smithi136.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-04T17:29:56.546 INFO:teuthology.orchestra.run.smithi136.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-04T17:29:56.703 INFO:teuthology.orchestra.run.smithi136.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-04T17:29:56.706 INFO:teuthology.orchestra.run.smithi136.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-04T17:29:56.707 INFO:teuthology.orchestra.run.smithi136.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-04T17:29:56.730 INFO:teuthology.orchestra.run.smithi136.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-04T17:29:56.734 INFO:teuthology.orchestra.run.smithi136.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-04T17:29:57.091 INFO:teuthology.orchestra.run.smithi136.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1871-g3aec3903-1jammy [243 kB] 2024-03-04T17:29:57.138 INFO:teuthology.orchestra.run.smithi136.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1871-g3aec3903-1jammy [124 kB] 2024-03-04T17:29:57.138 INFO:teuthology.orchestra.run.smithi136.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1871-g3aec3903-1jammy [1,521 kB] 2024-03-04T17:29:57.230 INFO:teuthology.orchestra.run.smithi153.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1871-g3aec3903-1jammy [5,077 kB] 2024-03-04T17:29:57.297 INFO:teuthology.orchestra.run.smithi136.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1871-g3aec3903-1jammy [6,237 kB] 2024-03-04T17:29:57.782 INFO:teuthology.orchestra.run.smithi153.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1871-g3aec3903-1jammy [243 kB] 2024-03-04T17:29:57.835 INFO:teuthology.orchestra.run.smithi153.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1871-g3aec3903-1jammy [124 kB] 2024-03-04T17:29:57.857 INFO:teuthology.orchestra.run.smithi153.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1871-g3aec3903-1jammy [1,521 kB] 2024-03-04T17:29:58.042 INFO:teuthology.orchestra.run.smithi153.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1871-g3aec3903-1jammy [6,237 kB] 2024-03-04T17:29:58.112 INFO:teuthology.orchestra.run.smithi136.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1871-g3aec3903-1jammy [22.7 MB] 2024-03-04T17:29:58.862 INFO:teuthology.orchestra.run.smithi153.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1871-g3aec3903-1jammy [22.7 MB] 2024-03-04T17:30:01.341 INFO:teuthology.orchestra.run.smithi136.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1871-g3aec3903-1jammy [14.0 kB] 2024-03-04T17:30:01.342 INFO:teuthology.orchestra.run.smithi136.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1871-g3aec3903-1jammy [866 kB] 2024-03-04T17:30:01.455 INFO:teuthology.orchestra.run.smithi136.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1871-g3aec3903-1jammy [2,190 kB] 2024-03-04T17:30:01.819 INFO:teuthology.orchestra.run.smithi136.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1871-g3aec3903-1jammy [777 kB] 2024-03-04T17:30:01.942 INFO:teuthology.orchestra.run.smithi136.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1871-g3aec3903-1jammy [144 kB] 2024-03-04T17:30:01.969 INFO:teuthology.orchestra.run.smithi136.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1871-g3aec3903-1jammy [1,922 kB] 2024-03-04T17:30:02.304 INFO:teuthology.orchestra.run.smithi136.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1871-g3aec3903-1jammy [8,625 kB] 2024-03-04T17:30:03.479 INFO:teuthology.orchestra.run.smithi153.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1871-g3aec3903-1jammy [14.0 kB] 2024-03-04T17:30:03.480 INFO:teuthology.orchestra.run.smithi153.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1871-g3aec3903-1jammy [866 kB] 2024-03-04T17:30:03.569 INFO:teuthology.orchestra.run.smithi153.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1871-g3aec3903-1jammy [2,190 kB] 2024-03-04T17:30:03.592 INFO:teuthology.orchestra.run.smithi136.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1871-g3aec3903-1jammy [14.2 kB] 2024-03-04T17:30:03.592 INFO:teuthology.orchestra.run.smithi136.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1871-g3aec3903-1jammy [50.7 MB] 2024-03-04T17:30:03.936 INFO:teuthology.orchestra.run.smithi153.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1871-g3aec3903-1jammy [777 kB] 2024-03-04T17:30:04.063 INFO:teuthology.orchestra.run.smithi153.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1871-g3aec3903-1jammy [144 kB] 2024-03-04T17:30:04.095 INFO:teuthology.orchestra.run.smithi153.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1871-g3aec3903-1jammy [1,922 kB] 2024-03-04T17:30:04.420 INFO:teuthology.orchestra.run.smithi153.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1871-g3aec3903-1jammy [8,625 kB] 2024-03-04T17:30:05.954 INFO:teuthology.orchestra.run.smithi153.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1871-g3aec3903-1jammy [14.2 kB] 2024-03-04T17:30:05.973 INFO:teuthology.orchestra.run.smithi153.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1871-g3aec3903-1jammy [50.7 MB] 2024-03-04T17:30:10.999 INFO:teuthology.orchestra.run.smithi136.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1871-g3aec3903-1jammy [121 kB] 2024-03-04T17:30:11.001 INFO:teuthology.orchestra.run.smithi136.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1871-g3aec3903-1jammy [38.4 kB] 2024-03-04T17:30:11.002 INFO:teuthology.orchestra.run.smithi136.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1871-g3aec3903-1jammy [13.1 MB] 2024-03-04T17:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1871-g3aec3903-1jammy [91.7 kB] 2024-03-04T17:30:14.316 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 177 MB in 24s (7,407 kB/s) 2024-03-04T17:30:14.480 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-04T17:30:14.578 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165994 files and directories currently installed.) 2024-03-04T17:30:14.583 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-04T17:30:14.677 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-04T17:30:14.900 INFO:teuthology.orchestra.run.smithi153.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1871-g3aec3903-1jammy [121 kB] 2024-03-04T17:30:14.975 INFO:teuthology.orchestra.run.smithi153.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1871-g3aec3903-1jammy [38.4 kB] 2024-03-04T17:30:14.976 INFO:teuthology.orchestra.run.smithi153.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1871-g3aec3903-1jammy [13.1 MB] 2024-03-04T17:30:15.022 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-04T17:30:15.048 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-04T17:30:15.088 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-04T17:30:15.417 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-04T17:30:15.443 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-04T17:30:15.483 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-04T17:30:15.897 INFO:teuthology.orchestra.run.smithi153.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-rgw-d4n/3aec390336fdfdc8f927818b718cfb0361bd47c8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1871-g3aec3903-1jammy [91.7 kB] 2024-03-04T17:30:15.920 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-04T17:30:15.946 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-04T17:30:15.995 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-04T17:30:16.457 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 177 MB in 26s (6,685 kB/s) 2024-03-04T17:30:16.593 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-04T17:30:16.606 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-04T17:30:16.632 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-04T17:30:16.673 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-04T17:30:16.691 INFO:teuthology.orchestra.run.smithi153.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165994 files and directories currently installed.) 2024-03-04T17:30:16.696 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-04T17:30:16.766 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-04T17:30:16.993 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-04T17:30:17.019 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-04T17:30:17.045 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-04T17:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-04T17:30:17.071 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-04T17:30:17.111 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-04T17:30:17.421 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-04T17:30:17.447 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-04T17:30:17.465 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-04T17:30:17.487 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-04T17:30:17.491 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-04T17:30:17.531 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-04T17:30:17.815 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librados2. 2024-03-04T17:30:17.842 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../007-librados2_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:17.882 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librados2 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:17.926 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-04T17:30:17.952 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-04T17:30:17.994 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-04T17:30:18.318 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librbd1. 2024-03-04T17:30:18.344 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../008-librbd1_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:18.385 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librbd1 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:18.387 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-04T17:30:18.414 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-04T17:30:18.471 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-04T17:30:18.804 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libcephfs2. 2024-03-04T17:30:18.830 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:18.841 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-04T17:30:18.867 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-04T17:30:18.872 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libcephfs2 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:18.907 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-04T17:30:19.252 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-04T17:30:19.280 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-04T17:30:19.290 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rados. 2024-03-04T17:30:19.310 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-04T17:30:19.316 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:19.380 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rados (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:19.606 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librados2. 2024-03-04T17:30:19.631 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../007-librados2_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:19.671 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librados2 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:19.676 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-04T17:30:19.703 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:19.734 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-ceph-argparse (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:19.971 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cephfs. 2024-03-04T17:30:19.991 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:20.029 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cephfs (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:20.149 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librbd1. 2024-03-04T17:30:20.179 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../008-librbd1_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:20.216 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librbd1 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:20.352 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-04T17:30:20.364 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:20.406 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-ceph-common (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:20.751 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-04T17:30:20.777 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-04T17:30:20.818 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-04T17:30:20.818 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libcephfs2. 2024-03-04T17:30:20.845 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:20.886 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libcephfs2 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:21.104 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-prettytable. 2024-03-04T17:30:21.132 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rados. 2024-03-04T17:30:21.132 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-04T17:30:21.156 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:21.188 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rados (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:21.496 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-04T17:30:21.516 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-04T17:30:21.544 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:21.582 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-ceph-argparse (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:21.835 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cephfs. 2024-03-04T17:30:21.861 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:21.901 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cephfs (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:22.132 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rbd. 2024-03-04T17:30:22.159 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:22.204 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rbd (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:22.246 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-04T17:30:22.274 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:22.560 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-04T17:30:22.586 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-04T17:30:22.626 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-04T17:30:22.787 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-ceph-common (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:22.904 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-04T17:30:22.919 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-04T17:30:22.962 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-04T17:30:23.240 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-04T17:30:23.255 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-04T17:30:23.290 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-04T17:30:23.366 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-04T17:30:23.392 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-04T17:30:23.441 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-04T17:30:23.628 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package lua5.1. 2024-03-04T17:30:23.654 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-04T17:30:23.695 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-04T17:30:23.785 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-prettytable. 2024-03-04T17:30:23.812 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-04T17:30:23.852 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-04T17:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package lua-any. 2024-03-04T17:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-04T17:30:24.105 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-04T17:30:24.112 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rbd. 2024-03-04T17:30:24.128 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:24.172 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rbd (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:24.332 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package zip. 2024-03-04T17:30:24.348 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-04T17:30:24.382 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking zip (3.0-12build2) ... 2024-03-04T17:30:24.582 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-04T17:30:24.616 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-04T17:30:24.658 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-04T17:30:24.811 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package unzip. 2024-03-04T17:30:24.839 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-04T17:30:24.877 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-04T17:30:24.985 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-04T17:30:25.011 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-04T17:30:25.060 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-04T17:30:25.248 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package luarocks. 2024-03-04T17:30:25.274 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-04T17:30:25.314 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-04T17:30:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-04T17:30:25.414 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-04T17:30:25.454 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-04T17:30:25.697 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librgw2. 2024-03-04T17:30:25.724 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../025-librgw2_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:25.764 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librgw2 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:25.783 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package lua5.1. 2024-03-04T17:30:25.809 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-04T17:30:25.894 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-04T17:30:26.274 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rgw. 2024-03-04T17:30:26.289 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:26.328 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rgw (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:26.328 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package lua-any. 2024-03-04T17:30:26.354 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-04T17:30:26.402 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-04T17:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-04T17:30:26.689 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package zip. 2024-03-04T17:30:26.696 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-04T17:30:26.714 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-04T17:30:26.736 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-04T17:30:26.755 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking zip (3.0-12build2) ... 2024-03-04T17:30:27.047 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libradosstriper1. 2024-03-04T17:30:27.071 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:27.084 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package unzip. 2024-03-04T17:30:27.111 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-04T17:30:27.115 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libradosstriper1 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:27.150 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-04T17:30:27.487 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-common. 2024-03-04T17:30:27.510 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:27.542 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-common (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:27.596 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package luarocks. 2024-03-04T17:30:27.621 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-04T17:30:27.670 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-04T17:30:28.053 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librgw2. 2024-03-04T17:30:28.079 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../025-librgw2_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:28.120 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librgw2 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:28.638 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rgw. 2024-03-04T17:30:28.654 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:28.688 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rgw (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:28.855 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-base. 2024-03-04T17:30:28.882 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:28.933 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-base (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:29.025 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-04T17:30:29.051 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-04T17:30:29.091 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-04T17:30:29.427 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libradosstriper1. 2024-03-04T17:30:29.454 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:29.494 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libradosstriper1 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:29.527 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-04T17:30:29.553 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-04T17:30:29.645 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-04T17:30:29.872 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-common. 2024-03-04T17:30:29.901 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:29.939 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-common (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:30.055 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cheroot. 2024-03-04T17:30:30.083 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-04T17:30:30.121 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-04T17:30:30.449 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-04T17:30:30.477 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-04T17:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-04T17:30:30.805 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-04T17:30:30.829 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-04T17:30:30.868 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-04T17:30:31.172 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-04T17:30:31.201 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-04T17:30:31.228 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-base. 2024-03-04T17:30:31.238 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-04T17:30:31.254 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:31.303 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-base (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-tempora. 2024-03-04T17:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-04T17:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-04T17:30:31.866 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-04T17:30:31.877 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-portend. 2024-03-04T17:30:31.897 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-04T17:30:31.905 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-04T17:30:31.932 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-04T17:30:31.935 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-04T17:30:32.230 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-04T17:30:32.235 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cheroot. 2024-03-04T17:30:32.256 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-04T17:30:32.261 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-04T17:30:32.297 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-04T17:30:32.301 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-04T17:30:32.624 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-04T17:30:32.641 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-04T17:30:32.647 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-04T17:30:32.668 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-04T17:30:32.688 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-04T17:30:32.707 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-04T17:30:32.976 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-04T17:30:33.000 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-04T17:30:33.037 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-04T17:30:33.046 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-natsort. 2024-03-04T17:30:33.074 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-04T17:30:33.113 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-04T17:30:33.302 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-04T17:30:33.328 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-04T17:30:33.368 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-04T17:30:33.391 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-logutils. 2024-03-04T17:30:33.418 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-04T17:30:33.458 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-04T17:30:33.671 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-tempora. 2024-03-04T17:30:33.698 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-04T17:30:33.737 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-04T17:30:33.827 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-04T17:30:33.856 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-04T17:30:33.902 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-04T17:30:34.042 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-portend. 2024-03-04T17:30:34.069 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-04T17:30:34.104 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-04T17:30:34.247 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-mako. 2024-03-04T17:30:34.274 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-04T17:30:34.313 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-04T17:30:34.400 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-04T17:30:34.421 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-04T17:30:34.462 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-04T17:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-04T17:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-04T17:30:34.716 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-04T17:30:34.764 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-04T17:30:34.791 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-04T17:30:34.832 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-04T17:30:35.028 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-04T17:30:35.055 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-04T17:30:35.094 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-04T17:30:35.151 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-natsort. 2024-03-04T17:30:35.178 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-04T17:30:35.217 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-04T17:30:35.456 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webob. 2024-03-04T17:30:35.482 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-04T17:30:35.529 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-logutils. 2024-03-04T17:30:35.530 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-04T17:30:35.556 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-04T17:30:35.595 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-04T17:30:35.898 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-04T17:30:35.925 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-04T17:30:35.946 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-waitress. 2024-03-04T17:30:35.965 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-04T17:30:35.968 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-04T17:30:36.015 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-04T17:30:36.328 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-tempita. 2024-03-04T17:30:36.354 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-04T17:30:36.394 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-04T17:30:36.409 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-mako. 2024-03-04T17:30:36.436 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-04T17:30:36.476 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-04T17:30:36.698 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-paste. 2024-03-04T17:30:36.725 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-04T17:30:36.755 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-04T17:30:36.765 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-04T17:30:36.770 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-04T17:30:36.803 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-04T17:30:37.081 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-04T17:30:37.110 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-04T17:30:37.110 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-04T17:30:37.131 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-04T17:30:37.148 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-04T17:30:37.167 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-04T17:30:37.467 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-webob. 2024-03-04T17:30:37.494 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-04T17:30:37.495 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-04T17:30:37.522 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-04T17:30:37.534 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-04T17:30:37.562 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-04T17:30:37.874 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webtest. 2024-03-04T17:30:37.900 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-04T17:30:37.913 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-waitress. 2024-03-04T17:30:37.938 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-04T17:30:37.940 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-04T17:30:37.982 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-04T17:30:38.209 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pecan. 2024-03-04T17:30:38.225 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-04T17:30:38.259 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-04T17:30:38.289 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-tempita. 2024-03-04T17:30:38.316 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-04T17:30:38.356 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-04T17:30:38.556 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-certifi. 2024-03-04T17:30:38.585 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-04T17:30:38.614 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-04T17:30:38.626 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-paste. 2024-03-04T17:30:38.652 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-04T17:30:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-04T17:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-idna. 2024-03-04T17:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-04T17:30:39.000 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-04T17:30:39.020 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-04T17:30:39.035 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-04T17:30:39.078 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-04T17:30:39.312 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-urllib3. 2024-03-04T17:30:39.331 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-04T17:30:39.341 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-04T17:30:39.360 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-04T17:30:39.379 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-04T17:30:39.389 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-04T17:30:39.692 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-webtest. 2024-03-04T17:30:39.699 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-requests. 2024-03-04T17:30:39.725 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-04T17:30:39.726 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-04T17:30:39.760 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-04T17:30:39.765 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-04T17:30:40.053 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pecan. 2024-03-04T17:30:40.080 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-04T17:30:40.120 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-04T17:30:40.218 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-04T17:30:40.245 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-04T17:30:40.285 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-04T17:30:40.485 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-certifi. 2024-03-04T17:30:40.509 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-04T17:30:40.549 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-04T17:30:40.596 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-04T17:30:40.612 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:40.869 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-idna. 2024-03-04T17:30:40.895 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-04T17:30:40.935 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-04T17:30:41.054 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-04T17:30:41.078 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:41.117 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:41.221 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-urllib3. 2024-03-04T17:30:41.237 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-04T17:30:41.271 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-04T17:30:41.428 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-04T17:30:41.455 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-04T17:30:41.503 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-04T17:30:41.608 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-requests. 2024-03-04T17:30:41.640 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-04T17:30:41.666 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-04T17:30:41.806 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-openssl. 2024-03-04T17:30:41.836 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-04T17:30:41.873 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-04T17:30:41.952 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-04T17:30:41.979 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-04T17:30:42.019 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-04T17:30:42.272 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr. 2024-03-04T17:30:42.299 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:42.367 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:42.432 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-04T17:30:42.457 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:42.490 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:42.771 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mon. 2024-03-04T17:30:42.800 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:42.837 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mon (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:42.885 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-04T17:30:42.911 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:42.951 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-04T17:30:43.273 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-04T17:30:43.312 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-04T17:30:43.458 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-osd. 2024-03-04T17:30:43.485 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:43.524 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-osd (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:43.556 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-openssl. 2024-03-04T17:30:43.583 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-04T17:30:43.623 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-04T17:30:43.976 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr. 2024-03-04T17:30:44.004 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:44.043 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mon. 2024-03-04T17:30:44.464 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:44.505 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mon (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:44.527 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph. 2024-03-04T17:30:44.543 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../066-ceph_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:44.577 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:44.914 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-fuse. 2024-03-04T17:30:44.941 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:44.980 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-fuse (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:45.157 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-osd. 2024-03-04T17:30:45.184 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:45.224 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-osd (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:45.375 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mds. 2024-03-04T17:30:45.401 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:45.441 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mds (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:45.802 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package cephadm. 2024-03-04T17:30:45.818 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../069-cephadm_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:45.852 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking cephadm (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:46.155 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-04T17:30:46.182 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-04T17:30:46.222 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-04T17:30:46.310 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph. 2024-03-04T17:30:46.326 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../066-ceph_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:46.368 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:46.508 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-04T17:30:46.523 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-04T17:30:46.558 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-04T17:30:46.832 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-fuse. 2024-03-04T17:30:46.856 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:46.896 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-fuse (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:47.284 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mds. 2024-03-04T17:30:47.309 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:47.348 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mds (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:47.643 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-babel. 2024-03-04T17:30:47.665 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-04T17:30:47.704 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-04T17:30:47.743 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package cephadm. 2024-03-04T17:30:47.771 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../069-cephadm_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:47.810 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking cephadm (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:48.032 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jinja2. 2024-03-04T17:30:48.060 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-04T17:30:48.079 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-04T17:30:48.099 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-04T17:30:48.106 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-04T17:30:48.137 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-04T17:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-04T17:30:48.471 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:48.473 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-04T17:30:48.500 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-04T17:30:48.510 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:48.539 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-04T17:30:48.855 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-04T17:30:48.882 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-04T17:30:48.922 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-04T17:30:49.200 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-routes. 2024-03-04T17:30:49.226 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-04T17:30:49.268 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-04T17:30:49.606 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-babel. 2024-03-04T17:30:49.606 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-04T17:30:49.629 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-04T17:30:49.669 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:49.669 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-04T17:30:49.997 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jinja2. 2024-03-04T17:30:50.024 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-04T17:30:50.064 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-04T17:30:50.298 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-04T17:30:50.325 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-04T17:30:50.364 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-04T17:30:50.409 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-04T17:30:50.440 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:50.475 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-04T17:30:50.840 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-04T17:30:50.876 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-joblib. 2024-03-04T17:30:50.878 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-04T17:30:50.903 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-04T17:30:50.943 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-04T17:30:51.180 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-routes. 2024-03-04T17:30:51.208 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-04T17:30:51.248 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-04T17:30:51.271 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-04T17:30:51.298 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-04T17:30:51.329 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-04T17:30:51.558 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-04T17:30:51.586 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:51.625 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:51.625 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-sklearn. 2024-03-04T17:30:51.651 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-04T17:30:51.690 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-04T17:30:52.087 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-04T17:30:52.114 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-04T17:30:52.154 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-04T17:30:52.399 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-04T17:30:52.426 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:52.551 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:52.623 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-joblib. 2024-03-04T17:30:52.651 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-04T17:30:52.683 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-04T17:30:53.035 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-04T17:30:53.063 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-04T17:30:53.101 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-04T17:30:53.429 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-sklearn. 2024-03-04T17:30:53.456 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-04T17:30:53.499 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-04T17:30:53.522 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cachetools. 2024-03-04T17:30:53.549 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-04T17:30:53.588 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-04T17:30:53.891 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-04T17:30:53.918 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-04T17:30:53.974 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-04T17:30:54.220 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-04T17:30:54.248 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:54.311 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-04T17:30:54.338 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-04T17:30:54.377 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-04T17:30:54.380 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:54.647 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rsa. 2024-03-04T17:30:54.674 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-04T17:30:54.705 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-04T17:30:54.975 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-google-auth. 2024-03-04T17:30:55.002 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-04T17:30:55.047 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-04T17:30:55.317 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cachetools. 2024-03-04T17:30:55.333 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-04T17:30:55.362 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-04T17:30:55.375 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-04T17:30:55.390 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-04T17:30:55.428 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-04T17:30:55.645 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-04T17:30:55.673 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-04T17:30:55.736 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-04T17:30:55.748 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-websocket. 2024-03-04T17:30:55.776 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-04T17:30:55.815 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-04T17:30:56.073 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-04T17:30:56.100 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-04T17:30:56.135 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-04T17:30:56.139 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-04T17:30:56.163 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-04T17:30:56.202 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-04T17:30:56.450 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rsa. 2024-03-04T17:30:56.478 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-04T17:30:56.517 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-04T17:30:56.882 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-04T17:30:56.900 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:56.940 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:56.962 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-google-auth. 2024-03-04T17:30:56.992 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-04T17:30:57.095 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-04T17:30:57.285 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-04T17:30:57.312 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-04T17:30:57.351 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-04T17:30:57.415 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-04T17:30:57.442 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-04T17:30:57.482 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-04T17:30:57.646 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-04T17:30:57.673 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-04T17:30:57.704 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-04T17:30:57.793 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-websocket. 2024-03-04T17:30:57.820 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-04T17:30:57.859 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-04T17:30:57.974 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package jq. 2024-03-04T17:30:58.001 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-04T17:30:58.040 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-04T17:30:58.171 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-04T17:30:58.199 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-04T17:30:58.237 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-04T17:30:58.277 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package socat. 2024-03-04T17:30:58.292 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-04T17:30:58.334 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-04T17:30:58.655 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package xmlstarlet. 2024-03-04T17:30:58.682 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-04T17:30:58.721 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-04T17:30:58.937 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-04T17:30:58.963 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:30:59.001 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:59.100 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-test. 2024-03-04T17:30:59.128 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:30:59.167 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-test (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:30:59.321 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-04T17:30:59.348 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-04T17:30:59.387 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-04T17:30:59.648 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-04T17:30:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-04T17:30:59.698 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-04T17:30:59.967 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package jq. 2024-03-04T17:30:59.994 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-04T17:31:00.034 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-04T17:31:00.287 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package socat. 2024-03-04T17:31:00.314 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-04T17:31:00.353 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-04T17:31:00.666 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package xmlstarlet. 2024-03-04T17:31:00.692 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-04T17:31:00.731 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-04T17:31:01.088 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-test. 2024-03-04T17:31:01.102 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:31:01.135 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-test (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:01.546 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-volume. 2024-03-04T17:31:01.574 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:31:01.621 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-volume (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:01.917 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-04T17:31:01.944 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:31:01.983 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libcephfs-dev (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:02.336 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-04T17:31:02.363 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-04T17:31:02.402 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-04T17:31:02.690 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-04T17:31:02.706 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-04T17:31:02.740 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-04T17:31:03.116 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package nvme-cli. 2024-03-04T17:31:03.139 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-04T17:31:03.177 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-04T17:31:03.336 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-volume. 2024-03-04T17:31:03.360 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1871-g3aec3903-1jammy_all.deb ... 2024-03-04T17:31:03.397 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-volume (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:03.591 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package pkg-config. 2024-03-04T17:31:03.619 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-04T17:31:03.657 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-04T17:31:03.684 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-04T17:31:03.712 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:31:03.742 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libcephfs-dev (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:03.888 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-04T17:31:03.901 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-04T17:31:03.952 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-04T17:31:04.045 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-04T17:31:04.073 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-04T17:31:04.112 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-04T17:31:04.338 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-04T17:31:04.355 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-04T17:31:04.389 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-04T17:31:04.433 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-04T17:31:04.460 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-04T17:31:04.499 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-04T17:31:04.658 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pastescript. 2024-03-04T17:31:04.685 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-04T17:31:04.725 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-04T17:31:04.895 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package nvme-cli. 2024-03-04T17:31:04.922 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-04T17:31:04.961 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-04T17:31:05.045 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pluggy. 2024-03-04T17:31:05.072 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-04T17:31:05.111 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-04T17:31:05.333 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package pkg-config. 2024-03-04T17:31:05.361 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-04T17:31:05.399 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-04T17:31:05.423 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-psutil. 2024-03-04T17:31:05.450 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-04T17:31:05.492 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-04T17:31:05.643 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-04T17:31:05.660 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-04T17:31:05.697 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-04T17:31:05.817 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-py. 2024-03-04T17:31:05.844 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-04T17:31:05.890 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-04T17:31:06.130 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-04T17:31:06.157 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-04T17:31:06.196 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-04T17:31:06.212 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pygments. 2024-03-04T17:31:06.239 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-04T17:31:06.278 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-04T17:31:06.466 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pastescript. 2024-03-04T17:31:06.494 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-04T17:31:06.523 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-04T17:31:06.766 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-04T17:31:06.793 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-04T17:31:06.827 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pluggy. 2024-03-04T17:31:06.832 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-04T17:31:06.858 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-04T17:31:06.893 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-04T17:31:07.145 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-toml. 2024-03-04T17:31:07.170 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-04T17:31:07.205 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-psutil. 2024-03-04T17:31:07.210 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-04T17:31:07.233 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-04T17:31:07.272 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-04T17:31:07.529 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pytest. 2024-03-04T17:31:07.545 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-04T17:31:07.579 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-04T17:31:07.907 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-py. 2024-03-04T17:31:07.916 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-simplejson. 2024-03-04T17:31:07.935 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-04T17:31:07.943 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-04T17:31:07.974 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-04T17:31:07.984 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-04T17:31:08.251 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-04T17:31:08.268 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-04T17:31:08.310 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-04T17:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pygments. 2024-03-04T17:31:08.338 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-04T17:31:08.377 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-04T17:31:08.864 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-04T17:31:08.892 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-04T17:31:08.930 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-04T17:31:09.031 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package radosgw. 2024-03-04T17:31:09.058 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../116-radosgw_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:31:09.097 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking radosgw (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:09.183 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-toml. 2024-03-04T17:31:09.211 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-04T17:31:09.241 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-04T17:31:09.544 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pytest. 2024-03-04T17:31:09.575 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-04T17:31:09.611 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-04T17:31:09.845 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package rbd-fuse. 2024-03-04T17:31:09.858 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:31:09.892 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking rbd-fuse (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:09.905 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-simplejson. 2024-03-04T17:31:09.922 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-04T17:31:09.955 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-04T17:31:10.191 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-04T17:31:10.196 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-04T17:31:10.210 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-04T17:31:10.252 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-04T17:31:10.447 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-04T17:31:10.598 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-04T17:31:10.863 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-04T17:31:10.973 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package radosgw. 2024-03-04T17:31:11.000 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../116-radosgw_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:31:11.039 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking radosgw (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:11.192 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-04T17:31:11.617 INFO:teuthology.orchestra.run.smithi136.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-04T17:31:11.683 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up cephadm (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:11.824 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package rbd-fuse. 2024-03-04T17:31:11.841 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1871-g3aec3903-1jammy_amd64.deb ... 2024-03-04T17:31:11.874 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking rbd-fuse (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:12.153 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-04T17:31:12.175 INFO:teuthology.orchestra.run.smithi136.stdout:Adding system user cephadm....done 2024-03-04T17:31:12.339 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-04T17:31:12.412 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-04T17:31:12.538 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-04T17:31:12.623 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-04T17:31:12.803 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-04T17:31:12.875 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-04T17:31:13.001 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-04T17:31:13.144 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-04T17:31:13.243 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-04T17:31:13.485 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-04T17:31:13.580 INFO:teuthology.orchestra.run.smithi153.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-04T17:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-04T17:31:13.665 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up cephadm (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:13.924 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-04T17:31:14.177 INFO:teuthology.orchestra.run.smithi153.stdout:Adding system user cephadm....done 2024-03-04T17:31:14.285 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-04T17:31:14.329 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-04T17:31:14.545 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-04T17:31:14.605 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-04T17:31:14.795 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-04T17:31:14.865 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-04T17:31:14.990 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-04T17:31:15.064 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-04T17:31:15.242 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-04T17:31:15.323 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-04T17:31:15.492 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-04T17:31:15.602 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-04T17:31:15.609 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-04T17:31:15.884 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-ceph-argparse (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:15.887 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-04T17:31:16.247 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-04T17:31:16.301 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-04T17:31:16.427 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-04T17:31:16.499 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-04T17:31:16.528 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-04T17:31:16.631 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-04T17:31:16.783 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-04T17:31:16.880 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-04T17:31:17.009 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-04T17:31:17.260 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-04T17:31:17.271 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-04T17:31:17.318 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-04T17:31:17.383 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-04T17:31:17.503 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-04T17:31:17.508 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-04T17:31:17.754 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-ceph-argparse (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:17.868 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-04T17:31:17.969 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-04T17:31:18.105 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-04T17:31:18.220 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-04T17:31:18.231 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-04T17:31:18.332 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-04T17:31:18.433 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-04T17:31:18.488 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-04T17:31:18.684 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-04T17:31:18.789 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up zip (3.0-12build2) ... 2024-03-04T17:31:18.915 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-04T17:31:19.055 INFO:teuthology.orchestra.run.smithi153.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-04T17:31:19.092 INFO:teuthology.orchestra.run.smithi153.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-04T17:31:19.170 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-04T17:31:19.297 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-04T17:31:19.550 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-04T17:31:19.656 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-04T17:31:19.760 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-04T17:31:19.826 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-04T17:31:19.927 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-04T17:31:20.024 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-04T17:31:20.046 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-04T17:31:20.284 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-04T17:31:20.329 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-04T17:31:20.593 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up zip (3.0-12build2) ... 2024-03-04T17:31:20.703 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-04T17:31:20.724 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-04T17:31:20.989 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-04T17:31:21.366 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-04T17:31:21.371 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-04T17:31:21.614 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-04T17:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-04T17:31:21.745 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-04T17:31:21.867 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-04T17:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-04T17:31:22.159 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-04T17:31:22.244 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-04T17:31:22.370 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-ceph-common (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:22.526 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-04T17:31:22.671 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-idna (3.3-1) ... 2024-03-04T17:31:22.786 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-04T17:31:22.989 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-04T17:31:23.171 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-04T17:31:23.456 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-04T17:31:23.807 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-04T17:31:24.058 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-04T17:31:24.184 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-ceph-common (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:24.194 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-04T17:31:24.429 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-04T17:31:24.502 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-idna (3.3-1) ... 2024-03-04T17:31:24.564 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-04T17:31:24.803 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-04T17:31:24.839 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-04T17:31:24.965 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-04T17:31:25.091 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-04T17:31:25.393 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-04T17:31:25.627 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-04T17:31:25.853 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-04T17:31:25.951 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-04T17:31:25.996 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-04T17:31:26.185 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-04T17:31:26.288 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-04T17:31:26.320 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-04T17:31:26.539 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-04T17:31:26.579 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-04T17:31:26.680 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-04T17:31:26.781 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-04T17:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-04T17:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-04T17:31:27.066 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-04T17:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-04T17:31:27.317 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-04T17:31:27.543 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-04T17:31:27.576 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-04T17:31:27.661 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-04T17:31:27.677 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-04T17:31:27.937 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-04T17:31:27.989 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-04T17:31:28.064 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-04T17:31:28.364 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-04T17:31:28.446 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-04T17:31:28.665 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-04T17:31:28.698 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-04T17:31:28.925 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-04T17:31:28.957 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-04T17:31:29.026 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-04T17:31:29.129 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-04T17:31:29.198 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-04T17:31:29.208 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-04T17:31:29.436 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-04T17:31:29.475 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-04T17:31:29.562 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-04T17:31:29.607 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-04T17:31:29.888 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-04T17:31:29.955 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-04T17:31:30.012 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-04T17:31:30.217 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-04T17:31:30.296 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-04T17:31:30.341 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-04T17:31:30.581 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-04T17:31:30.592 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-04T17:31:30.693 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-04T17:31:30.815 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-04T17:31:30.942 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-04T17:31:30.985 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-04T17:31:31.050 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-04T17:31:31.132 INFO:teuthology.orchestra.run.smithi153.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-04T17:31:31.286 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-04T17:31:31.369 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-04T17:31:31.504 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-04T17:31:31.576 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-04T17:31:31.697 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-04T17:31:31.930 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-04T17:31:31.974 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-04T17:31:32.189 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-04T17:31:32.249 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-04T17:31:32.315 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-04T17:31:32.574 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-04T17:31:32.585 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-04T17:31:32.700 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-04T17:31:32.995 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-04T17:31:33.035 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-04T17:31:33.291 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-04T17:31:33.344 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-04T17:31:33.637 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-04T17:31:33.640 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-04T17:31:33.739 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-04T17:31:33.766 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librados2 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:33.917 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:33.997 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-04T17:31:34.019 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-04T17:31:34.257 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-04T17:31:34.338 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libcephfs2 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:34.469 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libradosstriper1 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:34.590 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-04T17:31:34.618 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-04T17:31:34.874 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librbd1 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:35.001 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-modules-core (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:35.012 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-04T17:31:35.127 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-fuse (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:35.289 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-04T17:31:35.347 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-04T17:31:35.347 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-04T17:31:35.615 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-04T17:31:35.767 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librados2 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:35.893 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:35.895 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libcephfs-dev (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:36.019 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-04T17:31:36.021 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rados (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:36.152 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librgw2 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:36.355 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libcephfs2 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:36.456 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libradosstriper1 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:36.566 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-04T17:31:36.673 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-04T17:31:36.842 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librbd1 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:36.951 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-modules-core (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:37.060 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-fuse (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:37.273 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-04T17:31:37.274 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-04T17:31:37.761 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libcephfs-dev (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:37.884 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rbd (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:37.887 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rados (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:37.988 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librgw2 (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:37.993 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up rbd-fuse (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:38.089 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-04T17:31:38.103 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rgw (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:38.237 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cephfs (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:38.363 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-common (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:38.907 INFO:teuthology.orchestra.run.smithi136.stdout:Adding group ceph....done 2024-03-04T17:31:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rbd (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:39.341 INFO:teuthology.orchestra.run.smithi136.stdout:Adding system user ceph....done 2024-03-04T17:31:39.410 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up rbd-fuse (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:39.502 INFO:teuthology.orchestra.run.smithi136.stdout:Setting system user ceph properties....done 2024-03-04T17:31:39.515 INFO:teuthology.orchestra.run.smithi136.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-04T17:31:39.545 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rgw (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:39.653 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-04T17:31:39.699 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cephfs (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:39.822 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-common (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:39.993 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-04T17:31:40.390 INFO:teuthology.orchestra.run.smithi153.stdout:Adding group ceph....done 2024-03-04T17:31:40.480 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-test (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:40.682 INFO:teuthology.orchestra.run.smithi153.stdout:Adding system user ceph....done 2024-03-04T17:31:40.843 INFO:teuthology.orchestra.run.smithi153.stdout:Setting system user ceph properties....done 2024-03-04T17:31:40.855 INFO:teuthology.orchestra.run.smithi153.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-04T17:31:40.903 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up radosgw (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:40.988 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-04T17:31:41.301 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-04T17:31:41.398 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-04T17:31:41.398 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-04T17:31:41.765 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-test (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:41.890 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up radosgw (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:41.918 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-base (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:42.240 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-04T17:31:42.372 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-04T17:31:42.372 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-04T17:31:42.703 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mds (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:42.867 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-base (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:42.944 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-04T17:31:42.944 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-04T17:31:43.217 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-04T17:31:43.413 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:43.629 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-04T17:31:43.630 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-04T17:31:43.719 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mds (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:43.943 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-04T17:31:43.943 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-04T17:31:44.097 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-osd (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:44.379 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-04T17:31:44.379 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-04T17:31:44.463 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:44.659 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-04T17:31:44.659 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-04T17:31:44.841 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:44.966 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:45.130 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mon (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:45.173 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-osd (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:45.348 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-04T17:31:45.349 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-04T17:31:45.452 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-04T17:31:45.452 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-04T17:31:45.835 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-cephadm (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:45.916 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:46.042 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-dashboard (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:46.193 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mon (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:46.339 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-volume (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:46.407 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-04T17:31:46.407 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-04T17:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-04T17:31:46.818 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-04T17:31:46.902 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-cephadm (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:46.987 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-04T17:31:47.045 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:47.171 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-dashboard (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:47.341 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-volume (19.0.0-1871-g3aec3903-1jammy) ... 2024-03-04T17:31:47.700 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-04T17:31:47.892 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-04T17:31:48.111 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-04T17:31:48.129 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-04T17:31:48.795 INFO:teuthology.orchestra.run.smithi136.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-04T17:31:48.798 DEBUG:teuthology.parallel:result is None 2024-03-04T17:31:49.196 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-04T17:31:49.851 INFO:teuthology.orchestra.run.smithi153.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-04T17:31:49.856 DEBUG:teuthology.parallel:result is None 2024-03-04T17:31:49.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3aec390336fdfdc8f927818b718cfb0361bd47c8 2024-03-04T17:31:50.088 DEBUG:teuthology.orchestra.run.smithi136:> dpkg-query -W -f '${Version}' ceph 2024-03-04T17:31:50.107 INFO:teuthology.orchestra.run.smithi136.stdout:19.0.0-1871-g3aec3903-1jammy 2024-03-04T17:31:50.107 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1871-g3aec3903-1jammy 2024-03-04T17:31:50.107 INFO:teuthology.task.install:The correct ceph version 19.0.0-1871-g3aec3903-1jammy is installed. 2024-03-04T17:31:50.110 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3aec390336fdfdc8f927818b718cfb0361bd47c8 2024-03-04T17:31:50.334 DEBUG:teuthology.orchestra.run.smithi153:> dpkg-query -W -f '${Version}' ceph 2024-03-04T17:31:50.355 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-1871-g3aec3903-1jammy 2024-03-04T17:31:50.355 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1871-g3aec3903-1jammy 2024-03-04T17:31:50.355 INFO:teuthology.task.install:The correct ceph version 19.0.0-1871-g3aec3903-1jammy is installed. 2024-03-04T17:31:50.358 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-04T17:31:50.376 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:31:50.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T17:31:50.413 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:31:50.413 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T17:31:50.427 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-04T17:31:50.427 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:31:50.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T17:31:50.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T17:31:50.523 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:31:50.523 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T17:31:50.537 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T17:31:50.591 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-04T17:31:50.591 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:31:50.591 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T17:31:50.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T17:31:50.659 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:31:50.659 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T17:31:50.713 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T17:31:50.766 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-04T17:31:50.767 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:31:50.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T17:31:50.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T17:31:50.835 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:31:50.835 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T17:31:50.848 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T17:31:50.904 INFO:teuthology.run_tasks:Running task ceph... 2024-03-04T17:31:51.049 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-04T17:31:51.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /var/log/ceph 2024-03-04T17:31:51.052 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2024-03-04T17:31:51.067 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-04T17:31:51.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-04T17:31:51.109 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-04T17:31:51.122 INFO:tasks.ceph:Creating extra log directories... 2024-03-04T17:31:51.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-04T17:31:51.165 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-04T17:31:51.200 INFO:tasks.ceph:Creating ceph cluster c1... 2024-03-04T17:31:51.200 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-04T17:31:51.200 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi/7581342', 'branch': 'wip-rgw-d4n', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7581342', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-rgw-d4n'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-rgw-d4n', 'sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8'}}, 'owner': 'scheduled_suriarte@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-rgw-d4n', 'suite_path': '/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/pritha-srivastava/ceph.git', 'suite_sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8', 'targets': {'smithi136.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAqHjmXINRGS++vEJkLSuQVoCDLD7Q/OTh0RAS4ORRa34gLI/XifPQGnY9NX/bUvqJ8L9VxxHFw5XRlWZkskD6M=', 'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM5+Lf8d9jkIUZDTAHuoCGKmJtNNK7KduqKFetIFOBuUOXs7F9wANcTWVBomJoQHbfTA4oYoHd9PFqtZins8FCs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8'}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-03-04_17:13:34', 'tube': 'smithi', 'user': 'suriarte', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-04T17:31:51.201 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-03-04T17:31:51.222 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-03-04T17:31:51.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T17:31:51.273 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T17:31:51.294 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:31:51.294 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T17:31:51.328 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T17:31:51.328 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-03-04T17:31:51.377 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T17:31:51.377 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T17:31:51.377 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-03-04T17:31:51.377 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T17:31:51.377 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-04 17:26:17.673294431 +0000 2024-03-04T17:31:51.377 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-04 17:26:17.413296753 +0000 2024-03-04T17:31:51.377 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-04 17:26:17.413296753 +0000 2024-03-04T17:31:51.377 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-03-04T17:31:51.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T17:31:51.431 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-04T17:31:51.431 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-04T17:31:51.431 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.00045532 s, 1.1 MB/s 2024-03-04T17:31:51.432 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T17:31:51.485 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-03-04T17:31:51.532 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T17:31:51.532 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T17:31:51.532 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-03-04T17:31:51.532 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T17:31:51.533 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-04 17:26:18.189289822 +0000 2024-03-04T17:31:51.533 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-04 17:26:17.897292430 +0000 2024-03-04T17:31:51.533 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-04 17:26:17.897292430 +0000 2024-03-04T17:31:51.533 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-03-04T17:31:51.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T17:31:51.587 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-04T17:31:51.587 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-04T17:31:51.588 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000433971 s, 1.2 MB/s 2024-03-04T17:31:51.588 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T17:31:51.639 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-03-04T17:31:51.690 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T17:31:51.690 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T17:31:51.690 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 920 Links: 1 2024-03-04T17:31:51.690 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T17:31:51.690 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-04 17:26:18.653285677 +0000 2024-03-04T17:31:51.690 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-04 17:26:18.421287749 +0000 2024-03-04T17:31:51.690 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-04 17:26:18.421287749 +0000 2024-03-04T17:31:51.690 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-03-04T17:31:51.691 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T17:31:51.742 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-04T17:31:51.743 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-04T17:31:51.743 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000495976 s, 1.0 MB/s 2024-03-04T17:31:51.744 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T17:31:51.797 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-03-04T17:31:51.845 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T17:31:51.845 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T17:31:51.845 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 933 Links: 1 2024-03-04T17:31:51.845 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T17:31:51.846 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-04 17:26:19.177280997 +0000 2024-03-04T17:31:51.846 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-04 17:26:18.905283426 +0000 2024-03-04T17:31:51.846 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-04 17:26:18.905283426 +0000 2024-03-04T17:31:51.846 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-03-04T17:31:51.846 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T17:31:51.897 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-04T17:31:51.898 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-04T17:31:51.898 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.00031918 s, 1.6 MB/s 2024-03-04T17:31:51.899 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T17:31:51.949 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T17:31:51.949 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-04T17:31:51.949 INFO:tasks.ceph:Generating config... 2024-03-04T17:31:51.950 INFO:tasks.ceph:[client] debug rgw = 20 2024-03-04T17:31:51.950 INFO:tasks.ceph:[client] debug rgw lifecycle = 20 2024-03-04T17:31:51.950 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-03-04T17:31:51.950 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-03-04T17:31:51.950 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-03-04T17:31:51.951 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-03-04T17:31:51.951 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-03-04T17:31:51.951 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-03-04T17:31:51.951 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-03-04T17:31:51.951 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-03-04T17:31:51.951 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-03-04T17:31:51.951 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-03-04T17:31:51.951 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-03-04T17:31:51.951 INFO:tasks.ceph:[client] setgroup = ceph 2024-03-04T17:31:51.951 INFO:tasks.ceph:[client] setuser = ceph 2024-03-04T17:31:51.951 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-03-04T17:31:51.951 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-04T17:31:51.951 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-04T17:31:51.951 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-04T17:31:51.952 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-04T17:31:51.952 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-04T17:31:51.952 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-04T17:31:51.952 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-04T17:31:51.952 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-03-04T17:31:51.952 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-03-04T17:31:51.952 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-03-04T17:31:51.952 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-03-04T17:31:51.954 INFO:tasks.ceph:Setting up c1.mon.a... 2024-03-04T17:31:51.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2024-03-04T17:31:52.024 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/c1.keyring 2024-03-04T17:31:52.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2024-03-04T17:31:52.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-03-04T17:31:52.179 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.136')] 2024-03-04T17:31:52.179 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.136'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-03-04T17:31:52.180 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:31:52.180 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T17:31:52.229 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.136 --print /home/ubuntu/cephtest/c1.monmap 2024-03-04T17:31:52.312 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:31:52.313 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:31:52.314 INFO:teuthology.orchestra.run.smithi136.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2024-03-04T17:31:52.314 INFO:teuthology.orchestra.run.smithi136.stdout:monmaptool: generated fsid 3b7fafa4-847c-4367-bdf5-f9f1b7e3e746 2024-03-04T17:31:52.314 INFO:teuthology.orchestra.run.smithi136.stdout:setting min_mon_release = quincy 2024-03-04T17:31:52.314 INFO:teuthology.orchestra.run.smithi136.stdout:epoch 0 2024-03-04T17:31:52.314 INFO:teuthology.orchestra.run.smithi136.stdout:fsid 3b7fafa4-847c-4367-bdf5-f9f1b7e3e746 2024-03-04T17:31:52.314 INFO:teuthology.orchestra.run.smithi136.stdout:last_changed 2024-03-04T17:31:52.308312+0000 2024-03-04T17:31:52.314 INFO:teuthology.orchestra.run.smithi136.stdout:created 2024-03-04T17:31:52.308312+0000 2024-03-04T17:31:52.314 INFO:teuthology.orchestra.run.smithi136.stdout:min_mon_release 17 (quincy) 2024-03-04T17:31:52.314 INFO:teuthology.orchestra.run.smithi136.stdout:election_strategy: 1 2024-03-04T17:31:52.314 INFO:teuthology.orchestra.run.smithi136.stdout:0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-03-04T17:31:52.314 INFO:teuthology.orchestra.run.smithi136.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2024-03-04T17:31:52.315 DEBUG:teuthology.orchestra.run.smithi136:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T17:31:52.362 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 3b7fafa4-847c-4367-bdf5-f9f1b7e3e746... 2024-03-04T17:31:52.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-03-04T17:31:52.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-03-04T17:31:52.433 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2024-03-04T17:31:52.433 INFO:teuthology.orchestra.run.smithi136.stdout: chdir = "" 2024-03-04T17:31:52.433 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T17:31:52.433 INFO:teuthology.orchestra.run.smithi136.stdout: auth supported = cephx 2024-03-04T17:31:52.433 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: filestore xattr use omap = true 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: mon clock drift allowed = 1.000 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: osd crush chooseleaf type = 0 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: auth debug = true 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on old message = true 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on bug = true 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: mon pg warn max object skew = 0 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T17:31:52.434 INFO:teuthology.orchestra.run.smithi136.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default size = 2 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow primary affinity = true 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow pg remap = true 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on legacy crush tunables = false 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on crush straw calc version zero = false 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on no sortbitwise = false 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on osd down out interval zero = false 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on too few osds = false 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: mon_allow_pool_size_one = true 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.435 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: osd default data pool replay window = 5 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: mon allow pool delete = true 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: mon cluster log file level = debug 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: debug asserts on shutdown = true 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: mon health detail to clog = false 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: mon host = 172.21.15.136 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = 3b7fafa4-847c-4367-bdf5-f9f1b7e3e746 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout:[osd] 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: osd journal size = 100 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.436 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub load threshold = 5.0 2024-03-04T17:31:52.437 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub max interval = 600 2024-03-04T17:31:52.437 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock profile = high_recovery_ops 2024-03-04T17:31:52.437 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.437 INFO:teuthology.orchestra.run.smithi136.stdout: osd recover clone overlap = true 2024-03-04T17:31:52.437 INFO:teuthology.orchestra.run.smithi136.stdout: osd recovery max chunk = 1048576 2024-03-04T17:31:52.437 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.437 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug shutdown = true 2024-03-04T17:31:52.437 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug op order = true 2024-03-04T17:31:52.437 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify stray on activate = true 2024-03-04T17:31:52.437 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug trim objects = true 2024-03-04T17:31:52.437 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.437 INFO:teuthology.orchestra.run.smithi136.stdout: osd open classes on start = true 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug pg log writeout = true 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: osd deep scrub update digest min age = 30 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: osd map max advance = 10 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: journal zero on create = true 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: filestore ondisk finisher threads = 3 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: filestore apply finisher threads = 3 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: bdev debug aio = true 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug misdirected ops = true 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-04T17:31:52.438 INFO:teuthology.orchestra.run.smithi136.stdout: debug osd = 20 2024-03-04T17:31:52.439 INFO:teuthology.orchestra.run.smithi136.stdout: rocksdb delete range threshold = 0 2024-03-04T17:31:52.439 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.439 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr] 2024-03-04T17:31:52.439 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-04T17:31:52.439 INFO:teuthology.orchestra.run.smithi136.stdout: debug mgr = 20 2024-03-04T17:31:52.439 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-03-04T17:31:52.439 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-03-04T17:31:52.439 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-03-04T17:31:52.439 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-03-04T17:31:52.439 INFO:teuthology.orchestra.run.smithi136.stdout: mgr/telemetry/nag = false 2024-03-04T17:31:52.439 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.439 INFO:teuthology.orchestra.run.smithi136.stdout:[mon] 2024-03-04T17:31:52.440 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-04T17:31:52.440 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-03-04T17:31:52.440 INFO:teuthology.orchestra.run.smithi136.stdout: debug paxos = 20 2024-03-04T17:31:52.440 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-03-04T17:31:52.440 INFO:teuthology.orchestra.run.smithi136.stdout: mon data avail warn = 5 2024-03-04T17:31:52.440 INFO:teuthology.orchestra.run.smithi136.stdout: mon mgr mkfs grace = 240 2024-03-04T17:31:52.440 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-03-04T17:31:52.440 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd reporter subtree level = osd 2024-03-04T17:31:52.441 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd prime pg temp = true 2024-03-04T17:31:52.441 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-03-04T17:31:52.441 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.441 INFO:teuthology.orchestra.run.smithi136.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T17:31:52.441 INFO:teuthology.orchestra.run.smithi136.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T17:31:52.441 INFO:teuthology.orchestra.run.smithi136.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T17:31:52.441 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.441 INFO:teuthology.orchestra.run.smithi136.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T17:31:52.441 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T17:31:52.441 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T17:31:52.441 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.442 INFO:teuthology.orchestra.run.smithi136.stdout: # 1m isn't quite enough 2024-03-04T17:31:52.442 INFO:teuthology.orchestra.run.smithi136.stdout: mon_down_mkfs_grace = 2m 2024-03-04T17:31:52.442 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.442 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_filestore_osds = false 2024-03-04T17:31:52.442 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:31:52.442 INFO:teuthology.orchestra.run.smithi136.stdout:[client] 2024-03-04T17:31:52.442 INFO:teuthology.orchestra.run.smithi136.stdout: rgw cache enabled = true 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable ops log = true 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable usage log = true 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: debug rgw = 20 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: debug rgw lifecycle = 20 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: debug rgw sync = 20 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: rgw crypt require ssl = False 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: rgw crypt s3 kms backend = testing 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: rgw curl low speed time = 300 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: rgw data log num shards = 4 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: rgw md log max shards = 4 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: rgw sync data inject err probability = 0.1 2024-03-04T17:31:52.443 INFO:teuthology.orchestra.run.smithi136.stdout: rgw sync log trim interval = 0 2024-03-04T17:31:52.444 INFO:teuthology.orchestra.run.smithi136.stdout: rgw sync meta inject err probability = 0.1 2024-03-04T17:31:52.444 INFO:teuthology.orchestra.run.smithi136.stdout: rgw sync obj etag verify = True 2024-03-04T17:31:52.444 INFO:teuthology.orchestra.run.smithi136.stdout: setgroup = ceph 2024-03-04T17:31:52.444 INFO:teuthology.orchestra.run.smithi136.stdout: setuser = ceph 2024-03-04T17:31:52.444 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.a] 2024-03-04T17:31:52.444 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2024-03-04T17:31:52.444 INFO:teuthology.orchestra.run.smithi136.stdout: rgw data notify interval msec = 0 2024-03-04T17:31:52.444 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.0] 2024-03-04T17:31:52.444 INFO:teuthology.orchestra.run.smithi136.stdout: osd_max_omap_entries_per_request = 10 2024-03-04T17:31:52.444 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.1] 2024-03-04T17:31:52.444 INFO:teuthology.orchestra.run.smithi136.stdout: osd_max_omap_entries_per_request = 1000 2024-03-04T17:31:52.444 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.2] 2024-03-04T17:31:52.460 INFO:teuthology.orchestra.run.smithi136.stdout: osd_max_omap_entries_per_request = 10000 2024-03-04T17:31:52.461 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-03-04T17:31:52.461 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2024-03-04T17:31:52.461 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T17:31:52.461 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2024-03-04T17:31:52.461 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T17:31:52.462 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.463 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = 172.21.15.136 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 3b7fafa4-847c-4367-bdf5-f9f1b7e3e746 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2024-03-04T17:31:52.464 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug trim objects = true 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: rocksdb delete range threshold = 0 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.466 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2024-03-04T17:31:52.467 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2024-03-04T17:31:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: debug rgw = 20 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: debug rgw lifecycle = 20 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: debug rgw sync = 20 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: rgw crypt require ssl = False 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: rgw crypt s3 kms backend = testing 2024-03-04T17:31:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-03-04T17:31:52.470 INFO:teuthology.orchestra.run.smithi153.stdout: rgw curl low speed time = 300 2024-03-04T17:31:52.470 INFO:teuthology.orchestra.run.smithi153.stdout: rgw data log num shards = 4 2024-03-04T17:31:52.471 INFO:teuthology.orchestra.run.smithi153.stdout: rgw md log max shards = 4 2024-03-04T17:31:52.471 INFO:teuthology.orchestra.run.smithi153.stdout: rgw sync data inject err probability = 0.1 2024-03-04T17:31:52.471 INFO:teuthology.orchestra.run.smithi153.stdout: rgw sync log trim interval = 0 2024-03-04T17:31:52.471 INFO:teuthology.orchestra.run.smithi153.stdout: rgw sync meta inject err probability = 0.1 2024-03-04T17:31:52.471 INFO:teuthology.orchestra.run.smithi153.stdout: rgw sync obj etag verify = True 2024-03-04T17:31:52.471 INFO:teuthology.orchestra.run.smithi153.stdout: setgroup = ceph 2024-03-04T17:31:52.471 INFO:teuthology.orchestra.run.smithi153.stdout: setuser = ceph 2024-03-04T17:31:52.471 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2024-03-04T17:31:52.471 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-03-04T17:31:52.472 INFO:teuthology.orchestra.run.smithi153.stdout: rgw data notify interval msec = 0 2024-03-04T17:31:52.472 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2024-03-04T17:31:52.472 INFO:teuthology.orchestra.run.smithi153.stdout: osd_max_omap_entries_per_request = 10 2024-03-04T17:31:52.472 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2024-03-04T17:31:52.472 INFO:teuthology.orchestra.run.smithi153.stdout: osd_max_omap_entries_per_request = 1000 2024-03-04T17:31:52.472 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2024-03-04T17:31:52.472 INFO:teuthology.orchestra.run.smithi153.stdout: osd_max_omap_entries_per_request = 10000 2024-03-04T17:31:52.472 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2024-03-04T17:31:52.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2024-03-04T17:31:52.551 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-04T17:31:52.551 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:31:52.551 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2024-03-04T17:31:52.556 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:31:52.556 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2024-03-04T17:31:52.604 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi136.front.sepia.ceph.com 2024-03-04T17:31:52.605 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:31:52.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/c1.keyring 2024-03-04T17:31:52.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-03-04T17:31:52.669 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:31:52.669 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-03-04T17:31:52.717 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2024-03-04T17:31:52.717 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:31:52.717 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/c1.keyring 2024-03-04T17:31:52.717 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-03-04T17:31:52.742 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:31:52.742 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-03-04T17:31:52.787 INFO:tasks.ceph:Setting up mon nodes... 2024-03-04T17:31:52.787 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-04T17:31:52.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2024-03-04T17:31:52.830 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2024-03-04T17:31:52.833 INFO:tasks.ceph:Setting up mds nodes... 2024-03-04T17:31:52.833 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-04T17:31:52.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2024-03-04T17:31:52.912 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/c1.client.0.keyring 2024-03-04T17:31:52.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2024-03-04T17:31:52.999 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/c1.client.1.keyring 2024-03-04T17:31:53.008 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-04T17:31:53.009 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-04T17:31:53.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2024-03-04T17:31:53.064 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T17:31:53.064 INFO:tasks.ceph:role: c1.osd.0 2024-03-04T17:31:53.070 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-04T17:31:53.070 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-04T17:31:53.130 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T17:31:53.130 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T17:31:53.130 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T17:31:53.130 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-04T17:31:53.130 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T17:31:53.130 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-04T17:31:53.131 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T17:31:53.131 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-04T17:31:53.131 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T17:31:53.131 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T17:31:53.639 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-04T17:31:53.641 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-04T17:31:53.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2024-03-04T17:31:53.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2024-03-04T17:31:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T17:31:53.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2024-03-04T17:31:53.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2024-03-04T17:31:53.815 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T17:31:53.816 INFO:tasks.ceph:role: c1.osd.1 2024-03-04T17:31:53.816 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-04T17:31:53.816 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-04T17:31:53.884 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T17:31:53.884 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T17:31:53.884 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T17:31:53.884 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-04T17:31:53.885 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T17:31:53.885 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-04T17:31:53.885 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T17:31:53.885 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-04T17:31:53.885 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T17:31:53.885 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T17:31:54.329 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-04T17:31:54.331 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-04T17:31:54.331 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2024-03-04T17:31:54.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2024-03-04T17:31:54.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T17:31:54.432 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2024-03-04T17:31:54.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2024-03-04T17:31:54.487 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T17:31:54.487 INFO:tasks.ceph:role: c1.osd.2 2024-03-04T17:31:54.487 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-04T17:31:54.487 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-04T17:31:54.551 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T17:31:54.551 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T17:31:54.551 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T17:31:54.551 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-04T17:31:54.551 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T17:31:54.551 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-04T17:31:54.551 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T17:31:54.552 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-04T17:31:54.552 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T17:31:54.552 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T17:31:55.037 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-04T17:31:55.039 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-04T17:31:55.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2024-03-04T17:31:55.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2024-03-04T17:31:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T17:31:55.147 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2024-03-04T17:31:55.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2024-03-04T17:31:55.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:55.257+0000 7fdcc21547c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2024-03-04T17:31:55.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:55.261+0000 7fdcc21547c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2024-03-04T17:31:55.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:55.261+0000 7fdcc21547c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-03-04T17:31:55.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:55.261+0000 7fdcc21547c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-03-04T17:31:55.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:55.261+0000 7fdcc21547c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-03-04T17:31:55.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:55.265+0000 7fdcc21547c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2024-03-04T17:31:57.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2024-03-04T17:31:57.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2024-03-04T17:31:57.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:57.569+0000 7f6918a297c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2024-03-04T17:31:57.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:57.569+0000 7f6918a297c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2024-03-04T17:31:57.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:57.569+0000 7f6918a297c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-03-04T17:31:57.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:57.569+0000 7f6918a297c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-03-04T17:31:57.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:57.569+0000 7f6918a297c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-03-04T17:31:57.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:57.569+0000 7f6918a297c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2024-03-04T17:31:59.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2024-03-04T17:31:59.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2024-03-04T17:31:59.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:59.873+0000 7fcecf79d7c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2024-03-04T17:31:59.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:59.877+0000 7fcecf79d7c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2024-03-04T17:31:59.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:59.877+0000 7fcecf79d7c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-03-04T17:31:59.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:59.877+0000 7fcecf79d7c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-03-04T17:31:59.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:59.877+0000 7fcecf79d7c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-03-04T17:31:59.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:31:59.877+0000 7fcecf79d7c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2024-03-04T17:32:02.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2024-03-04T17:32:02.139 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-04T17:32:02.140 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:32:02.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2024-03-04T17:32:02.195 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:32:02.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2024-03-04T17:32:02.251 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:32:02.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2024-03-04T17:32:02.308 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:32:02.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2024-03-04T17:32:02.363 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:32:02.364 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2024-03-04T17:32:02.412 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:32:02.412 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2024-03-04T17:32:02.460 INFO:tasks.ceph:Adding keys to all mons... 2024-03-04T17:32:02.460 DEBUG:teuthology.orchestra.run.smithi136:> sudo tee -a /etc/ceph/c1.keyring 2024-03-04T17:32:02.514 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.x] 2024-03-04T17:32:02.514 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCIBeZlIWSRMRAAODCY976DARpXOw9OHWHOpw== 2024-03-04T17:32:02.514 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.0] 2024-03-04T17:32:02.514 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCLBeZlxJ2ZDxAA8PALHe8IKO/QvdfBezOPUg== 2024-03-04T17:32:02.515 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.1] 2024-03-04T17:32:02.515 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCNBeZlgkxBIhAAb/PGtmV1SVG8Gh/26+HHuw== 2024-03-04T17:32:02.515 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.2] 2024-03-04T17:32:02.515 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCPBeZlDI58NBAAO3TZu4nEmw153RVkmkZOjQ== 2024-03-04T17:32:02.515 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2024-03-04T17:32:02.515 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCIBeZlYWp0NhAAKyjVIqaG0LXPfki/eTViyg== 2024-03-04T17:32:02.515 INFO:teuthology.orchestra.run.smithi136.stdout:[client.1] 2024-03-04T17:32:02.515 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCIBeZlzeGZOxAABoWkzlXSugs27aMy14icpA== 2024-03-04T17:32:02.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-04T17:32:02.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T17:32:02.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T17:32:02.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T17:32:02.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T17:32:02.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T17:32:03.009 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-04T17:32:03.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2024-03-04T17:32:03.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2024-03-04T17:32:03.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2024-03-04T17:32:03.125 DEBUG:teuthology.orchestra.run.smithi136:> rm -- /home/ubuntu/cephtest/c1.monmap 2024-03-04T17:32:03.171 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2024-03-04T17:32:03.171 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2024-03-04T17:32:03.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2024-03-04T17:32:03.217 INFO:tasks.ceph.c1.mon.a:Started 2024-03-04T17:32:03.217 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2024-03-04T17:32:03.217 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2024-03-04T17:32:03.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2024-03-04T17:32:03.219 INFO:tasks.ceph.c1.mgr.x:Started 2024-03-04T17:32:03.219 INFO:tasks.ceph:Setting crush tunables to default 2024-03-04T17:32:03.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster c1 osd crush tunables default 2024-03-04T17:32:03.455 INFO:teuthology.orchestra.run.smithi136.stderr:adjusted tunables profile to default 2024-03-04T17:32:03.477 INFO:tasks.ceph:check_enable_crimson: False 2024-03-04T17:32:03.477 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2024-03-04T17:32:03.478 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:32:03.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2024-03-04T17:32:03.491 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:32:03.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2024-03-04T17:32:03.546 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:32:03.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2024-03-04T17:32:03.583 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:03.577+0000 7f5080f53180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T17:32:03.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster c1 osd new a8c506a7-b951-4547-a5ef-91c2ce404f9e 0 2024-03-04T17:32:03.840 INFO:teuthology.orchestra.run.smithi136.stdout:0 2024-03-04T17:32:03.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster c1 osd new 4f082249-8fc1-4369-a950-a16a958f1a43 1 2024-03-04T17:32:04.043 INFO:teuthology.orchestra.run.smithi136.stdout:1 2024-03-04T17:32:04.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster c1 osd new 9f2c9da7-9067-4110-add9-194db55a16f3 2 2024-03-04T17:32:04.093 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:04.089+0000 7f5080f53180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T17:32:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:2 2024-03-04T17:32:04.267 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2024-03-04T17:32:04.267 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2024-03-04T17:32:04.302 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:04.297+0000 7f5080f53180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T17:32:04.304 INFO:tasks.ceph.c1.osd.0:Started 2024-03-04T17:32:04.304 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2024-03-04T17:32:04.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2024-03-04T17:32:04.306 INFO:tasks.ceph.c1.osd.1:Started 2024-03-04T17:32:04.306 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2024-03-04T17:32:04.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2024-03-04T17:32:04.308 INFO:tasks.ceph.c1.osd.2:Started 2024-03-04T17:32:04.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-03-04T17:32:04.485 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:04.486 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":5,"fsid":"3b7fafa4-847c-4367-bdf5-f9f1b7e3e746","created":"2024-03-04T17:32:03.338712+0000","modified":"2024-03-04T17:32:04.251202+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T17:32:04.251202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a8c506a7-b951-4547-a5ef-91c2ce404f9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4f082249-8fc1-4369-a950-a16a958f1a43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9f2c9da7-9067-4110-add9-194db55a16f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T17:32:04.494 INFO:tasks.ceph.ceph_manager.c1:[] 2024-03-04T17:32:04.494 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-04T17:32:04.798 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-03-04T17:32:04.889 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:04.885+0000 7f5080f53180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T17:32:04.940 INFO:tasks.ceph.c1.osd.0.smithi136.stderr:2024-03-04T17:32:04.937+0000 7fc7230b97c0 -1 Falling back to public interface 2024-03-04T17:32:04.941 INFO:tasks.ceph.c1.osd.2.smithi136.stderr:2024-03-04T17:32:04.937+0000 7f1784a057c0 -1 Falling back to public interface 2024-03-04T17:32:04.960 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:04.957+0000 7f5080f53180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T17:32:04.968 INFO:tasks.ceph.c1.osd.1.smithi136.stderr:2024-03-04T17:32:04.965+0000 7fe8418107c0 -1 Falling back to public interface 2024-03-04T17:32:04.989 INFO:teuthology.misc.health.smithi136.stdout: 2024-03-04T17:32:04.989 INFO:teuthology.misc.health.smithi136.stdout:{"epoch":5,"fsid":"3b7fafa4-847c-4367-bdf5-f9f1b7e3e746","created":"2024-03-04T17:32:03.338712+0000","modified":"2024-03-04T17:32:04.251202+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T17:32:04.251202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a8c506a7-b951-4547-a5ef-91c2ce404f9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4f082249-8fc1-4369-a950-a16a958f1a43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9f2c9da7-9067-4110-add9-194db55a16f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T17:32:05.004 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-04T17:32:05.017 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:05.013+0000 7f5080f53180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T17:32:05.093 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:05.089+0000 7f5080f53180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T17:32:05.144 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:05.141+0000 7f5080f53180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T17:32:05.221 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:05.217+0000 7f5080f53180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T17:32:05.275 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:05.269+0000 7f5080f53180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T17:32:05.423 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:05.417+0000 7f5080f53180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T17:32:05.484 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:05.481+0000 7f5080f53180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T17:32:06.132 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-04T17:32:06.132 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-04T17:32:06.132 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr: from numpy import show_config as show_numpy_config 2024-03-04T17:32:06.175 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:06.169+0000 7f5080f53180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T17:32:06.349 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:06.345+0000 7f5080f53180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T17:32:06.522 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:06.517+0000 7f5080f53180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T17:32:06.593 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:06.589+0000 7f5080f53180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T17:32:06.739 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:06.733+0000 7f5080f53180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T17:32:07.356 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:07.353+0000 7f5080f53180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T17:32:07.435 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:07.429+0000 7f5080f53180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T17:32:07.623 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:07.617+0000 7f5080f53180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T17:32:07.669 INFO:tasks.ceph.c1.osd.2.smithi136.stderr:2024-03-04T17:32:07.665+0000 7f1784a057c0 -1 osd.2 0 log_to_monitors true 2024-03-04T17:32:07.700 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:07.697+0000 7f5080f53180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T17:32:07.757 INFO:tasks.ceph.c1.osd.1.smithi136.stderr:2024-03-04T17:32:07.753+0000 7fe8418107c0 -1 osd.1 0 log_to_monitors true 2024-03-04T17:32:07.805 INFO:tasks.ceph.c1.osd.0.smithi136.stderr:2024-03-04T17:32:07.797+0000 7fc7230b97c0 -1 osd.0 0 log_to_monitors true 2024-03-04T17:32:08.004 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:08.001+0000 7f5080f53180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T17:32:08.065 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:08.061+0000 7f5080f53180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T17:32:09.100 INFO:tasks.ceph.c1.mgr.x.smithi136.stderr:2024-03-04T17:32:09.097+0000 7f50283a1640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-04T17:32:09.127 INFO:tasks.ceph.c1.osd.2.smithi136.stderr:2024-03-04T17:32:09.125+0000 7f17809ae640 -1 osd.2 0 waiting for initial osdmap 2024-03-04T17:32:09.148 INFO:tasks.ceph.c1.osd.2.smithi136.stderr:2024-03-04T17:32:09.145+0000 7f17787e3640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T17:32:10.500 INFO:tasks.ceph.c1.osd.0.smithi136.stderr:2024-03-04T17:32:10.497+0000 7fc71f062640 -1 osd.0 0 waiting for initial osdmap 2024-03-04T17:32:10.537 INFO:tasks.ceph.c1.osd.1.smithi136.stderr:2024-03-04T17:32:10.533+0000 7fe83d7b9640 -1 osd.1 0 waiting for initial osdmap 2024-03-04T17:32:10.544 INFO:tasks.ceph.c1.osd.0.smithi136.stderr:2024-03-04T17:32:10.541+0000 7fc716e97640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T17:32:10.565 INFO:tasks.ceph.c1.osd.1.smithi136.stderr:2024-03-04T17:32:10.557+0000 7fe8355ee640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T17:32:11.308 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-03-04T17:32:11.601 INFO:teuthology.misc.health.smithi136.stdout: 2024-03-04T17:32:11.602 INFO:teuthology.misc.health.smithi136.stdout:{"epoch":9,"fsid":"3b7fafa4-847c-4367-bdf5-f9f1b7e3e746","created":"2024-03-04T17:32:03.338712+0000","modified":"2024-03-04T17:32:10.734138+0000","last_up_change":"2024-03-04T17:32:10.734138+0000","last_in_change":"2024-03-04T17:32:04.251202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a8c506a7-b951-4547-a5ef-91c2ce404f9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":109054956},{"type":"v1","addr":"172.21.15.136:6803","nonce":109054956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":109054956},{"type":"v1","addr":"172.21.15.136:6807","nonce":109054956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6813","nonce":109054956},{"type":"v1","addr":"172.21.15.136:6815","nonce":109054956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":109054956},{"type":"v1","addr":"172.21.15.136:6811","nonce":109054956}]},"public_addr":"172.21.15.136:6803/109054956","cluster_addr":"172.21.15.136:6807/109054956","heartbeat_back_addr":"172.21.15.136:6815/109054956","heartbeat_front_addr":"172.21.15.136:6811/109054956","state":["exists","up"]},{"osd":1,"uuid":"4f082249-8fc1-4369-a950-a16a958f1a43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":95447031},{"type":"v1","addr":"172.21.15.136:6817","nonce":95447031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":95447031},{"type":"v1","addr":"172.21.15.136:6819","nonce":95447031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":95447031},{"type":"v1","addr":"172.21.15.136:6823","nonce":95447031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":95447031},{"type":"v1","addr":"172.21.15.136:6821","nonce":95447031}]},"public_addr":"172.21.15.136:6817/95447031","cluster_addr":"172.21.15.136:6819/95447031","heartbeat_back_addr":"172.21.15.136:6823/95447031","heartbeat_front_addr":"172.21.15.136:6821/95447031","state":["exists","up"]},{"osd":2,"uuid":"9f2c9da7-9067-4110-add9-194db55a16f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1830063209},{"type":"v1","addr":"172.21.15.136:6802","nonce":1830063209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1830063209},{"type":"v1","addr":"172.21.15.136:6806","nonce":1830063209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1830063209},{"type":"v1","addr":"172.21.15.136:6814","nonce":1830063209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1830063209},{"type":"v1","addr":"172.21.15.136:6810","nonce":1830063209}]},"public_addr":"172.21.15.136:6802/1830063209","cluster_addr":"172.21.15.136:6806/1830063209","heartbeat_back_addr":"172.21.15.136:6814/1830063209","heartbeat_front_addr":"172.21.15.136:6810/1830063209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T17:32:08.697652+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T17:32:11.619 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-03-04T17:32:11.619 INFO:tasks.ceph:Creating RBD pool 2024-03-04T17:32:11.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster c1 osd pool create rbd 8 2024-03-04T17:32:12.752 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'rbd' created 2024-03-04T17:32:12.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-04T17:32:13.753 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-04T17:32:13.784 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2024-03-04T17:32:13.785 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2024-03-04T17:32:13.785 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2024-03-04T17:32:13.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2024-03-04T17:32:13.786 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-04T17:32:14.128 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:14.146 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3173980500},{"type":"v1","addr":"172.21.15.136:6825","nonce":3173980500}]},"active_addr":"172.21.15.136:6825/3173980500","active_change":"2024-03-04T17:32:08.086505+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":3095535991}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":1925125203}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":4215834991}]}]} 2024-03-04T17:32:14.148 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2024-03-04T17:32:14.149 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2024-03-04T17:32:14.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-03-04T17:32:14.438 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:14.438 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":12,"fsid":"3b7fafa4-847c-4367-bdf5-f9f1b7e3e746","created":"2024-03-04T17:32:03.338712+0000","modified":"2024-03-04T17:32:13.753482+0000","last_up_change":"2024-03-04T17:32:10.734138+0000","last_in_change":"2024-03-04T17:32:04.251202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-03-04T17:32:11.950618+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-03-04T17:32:12.173213+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a8c506a7-b951-4547-a5ef-91c2ce404f9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":109054956},{"type":"v1","addr":"172.21.15.136:6803","nonce":109054956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":109054956},{"type":"v1","addr":"172.21.15.136:6807","nonce":109054956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6813","nonce":109054956},{"type":"v1","addr":"172.21.15.136:6815","nonce":109054956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":109054956},{"type":"v1","addr":"172.21.15.136:6811","nonce":109054956}]},"public_addr":"172.21.15.136:6803/109054956","cluster_addr":"172.21.15.136:6807/109054956","heartbeat_back_addr":"172.21.15.136:6815/109054956","heartbeat_front_addr":"172.21.15.136:6811/109054956","state":["exists","up"]},{"osd":1,"uuid":"4f082249-8fc1-4369-a950-a16a958f1a43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":95447031},{"type":"v1","addr":"172.21.15.136:6817","nonce":95447031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":95447031},{"type":"v1","addr":"172.21.15.136:6819","nonce":95447031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":95447031},{"type":"v1","addr":"172.21.15.136:6823","nonce":95447031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":95447031},{"type":"v1","addr":"172.21.15.136:6821","nonce":95447031}]},"public_addr":"172.21.15.136:6817/95447031","cluster_addr":"172.21.15.136:6819/95447031","heartbeat_back_addr":"172.21.15.136:6823/95447031","heartbeat_front_addr":"172.21.15.136:6821/95447031","state":["exists","up"]},{"osd":2,"uuid":"9f2c9da7-9067-4110-add9-194db55a16f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1830063209},{"type":"v1","addr":"172.21.15.136:6802","nonce":1830063209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1830063209},{"type":"v1","addr":"172.21.15.136:6806","nonce":1830063209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1830063209},{"type":"v1","addr":"172.21.15.136:6814","nonce":1830063209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1830063209},{"type":"v1","addr":"172.21.15.136:6810","nonce":1830063209}]},"public_addr":"172.21.15.136:6802/1830063209","cluster_addr":"172.21.15.136:6806/1830063209","heartbeat_back_addr":"172.21.15.136:6814/1830063209","heartbeat_front_addr":"172.21.15.136:6810/1830063209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T17:32:08.802685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T17:32:08.800816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T17:32:08.697652+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T17:32:14.456 INFO:tasks.ceph.ceph_manager.c1:all up! 2024-03-04T17:32:14.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-03-04T17:32:14.751 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:14.752 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":12,"fsid":"3b7fafa4-847c-4367-bdf5-f9f1b7e3e746","created":"2024-03-04T17:32:03.338712+0000","modified":"2024-03-04T17:32:13.753482+0000","last_up_change":"2024-03-04T17:32:10.734138+0000","last_in_change":"2024-03-04T17:32:04.251202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-03-04T17:32:11.950618+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-03-04T17:32:12.173213+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a8c506a7-b951-4547-a5ef-91c2ce404f9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":109054956},{"type":"v1","addr":"172.21.15.136:6803","nonce":109054956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":109054956},{"type":"v1","addr":"172.21.15.136:6807","nonce":109054956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6813","nonce":109054956},{"type":"v1","addr":"172.21.15.136:6815","nonce":109054956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":109054956},{"type":"v1","addr":"172.21.15.136:6811","nonce":109054956}]},"public_addr":"172.21.15.136:6803/109054956","cluster_addr":"172.21.15.136:6807/109054956","heartbeat_back_addr":"172.21.15.136:6815/109054956","heartbeat_front_addr":"172.21.15.136:6811/109054956","state":["exists","up"]},{"osd":1,"uuid":"4f082249-8fc1-4369-a950-a16a958f1a43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":95447031},{"type":"v1","addr":"172.21.15.136:6817","nonce":95447031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":95447031},{"type":"v1","addr":"172.21.15.136:6819","nonce":95447031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":95447031},{"type":"v1","addr":"172.21.15.136:6823","nonce":95447031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":95447031},{"type":"v1","addr":"172.21.15.136:6821","nonce":95447031}]},"public_addr":"172.21.15.136:6817/95447031","cluster_addr":"172.21.15.136:6819/95447031","heartbeat_back_addr":"172.21.15.136:6823/95447031","heartbeat_front_addr":"172.21.15.136:6821/95447031","state":["exists","up"]},{"osd":2,"uuid":"9f2c9da7-9067-4110-add9-194db55a16f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1830063209},{"type":"v1","addr":"172.21.15.136:6802","nonce":1830063209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1830063209},{"type":"v1","addr":"172.21.15.136:6806","nonce":1830063209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1830063209},{"type":"v1","addr":"172.21.15.136:6814","nonce":1830063209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1830063209},{"type":"v1","addr":"172.21.15.136:6810","nonce":1830063209}]},"public_addr":"172.21.15.136:6802/1830063209","cluster_addr":"172.21.15.136:6806/1830063209","heartbeat_back_addr":"172.21.15.136:6814/1830063209","heartbeat_front_addr":"172.21.15.136:6810/1830063209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T17:32:08.802685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T17:32:08.800816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T17:32:08.697652+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T17:32:14.770 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2024-03-04T17:32:14.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2024-03-04T17:32:14.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2024-03-04T17:32:14.918 INFO:teuthology.orchestra.run.smithi136.stdout:38654705667 2024-03-04T17:32:14.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-03-04T17:32:14.931 INFO:teuthology.orchestra.run.smithi136.stdout:34359738371 2024-03-04T17:32:14.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-03-04T17:32:14.941 INFO:teuthology.orchestra.run.smithi136.stdout:38654705667 2024-03-04T17:32:14.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-03-04T17:32:15.220 INFO:teuthology.orchestra.run.smithi136.stdout:38654705666 2024-03-04T17:32:15.237 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.1 2024-03-04T17:32:15.263 INFO:teuthology.orchestra.run.smithi136.stdout:38654705666 2024-03-04T17:32:15.275 INFO:teuthology.orchestra.run.smithi136.stdout:34359738370 2024-03-04T17:32:15.281 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.0 2024-03-04T17:32:15.293 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.2 2024-03-04T17:32:16.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-03-04T17:32:16.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-03-04T17:32:16.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-03-04T17:32:16.544 INFO:teuthology.orchestra.run.smithi136.stdout:38654705666 2024-03-04T17:32:16.561 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.1 2024-03-04T17:32:16.576 INFO:teuthology.orchestra.run.smithi136.stdout:38654705666 2024-03-04T17:32:16.593 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.0 2024-03-04T17:32:16.602 INFO:teuthology.orchestra.run.smithi136.stdout:34359738370 2024-03-04T17:32:16.620 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.2 2024-03-04T17:32:17.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-03-04T17:32:17.594 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-03-04T17:32:17.621 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-03-04T17:32:17.884 INFO:teuthology.orchestra.run.smithi136.stdout:38654705667 2024-03-04T17:32:17.896 INFO:teuthology.orchestra.run.smithi136.stdout:38654705667 2024-03-04T17:32:17.910 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705667 for osd.1 2024-03-04T17:32:17.910 DEBUG:teuthology.parallel:result is None 2024-03-04T17:32:17.914 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705667 for osd.0 2024-03-04T17:32:17.914 DEBUG:teuthology.parallel:result is None 2024-03-04T17:32:17.943 INFO:teuthology.orchestra.run.smithi136.stdout:34359738372 2024-03-04T17:32:17.959 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738372 for osd.2 2024-03-04T17:32:17.960 DEBUG:teuthology.parallel:result is None 2024-03-04T17:32:17.960 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-03-04T17:32:17.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-03-04T17:32:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:18.282 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-03-04T17:32:18.298 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-03-04T17:32:18.097414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82732,"kb_used_data":1488,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490068,"statfs":{"total":322122547200,"available":322037829632,"internally_reserved":0,"allocated":1523712,"data_stored":1270515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000526"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T17:32:14.763775+0000","last_change":"2024-03-04T17:32:13.767091+0000","last_active":"2024-03-04T17:32:14.763775+0000","last_peered":"2024-03-04T17:32:14.763775+0000","last_clean":"2024-03-04T17:32:14.763775+0000","last_became_active":"2024-03-04T17:32:13.766716+0000","last_became_peered":"2024-03-04T17:32:13.766716+0000","last_unstale":"2024-03-04T17:32:14.763775+0000","last_undegraded":"2024-03-04T17:32:14.763775+0000","last_fullsized":"2024-03-04T17:32:14.763775+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:41:06.375134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T17:32:14.763706+0000","last_change":"2024-03-04T17:32:13.767089+0000","last_active":"2024-03-04T17:32:14.763706+0000","last_peered":"2024-03-04T17:32:14.763706+0000","last_clean":"2024-03-04T17:32:14.763706+0000","last_became_active":"2024-03-04T17:32:13.766805+0000","last_became_peered":"2024-03-04T17:32:13.766805+0000","last_unstale":"2024-03-04T17:32:14.763706+0000","last_undegraded":"2024-03-04T17:32:14.763706+0000","last_fullsized":"2024-03-04T17:32:14.763706+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:25:36.518173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:13.767363+0000","last_change":"2024-03-04T17:32:13.765315+0000","last_active":"2024-03-04T17:32:13.767363+0000","last_peered":"2024-03-04T17:32:13.767363+0000","last_clean":"2024-03-04T17:32:13.767363+0000","last_became_active":"2024-03-04T17:32:13.764941+0000","last_became_peered":"2024-03-04T17:32:13.764941+0000","last_unstale":"2024-03-04T17:32:13.767363+0000","last_undegraded":"2024-03-04T17:32:13.767363+0000","last_fullsized":"2024-03-04T17:32:13.767363+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:32:08.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T17:32:14.763659+0000","last_change":"2024-03-04T17:32:13.767387+0000","last_active":"2024-03-04T17:32:14.763659+0000","last_peered":"2024-03-04T17:32:14.763659+0000","last_clean":"2024-03-04T17:32:14.763659+0000","last_became_active":"2024-03-04T17:32:13.766835+0000","last_became_peered":"2024-03-04T17:32:13.766835+0000","last_unstale":"2024-03-04T17:32:14.763659+0000","last_undegraded":"2024-03-04T17:32:14.763659+0000","last_fullsized":"2024-03-04T17:32:14.763659+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:16.574846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":136,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:14.284618+0000","last_change":"2024-03-04T17:32:13.768567+0000","last_active":"2024-03-04T17:32:14.284618+0000","last_peered":"2024-03-04T17:32:14.284618+0000","last_clean":"2024-03-04T17:32:14.284618+0000","last_became_active":"2024-03-04T17:32:13.768241+0000","last_became_peered":"2024-03-04T17:32:13.768241+0000","last_unstale":"2024-03-04T17:32:14.284618+0000","last_undegraded":"2024-03-04T17:32:14.284618+0000","last_fullsized":"2024-03-04T17:32:14.284618+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:29:34.735772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T17:32:14.763654+0000","last_change":"2024-03-04T17:32:13.765549+0000","last_active":"2024-03-04T17:32:14.763654+0000","last_peered":"2024-03-04T17:32:14.763654+0000","last_clean":"2024-03-04T17:32:14.763654+0000","last_became_active":"2024-03-04T17:32:13.765132+0000","last_became_peered":"2024-03-04T17:32:13.765132+0000","last_unstale":"2024-03-04T17:32:14.763654+0000","last_undegraded":"2024-03-04T17:32:14.763654+0000","last_fullsized":"2024-03-04T17:32:14.763654+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:49:14.739466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T17:32:15.407586+0000","last_change":"2024-03-04T17:32:13.768156+0000","last_active":"2024-03-04T17:32:15.407586+0000","last_peered":"2024-03-04T17:32:15.407586+0000","last_clean":"2024-03-04T17:32:15.407586+0000","last_became_active":"2024-03-04T17:32:13.767788+0000","last_became_peered":"2024-03-04T17:32:13.767788+0000","last_unstale":"2024-03-04T17:32:15.407586+0000","last_undegraded":"2024-03-04T17:32:15.407586+0000","last_fullsized":"2024-03-04T17:32:15.407586+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:12:32.576780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:13.764987+0000","last_change":"2024-03-04T17:32:13.763952+0000","last_active":"2024-03-04T17:32:13.764987+0000","last_peered":"2024-03-04T17:32:13.764987+0000","last_clean":"2024-03-04T17:32:13.764987+0000","last_became_active":"2024-03-04T17:32:13.763708+0000","last_became_peered":"2024-03-04T17:32:13.763708+0000","last_unstale":"2024-03-04T17:32:13.764987+0000","last_undegraded":"2024-03-04T17:32:13.764987+0000","last_fullsized":"2024-03-04T17:32:13.764987+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:36:06.298410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T17:32:14.763885+0000","last_change":"2024-03-04T17:32:13.767674+0000","last_active":"2024-03-04T17:32:14.763885+0000","last_peered":"2024-03-04T17:32:14.763885+0000","last_clean":"2024-03-04T17:32:14.763885+0000","last_became_active":"2024-03-04T17:32:13.767106+0000","last_became_peered":"2024-03-04T17:32:13.767106+0000","last_unstale":"2024-03-04T17:32:14.763885+0000","last_undegraded":"2024-03-04T17:32:14.763885+0000","last_fullsized":"2024-03-04T17:32:14.763885+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:02:34.962717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":708608,"data_stored":621577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830468,"statfs":{"total":107374182400,"available":107346399232,"internally_reserved":0,"allocated":114688,"data_stored":31209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27888,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829712,"statfs":{"total":107374182400,"available":107345625088,"internally_reserved":0,"allocated":700416,"data_stored":617729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T17:32:18.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-03-04T17:32:18.566 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:18.566 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-03-04T17:32:18.582 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-03-04T17:32:18.097414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82732,"kb_used_data":1488,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490068,"statfs":{"total":322122547200,"available":322037829632,"internally_reserved":0,"allocated":1523712,"data_stored":1270515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000526"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T17:32:14.763775+0000","last_change":"2024-03-04T17:32:13.767091+0000","last_active":"2024-03-04T17:32:14.763775+0000","last_peered":"2024-03-04T17:32:14.763775+0000","last_clean":"2024-03-04T17:32:14.763775+0000","last_became_active":"2024-03-04T17:32:13.766716+0000","last_became_peered":"2024-03-04T17:32:13.766716+0000","last_unstale":"2024-03-04T17:32:14.763775+0000","last_undegraded":"2024-03-04T17:32:14.763775+0000","last_fullsized":"2024-03-04T17:32:14.763775+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:41:06.375134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T17:32:14.763706+0000","last_change":"2024-03-04T17:32:13.767089+0000","last_active":"2024-03-04T17:32:14.763706+0000","last_peered":"2024-03-04T17:32:14.763706+0000","last_clean":"2024-03-04T17:32:14.763706+0000","last_became_active":"2024-03-04T17:32:13.766805+0000","last_became_peered":"2024-03-04T17:32:13.766805+0000","last_unstale":"2024-03-04T17:32:14.763706+0000","last_undegraded":"2024-03-04T17:32:14.763706+0000","last_fullsized":"2024-03-04T17:32:14.763706+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:25:36.518173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:13.767363+0000","last_change":"2024-03-04T17:32:13.765315+0000","last_active":"2024-03-04T17:32:13.767363+0000","last_peered":"2024-03-04T17:32:13.767363+0000","last_clean":"2024-03-04T17:32:13.767363+0000","last_became_active":"2024-03-04T17:32:13.764941+0000","last_became_peered":"2024-03-04T17:32:13.764941+0000","last_unstale":"2024-03-04T17:32:13.767363+0000","last_undegraded":"2024-03-04T17:32:13.767363+0000","last_fullsized":"2024-03-04T17:32:13.767363+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:32:08.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T17:32:14.763659+0000","last_change":"2024-03-04T17:32:13.767387+0000","last_active":"2024-03-04T17:32:14.763659+0000","last_peered":"2024-03-04T17:32:14.763659+0000","last_clean":"2024-03-04T17:32:14.763659+0000","last_became_active":"2024-03-04T17:32:13.766835+0000","last_became_peered":"2024-03-04T17:32:13.766835+0000","last_unstale":"2024-03-04T17:32:14.763659+0000","last_undegraded":"2024-03-04T17:32:14.763659+0000","last_fullsized":"2024-03-04T17:32:14.763659+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:16.574846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":136,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:14.284618+0000","last_change":"2024-03-04T17:32:13.768567+0000","last_active":"2024-03-04T17:32:14.284618+0000","last_peered":"2024-03-04T17:32:14.284618+0000","last_clean":"2024-03-04T17:32:14.284618+0000","last_became_active":"2024-03-04T17:32:13.768241+0000","last_became_peered":"2024-03-04T17:32:13.768241+0000","last_unstale":"2024-03-04T17:32:14.284618+0000","last_undegraded":"2024-03-04T17:32:14.284618+0000","last_fullsized":"2024-03-04T17:32:14.284618+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:29:34.735772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T17:32:14.763654+0000","last_change":"2024-03-04T17:32:13.765549+0000","last_active":"2024-03-04T17:32:14.763654+0000","last_peered":"2024-03-04T17:32:14.763654+0000","last_clean":"2024-03-04T17:32:14.763654+0000","last_became_active":"2024-03-04T17:32:13.765132+0000","last_became_peered":"2024-03-04T17:32:13.765132+0000","last_unstale":"2024-03-04T17:32:14.763654+0000","last_undegraded":"2024-03-04T17:32:14.763654+0000","last_fullsized":"2024-03-04T17:32:14.763654+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:49:14.739466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T17:32:15.407586+0000","last_change":"2024-03-04T17:32:13.768156+0000","last_active":"2024-03-04T17:32:15.407586+0000","last_peered":"2024-03-04T17:32:15.407586+0000","last_clean":"2024-03-04T17:32:15.407586+0000","last_became_active":"2024-03-04T17:32:13.767788+0000","last_became_peered":"2024-03-04T17:32:13.767788+0000","last_unstale":"2024-03-04T17:32:15.407586+0000","last_undegraded":"2024-03-04T17:32:15.407586+0000","last_fullsized":"2024-03-04T17:32:15.407586+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:12:32.576780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:13.764987+0000","last_change":"2024-03-04T17:32:13.763952+0000","last_active":"2024-03-04T17:32:13.764987+0000","last_peered":"2024-03-04T17:32:13.764987+0000","last_clean":"2024-03-04T17:32:13.764987+0000","last_became_active":"2024-03-04T17:32:13.763708+0000","last_became_peered":"2024-03-04T17:32:13.763708+0000","last_unstale":"2024-03-04T17:32:13.764987+0000","last_undegraded":"2024-03-04T17:32:13.764987+0000","last_fullsized":"2024-03-04T17:32:13.764987+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:36:06.298410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T17:32:14.763885+0000","last_change":"2024-03-04T17:32:13.767674+0000","last_active":"2024-03-04T17:32:14.763885+0000","last_peered":"2024-03-04T17:32:14.763885+0000","last_clean":"2024-03-04T17:32:14.763885+0000","last_became_active":"2024-03-04T17:32:13.767106+0000","last_became_peered":"2024-03-04T17:32:13.767106+0000","last_unstale":"2024-03-04T17:32:14.763885+0000","last_undegraded":"2024-03-04T17:32:14.763885+0000","last_fullsized":"2024-03-04T17:32:14.763885+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:12.747122+0000","last_clean_scrub_stamp":"2024-03-04T17:32:12.747122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:02:34.962717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":708608,"data_stored":621577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830468,"statfs":{"total":107374182400,"available":107346399232,"internally_reserved":0,"allocated":114688,"data_stored":31209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27888,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829712,"statfs":{"total":107374182400,"available":107345625088,"internally_reserved":0,"allocated":700416,"data_stored":617729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T17:32:18.583 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-03-04T17:32:18.583 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2024-03-04T17:32:18.583 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2024-03-04T17:32:18.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2024-03-04T17:32:18.887 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:18.887 INFO:teuthology.orchestra.run.smithi136.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T17:32:18.904 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2024-03-04T17:32:18.905 INFO:teuthology.run_tasks:Running task ceph... 2024-03-04T17:32:18.915 INFO:tasks.ceph:Creating ceph cluster c2... 2024-03-04T17:32:18.915 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-04T17:32:18.915 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi/7581342', 'branch': 'wip-rgw-d4n', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7581342', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-rgw-d4n'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-rgw-d4n', 'sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8'}}, 'owner': 'scheduled_suriarte@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-rgw-d4n', 'suite_path': '/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/pritha-srivastava/ceph.git', 'suite_sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8', 'targets': {'smithi136.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAqHjmXINRGS++vEJkLSuQVoCDLD7Q/OTh0RAS4ORRa34gLI/XifPQGnY9NX/bUvqJ8L9VxxHFw5XRlWZkskD6M=', 'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM5+Lf8d9jkIUZDTAHuoCGKmJtNNK7KduqKFetIFOBuUOXs7F9wANcTWVBomJoQHbfTA4oYoHd9PFqtZins8FCs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8'}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '3aec390336fdfdc8f927818b718cfb0361bd47c8'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-03-04_17:13:34', 'tube': 'smithi', 'user': 'suriarte', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-04T17:32:18.919 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-03-04T17:32:18.922 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-03-04T17:32:18.932 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T17:32:18.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T17:32:18.996 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:18.996 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T17:32:19.045 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T17:32:19.045 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-03-04T17:32:19.092 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T17:32:19.092 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T17:32:19.092 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 881 Links: 1 2024-03-04T17:32:19.092 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T17:32:19.092 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-04 17:26:17.656126290 +0000 2024-03-04T17:32:19.093 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-04 17:26:17.652126324 +0000 2024-03-04T17:32:19.093 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-04 17:26:17.652126324 +0000 2024-03-04T17:32:19.093 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-03-04T17:32:19.093 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T17:32:19.155 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-04T17:32:19.155 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-04T17:32:19.155 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000437843 s, 1.2 MB/s 2024-03-04T17:32:19.156 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T17:32:19.208 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-03-04T17:32:19.255 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T17:32:19.255 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T17:32:19.255 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 896 Links: 1 2024-03-04T17:32:19.255 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T17:32:19.255 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-04 17:26:18.340120187 +0000 2024-03-04T17:32:19.255 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-04 17:26:18.104122293 +0000 2024-03-04T17:32:19.255 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-04 17:26:18.104122293 +0000 2024-03-04T17:32:19.255 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-03-04T17:32:19.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T17:32:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-04T17:32:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-04T17:32:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000414177 s, 1.2 MB/s 2024-03-04T17:32:19.310 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T17:32:19.362 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-03-04T17:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T17:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T17:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 914 Links: 1 2024-03-04T17:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T17:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-04 17:26:18.848115656 +0000 2024-03-04T17:32:19.412 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-04 17:26:18.588117975 +0000 2024-03-04T17:32:19.412 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-04 17:26:18.588117975 +0000 2024-03-04T17:32:19.412 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-03-04T17:32:19.412 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T17:32:19.465 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-04T17:32:19.465 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-04T17:32:19.465 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000404113 s, 1.3 MB/s 2024-03-04T17:32:19.466 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T17:32:19.516 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-03-04T17:32:19.562 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T17:32:19.563 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T17:32:19.563 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 931 Links: 1 2024-03-04T17:32:19.563 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T17:32:19.563 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-04 17:26:19.100113407 +0000 2024-03-04T17:32:19.563 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-04 17:26:19.096113443 +0000 2024-03-04T17:32:19.563 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-04 17:26:19.096113443 +0000 2024-03-04T17:32:19.563 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-03-04T17:32:19.563 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T17:32:19.617 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-04T17:32:19.617 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-04T17:32:19.617 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000388177 s, 1.3 MB/s 2024-03-04T17:32:19.618 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T17:32:19.668 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T17:32:19.668 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-04T17:32:19.668 INFO:tasks.ceph:Generating config... 2024-03-04T17:32:19.669 INFO:tasks.ceph:[client] debug rgw = 20 2024-03-04T17:32:19.669 INFO:tasks.ceph:[client] debug rgw lifecycle = 20 2024-03-04T17:32:19.669 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-03-04T17:32:19.669 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-03-04T17:32:19.669 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-03-04T17:32:19.670 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-03-04T17:32:19.670 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-03-04T17:32:19.670 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-03-04T17:32:19.670 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-03-04T17:32:19.670 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-03-04T17:32:19.670 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-03-04T17:32:19.670 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-03-04T17:32:19.670 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-03-04T17:32:19.670 INFO:tasks.ceph:[client] setgroup = ceph 2024-03-04T17:32:19.670 INFO:tasks.ceph:[client] setuser = ceph 2024-03-04T17:32:19.670 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-03-04T17:32:19.670 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-04T17:32:19.671 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-04T17:32:19.671 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-04T17:32:19.671 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-04T17:32:19.671 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-04T17:32:19.671 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-04T17:32:19.671 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-04T17:32:19.671 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-03-04T17:32:19.671 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-03-04T17:32:19.671 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-03-04T17:32:19.671 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-03-04T17:32:19.671 INFO:tasks.ceph:Setting up c2.mon.a... 2024-03-04T17:32:19.672 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2024-03-04T17:32:19.741 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/c2.keyring 2024-03-04T17:32:19.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2024-03-04T17:32:19.837 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-03-04T17:32:19.891 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.153')] 2024-03-04T17:32:19.891 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.153'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-03-04T17:32:19.892 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:19.892 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T17:32:19.939 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.153 --print /home/ubuntu/cephtest/c2.monmap 2024-03-04T17:32:20.017 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:20.017 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:20.018 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2024-03-04T17:32:20.018 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: generated fsid 5f1d29fe-dcec-4834-91dc-1efe93fdf67a 2024-03-04T17:32:20.018 INFO:teuthology.orchestra.run.smithi153.stdout:setting min_mon_release = quincy 2024-03-04T17:32:20.018 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 0 2024-03-04T17:32:20.018 INFO:teuthology.orchestra.run.smithi153.stdout:fsid 5f1d29fe-dcec-4834-91dc-1efe93fdf67a 2024-03-04T17:32:20.018 INFO:teuthology.orchestra.run.smithi153.stdout:last_changed 2024-03-04T17:32:20.018689+0000 2024-03-04T17:32:20.018 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-03-04T17:32:20.018689+0000 2024-03-04T17:32:20.018 INFO:teuthology.orchestra.run.smithi153.stdout:min_mon_release 17 (quincy) 2024-03-04T17:32:20.018 INFO:teuthology.orchestra.run.smithi153.stdout:election_strategy: 1 2024-03-04T17:32:20.018 INFO:teuthology.orchestra.run.smithi153.stdout:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-03-04T17:32:20.018 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2024-03-04T17:32:20.019 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T17:32:20.066 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 5f1d29fe-dcec-4834-91dc-1efe93fdf67a... 2024-03-04T17:32:20.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-03-04T17:32:20.071 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-03-04T17:32:20.113 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2024-03-04T17:32:20.113 INFO:teuthology.orchestra.run.smithi136.stdout: chdir = "" 2024-03-04T17:32:20.113 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T17:32:20.113 INFO:teuthology.orchestra.run.smithi136.stdout: auth supported = cephx 2024-03-04T17:32:20.113 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.113 INFO:teuthology.orchestra.run.smithi136.stdout: filestore xattr use omap = true 2024-03-04T17:32:20.113 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.113 INFO:teuthology.orchestra.run.smithi136.stdout: mon clock drift allowed = 1.000 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: osd crush chooseleaf type = 0 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: auth debug = true 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on old message = true 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on bug = true 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: mon pg warn max object skew = 0 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default size = 2 2024-03-04T17:32:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow primary affinity = true 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow pg remap = true 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on legacy crush tunables = false 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on crush straw calc version zero = false 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on no sortbitwise = false 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on osd down out interval zero = false 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on too few osds = false 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: mon_allow_pool_size_one = true 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: osd default data pool replay window = 5 2024-03-04T17:32:20.115 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.116 INFO:teuthology.orchestra.run.smithi136.stdout: mon allow pool delete = true 2024-03-04T17:32:20.116 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.116 INFO:teuthology.orchestra.run.smithi136.stdout: mon cluster log file level = debug 2024-03-04T17:32:20.116 INFO:teuthology.orchestra.run.smithi136.stdout: debug asserts on shutdown = true 2024-03-04T17:32:20.116 INFO:teuthology.orchestra.run.smithi136.stdout: mon health detail to clog = false 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout: mon host = 172.21.15.153 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = 5f1d29fe-dcec-4834-91dc-1efe93fdf67a 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout:[osd] 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout: osd journal size = 100 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub load threshold = 5.0 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub max interval = 600 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock profile = high_recovery_ops 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout: osd recover clone overlap = true 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout: osd recovery max chunk = 1048576 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.117 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug shutdown = true 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug op order = true 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify stray on activate = true 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug trim objects = true 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: osd open classes on start = true 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug pg log writeout = true 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: osd deep scrub update digest min age = 30 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: osd map max advance = 10 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: journal zero on create = true 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: filestore ondisk finisher threads = 3 2024-03-04T17:32:20.118 INFO:teuthology.orchestra.run.smithi136.stdout: filestore apply finisher threads = 3 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: bdev debug aio = true 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug misdirected ops = true 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: debug osd = 20 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: rocksdb delete range threshold = 0 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr] 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: debug mgr = 20 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-03-04T17:32:20.119 INFO:teuthology.orchestra.run.smithi136.stdout: mgr/telemetry/nag = false 2024-03-04T17:32:20.120 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.120 INFO:teuthology.orchestra.run.smithi136.stdout:[mon] 2024-03-04T17:32:20.120 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-04T17:32:20.120 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-03-04T17:32:20.121 INFO:teuthology.orchestra.run.smithi136.stdout: debug paxos = 20 2024-03-04T17:32:20.121 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-03-04T17:32:20.121 INFO:teuthology.orchestra.run.smithi136.stdout: mon data avail warn = 5 2024-03-04T17:32:20.121 INFO:teuthology.orchestra.run.smithi136.stdout: mon mgr mkfs grace = 240 2024-03-04T17:32:20.121 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-03-04T17:32:20.121 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd reporter subtree level = osd 2024-03-04T17:32:20.121 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd prime pg temp = true 2024-03-04T17:32:20.121 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-03-04T17:32:20.121 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.121 INFO:teuthology.orchestra.run.smithi136.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T17:32:20.121 INFO:teuthology.orchestra.run.smithi136.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T17:32:20.121 INFO:teuthology.orchestra.run.smithi136.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: # 1m isn't quite enough 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: mon_down_mkfs_grace = 2m 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_filestore_osds = false 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout:[client] 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: rgw cache enabled = true 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable ops log = true 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable usage log = true 2024-03-04T17:32:20.122 INFO:teuthology.orchestra.run.smithi136.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: debug rgw = 20 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: debug rgw lifecycle = 20 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: debug rgw sync = 20 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: rgw crypt require ssl = False 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: rgw crypt s3 kms backend = testing 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: rgw curl low speed time = 300 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: rgw data log num shards = 4 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: rgw md log max shards = 4 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: rgw sync data inject err probability = 0.1 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: rgw sync log trim interval = 0 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: rgw sync meta inject err probability = 0.1 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: rgw sync obj etag verify = True 2024-03-04T17:32:20.123 INFO:teuthology.orchestra.run.smithi136.stdout: setgroup = ceph 2024-03-04T17:32:20.124 INFO:teuthology.orchestra.run.smithi136.stdout: setuser = ceph 2024-03-04T17:32:20.124 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.a] 2024-03-04T17:32:20.124 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2024-03-04T17:32:20.124 INFO:teuthology.orchestra.run.smithi136.stdout: rgw data notify interval msec = 0 2024-03-04T17:32:20.124 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.0] 2024-03-04T17:32:20.124 INFO:teuthology.orchestra.run.smithi136.stdout: osd_max_omap_entries_per_request = 10 2024-03-04T17:32:20.124 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.1] 2024-03-04T17:32:20.124 INFO:teuthology.orchestra.run.smithi136.stdout: osd_max_omap_entries_per_request = 1000 2024-03-04T17:32:20.124 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.2] 2024-03-04T17:32:20.124 INFO:teuthology.orchestra.run.smithi136.stdout: osd_max_omap_entries_per_request = 10000 2024-03-04T17:32:20.139 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-03-04T17:32:20.139 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2024-03-04T17:32:20.141 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = 172.21.15.153 2024-03-04T17:32:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 5f1d29fe-dcec-4834-91dc-1efe93fdf67a 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2024-03-04T17:32:20.143 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug trim objects = true 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2024-03-04T17:32:20.144 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout: rocksdb delete range threshold = 0 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2024-03-04T17:32:20.145 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.146 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T17:32:20.147 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T17:32:20.148 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T17:32:20.148 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.148 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2024-03-04T17:32:20.148 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2024-03-04T17:32:20.148 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.148 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2024-03-04T17:32:20.148 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:20.148 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2024-03-04T17:32:20.148 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2024-03-04T17:32:20.148 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: debug rgw = 20 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: debug rgw lifecycle = 20 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: debug rgw sync = 20 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: rgw crypt require ssl = False 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: rgw crypt s3 kms backend = testing 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: rgw curl low speed time = 300 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: rgw data log num shards = 4 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: rgw md log max shards = 4 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: rgw sync data inject err probability = 0.1 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: rgw sync log trim interval = 0 2024-03-04T17:32:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: rgw sync meta inject err probability = 0.1 2024-03-04T17:32:20.150 INFO:teuthology.orchestra.run.smithi153.stdout: rgw sync obj etag verify = True 2024-03-04T17:32:20.150 INFO:teuthology.orchestra.run.smithi153.stdout: setgroup = ceph 2024-03-04T17:32:20.150 INFO:teuthology.orchestra.run.smithi153.stdout: setuser = ceph 2024-03-04T17:32:20.150 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2024-03-04T17:32:20.150 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-03-04T17:32:20.150 INFO:teuthology.orchestra.run.smithi153.stdout: rgw data notify interval msec = 0 2024-03-04T17:32:20.150 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2024-03-04T17:32:20.150 INFO:teuthology.orchestra.run.smithi153.stdout: osd_max_omap_entries_per_request = 10 2024-03-04T17:32:20.150 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2024-03-04T17:32:20.150 INFO:teuthology.orchestra.run.smithi153.stdout: osd_max_omap_entries_per_request = 1000 2024-03-04T17:32:20.150 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2024-03-04T17:32:20.150 INFO:teuthology.orchestra.run.smithi153.stdout: osd_max_omap_entries_per_request = 10000 2024-03-04T17:32:20.151 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2024-03-04T17:32:20.151 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2024-03-04T17:32:20.246 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-04T17:32:20.246 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:20.246 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2024-03-04T17:32:20.251 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:20.252 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2024-03-04T17:32:20.303 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi136.front.sepia.ceph.com 2024-03-04T17:32:20.303 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:32:20.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/c2.keyring 2024-03-04T17:32:20.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-03-04T17:32:20.366 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-04T17:32:20.366 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-03-04T17:32:20.412 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2024-03-04T17:32:20.413 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:20.498 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/c2.keyring 2024-03-04T17:32:20.498 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-03-04T17:32:20.534 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:20.534 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-03-04T17:32:20.584 INFO:tasks.ceph:Setting up mon nodes... 2024-03-04T17:32:20.584 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-04T17:32:20.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2024-03-04T17:32:20.665 INFO:teuthology.orchestra.run.smithi153.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2024-03-04T17:32:20.668 INFO:tasks.ceph:Setting up mds nodes... 2024-03-04T17:32:20.668 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-04T17:32:20.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2024-03-04T17:32:20.742 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/c2.client.0.keyring 2024-03-04T17:32:20.752 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2024-03-04T17:32:20.826 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/c2.client.1.keyring 2024-03-04T17:32:20.836 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-04T17:32:20.836 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-04T17:32:20.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2024-03-04T17:32:20.890 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T17:32:20.890 INFO:tasks.ceph:role: c2.osd.0 2024-03-04T17:32:20.890 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-04T17:32:20.890 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-04T17:32:20.953 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T17:32:20.954 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T17:32:20.954 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T17:32:20.954 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-04T17:32:20.954 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T17:32:20.954 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-04T17:32:20.954 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T17:32:20.954 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-04T17:32:20.954 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T17:32:20.954 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T17:32:21.407 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-04T17:32:21.409 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-04T17:32:21.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2024-03-04T17:32:21.500 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2024-03-04T17:32:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T17:32:21.510 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: /sbin/restorecon: command not found 2024-03-04T17:32:21.510 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2024-03-04T17:32:21.564 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T17:32:21.564 INFO:tasks.ceph:role: c2.osd.1 2024-03-04T17:32:21.564 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-04T17:32:21.565 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-04T17:32:21.627 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T17:32:21.627 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T17:32:21.627 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T17:32:21.627 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-04T17:32:21.627 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T17:32:21.627 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-04T17:32:21.627 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T17:32:21.627 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-04T17:32:21.627 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T17:32:21.627 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T17:32:22.101 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-04T17:32:22.103 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-04T17:32:22.103 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2024-03-04T17:32:22.262 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2024-03-04T17:32:22.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T17:32:22.274 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: /sbin/restorecon: command not found 2024-03-04T17:32:22.274 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2024-03-04T17:32:22.332 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T17:32:22.332 INFO:tasks.ceph:role: c2.osd.2 2024-03-04T17:32:22.332 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-04T17:32:22.332 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-04T17:32:22.396 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T17:32:22.397 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T17:32:22.397 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T17:32:22.397 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-04T17:32:22.397 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T17:32:22.397 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-04T17:32:22.397 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T17:32:22.397 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-04T17:32:22.397 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T17:32:22.397 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T17:32:22.884 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-04T17:32:22.887 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-04T17:32:22.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2024-03-04T17:32:23.042 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2024-03-04T17:32:23.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T17:32:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: /sbin/restorecon: command not found 2024-03-04T17:32:23.054 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2024-03-04T17:32:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:23.214+0000 7fa47cff27c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2024-03-04T17:32:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:23.214+0000 7fa47cff27c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2024-03-04T17:32:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:23.214+0000 7fa47cff27c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-03-04T17:32:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:23.214+0000 7fa47cff27c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-03-04T17:32:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:23.214+0000 7fa47cff27c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-03-04T17:32:23.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:23.214+0000 7fa47cff27c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2024-03-04T17:32:25.382 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2024-03-04T17:32:25.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2024-03-04T17:32:25.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:25.474+0000 7f98894237c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2024-03-04T17:32:25.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:25.474+0000 7f98894237c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2024-03-04T17:32:25.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:25.474+0000 7f98894237c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-03-04T17:32:25.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:25.474+0000 7f98894237c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-03-04T17:32:25.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:25.474+0000 7f98894237c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-03-04T17:32:25.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:25.478+0000 7f98894237c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2024-03-04T17:32:27.653 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2024-03-04T17:32:27.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2024-03-04T17:32:27.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:27.786+0000 7f1ab56247c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2024-03-04T17:32:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:27.786+0000 7f1ab56247c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2024-03-04T17:32:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:27.786+0000 7f1ab56247c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-03-04T17:32:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:27.786+0000 7f1ab56247c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-03-04T17:32:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:27.786+0000 7f1ab56247c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-03-04T17:32:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:32:27.786+0000 7f1ab56247c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2024-03-04T17:32:29.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2024-03-04T17:32:30.036 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-04T17:32:30.036 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:30.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2024-03-04T17:32:30.089 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:30.089 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2024-03-04T17:32:30.144 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:30.144 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2024-03-04T17:32:30.198 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:30.198 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2024-03-04T17:32:30.254 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:30.254 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2024-03-04T17:32:30.303 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:30.303 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2024-03-04T17:32:30.351 INFO:tasks.ceph:Adding keys to all mons... 2024-03-04T17:32:30.351 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/c2.keyring 2024-03-04T17:32:30.405 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2024-03-04T17:32:30.405 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCkBeZlnCO3JxAALPHy/PDLDbj+5IrVufQvQg== 2024-03-04T17:32:30.405 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2024-03-04T17:32:30.405 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCnBeZl81fYDBAA8uys+v+97mjYBffOkWdhBQ== 2024-03-04T17:32:30.405 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2024-03-04T17:32:30.405 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCpBeZlXml6HBAASeJxP+9m7EC3CdRBrptJgw== 2024-03-04T17:32:30.405 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2024-03-04T17:32:30.405 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCrBeZlV24PLxAAKfkJM23q6v4pB6T0dgqDiw== 2024-03-04T17:32:30.405 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-03-04T17:32:30.405 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCkBeZlpG1RLBAABmwkBQty4krNTEYQ5OuNnQ== 2024-03-04T17:32:30.405 INFO:teuthology.orchestra.run.smithi153.stdout:[client.1] 2024-03-04T17:32:30.406 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCkBeZlWKc7MRAAp2ZLN01IDgAmmpKxN0TCtg== 2024-03-04T17:32:30.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-04T17:32:30.495 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T17:32:30.585 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T17:32:30.627 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T17:32:30.712 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T17:32:30.798 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T17:32:30.889 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-04T17:32:30.889 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2024-03-04T17:32:30.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2024-03-04T17:32:30.990 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2024-03-04T17:32:31.001 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/c2.monmap 2024-03-04T17:32:31.046 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2024-03-04T17:32:31.047 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2024-03-04T17:32:31.047 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2024-03-04T17:32:31.093 INFO:tasks.ceph.c2.mon.a:Started 2024-03-04T17:32:31.094 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2024-03-04T17:32:31.094 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2024-03-04T17:32:31.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2024-03-04T17:32:31.097 INFO:tasks.ceph.c2.mgr.x:Started 2024-03-04T17:32:31.097 INFO:tasks.ceph:Setting crush tunables to default 2024-03-04T17:32:31.097 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster c2 osd crush tunables default 2024-03-04T17:32:31.349 INFO:teuthology.orchestra.run.smithi153.stderr:adjusted tunables profile to default 2024-03-04T17:32:31.368 INFO:tasks.ceph:check_enable_crimson: False 2024-03-04T17:32:31.368 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2024-03-04T17:32:31.368 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:31.368 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2024-03-04T17:32:31.380 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:31.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2024-03-04T17:32:31.437 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-04T17:32:31.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2024-03-04T17:32:31.451 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:31.450+0000 7f80076f7180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T17:32:31.461 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster c2 osd new 6c852b2f-e2ad-474b-88e4-d9e330c05f88 0 2024-03-04T17:32:31.706 INFO:teuthology.orchestra.run.smithi153.stdout:0 2024-03-04T17:32:31.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster c2 osd new 6ef7e72e-dab4-4fbf-b7d4-e82b9d9fd3e1 1 2024-03-04T17:32:31.907 INFO:teuthology.orchestra.run.smithi153.stdout:1 2024-03-04T17:32:31.924 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster c2 osd new 105fb43b-575f-4bf1-89b4-be8d155bfa92 2 2024-03-04T17:32:31.962 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:31.962+0000 7f80076f7180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T17:32:32.117 INFO:teuthology.orchestra.run.smithi153.stdout:2 2024-03-04T17:32:32.134 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2024-03-04T17:32:32.134 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2024-03-04T17:32:32.170 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:32.166+0000 7f80076f7180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T17:32:32.174 INFO:tasks.ceph.c2.osd.0:Started 2024-03-04T17:32:32.174 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2024-03-04T17:32:32.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2024-03-04T17:32:32.179 INFO:tasks.ceph.c2.osd.1:Started 2024-03-04T17:32:32.179 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2024-03-04T17:32:32.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2024-03-04T17:32:32.181 INFO:tasks.ceph.c2.osd.2:Started 2024-03-04T17:32:32.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-03-04T17:32:32.352 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:32.352 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":5,"fsid":"5f1d29fe-dcec-4834-91dc-1efe93fdf67a","created":"2024-03-04T17:32:31.222798+0000","modified":"2024-03-04T17:32:32.116983+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T17:32:32.116983+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c852b2f-e2ad-474b-88e4-d9e330c05f88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ef7e72e-dab4-4fbf-b7d4-e82b9d9fd3e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"105fb43b-575f-4bf1-89b4-be8d155bfa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T17:32:32.368 INFO:tasks.ceph.ceph_manager.c2:[] 2024-03-04T17:32:32.369 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-04T17:32:32.671 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-03-04T17:32:32.747 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:32.746+0000 7f80076f7180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T17:32:32.803 INFO:tasks.ceph.c2.osd.0.smithi153.stderr:2024-03-04T17:32:32.802+0000 7f0495e247c0 -1 Falling back to public interface 2024-03-04T17:32:32.804 INFO:tasks.ceph.c2.osd.2.smithi153.stderr:2024-03-04T17:32:32.802+0000 7f28c66d87c0 -1 Falling back to public interface 2024-03-04T17:32:32.804 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:32.802+0000 7f80076f7180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T17:32:32.815 INFO:tasks.ceph.c2.osd.1.smithi153.stderr:2024-03-04T17:32:32.814+0000 7f4167a5c7c0 -1 Falling back to public interface 2024-03-04T17:32:32.849 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-04T17:32:32.849 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":5,"fsid":"5f1d29fe-dcec-4834-91dc-1efe93fdf67a","created":"2024-03-04T17:32:31.222798+0000","modified":"2024-03-04T17:32:32.116983+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T17:32:32.116983+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c852b2f-e2ad-474b-88e4-d9e330c05f88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ef7e72e-dab4-4fbf-b7d4-e82b9d9fd3e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"105fb43b-575f-4bf1-89b4-be8d155bfa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T17:32:32.862 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:32.862+0000 7f80076f7180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T17:32:32.864 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-04T17:32:32.940 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:32.938+0000 7f80076f7180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T17:32:32.992 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:32.990+0000 7f80076f7180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T17:32:33.069 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:33.066+0000 7f80076f7180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T17:32:33.123 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:33.122+0000 7f80076f7180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T17:32:33.272 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:33.270+0000 7f80076f7180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T17:32:33.334 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:33.330+0000 7f80076f7180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T17:32:33.969 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-04T17:32:33.969 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-04T17:32:33.969 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr: from numpy import show_config as show_numpy_config 2024-03-04T17:32:34.042 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:34.042+0000 7f80076f7180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T17:32:34.220 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:34.218+0000 7f80076f7180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T17:32:34.374 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:34.374+0000 7f80076f7180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T17:32:34.447 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:34.446+0000 7f80076f7180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T17:32:34.580 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:34.578+0000 7f80076f7180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T17:32:35.211 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:35.210+0000 7f80076f7180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T17:32:35.289 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:35.286+0000 7f80076f7180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T17:32:35.468 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:35.466+0000 7f80076f7180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T17:32:35.542 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:35.538+0000 7f80076f7180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T17:32:35.589 INFO:tasks.ceph.c2.osd.0.smithi153.stderr:2024-03-04T17:32:35.586+0000 7f0495e247c0 -1 osd.0 0 log_to_monitors true 2024-03-04T17:32:35.597 INFO:tasks.ceph.c2.osd.2.smithi153.stderr:2024-03-04T17:32:35.594+0000 7f28c66d87c0 -1 osd.2 0 log_to_monitors true 2024-03-04T17:32:35.841 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:35.838+0000 7f80076f7180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T17:32:35.892 INFO:tasks.ceph.c2.osd.1.smithi153.stderr:2024-03-04T17:32:35.890+0000 7f4167a5c7c0 -1 osd.1 0 log_to_monitors true 2024-03-04T17:32:35.906 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:35.902+0000 7f80076f7180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T17:32:37.012 INFO:tasks.ceph.c2.mgr.x.smithi153.stderr:2024-03-04T17:32:37.010+0000 7f7faeb41640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-04T17:32:37.722 INFO:tasks.ceph.c2.osd.0.smithi153.stderr:2024-03-04T17:32:37.718+0000 7f0491dcd640 -1 osd.0 0 waiting for initial osdmap 2024-03-04T17:32:37.753 INFO:tasks.ceph.c2.osd.0.smithi153.stderr:2024-03-04T17:32:37.750+0000 7f0489c02640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T17:32:37.765 INFO:tasks.ceph.c2.osd.2.smithi153.stderr:2024-03-04T17:32:37.762+0000 7f28c2681640 -1 osd.2 0 waiting for initial osdmap 2024-03-04T17:32:37.807 INFO:tasks.ceph.c2.osd.2.smithi153.stderr:2024-03-04T17:32:37.806+0000 7f28ba4b6640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T17:32:38.488 INFO:tasks.ceph.c2.osd.1.smithi153.stderr:2024-03-04T17:32:38.486+0000 7f4163a05640 -1 osd.1 0 waiting for initial osdmap 2024-03-04T17:32:38.512 INFO:tasks.ceph.c2.osd.1.smithi153.stderr:2024-03-04T17:32:38.510+0000 7f415b83a640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T17:32:39.167 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-03-04T17:32:39.460 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-04T17:32:39.460 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":9,"fsid":"5f1d29fe-dcec-4834-91dc-1efe93fdf67a","created":"2024-03-04T17:32:31.222798+0000","modified":"2024-03-04T17:32:38.725891+0000","last_up_change":"2024-03-04T17:32:38.725891+0000","last_in_change":"2024-03-04T17:32:32.116983+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c852b2f-e2ad-474b-88e4-d9e330c05f88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2612165542},{"type":"v1","addr":"172.21.15.153:6801","nonce":2612165542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":2612165542},{"type":"v1","addr":"172.21.15.153:6805","nonce":2612165542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6811","nonce":2612165542},{"type":"v1","addr":"172.21.15.153:6813","nonce":2612165542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":2612165542},{"type":"v1","addr":"172.21.15.153:6809","nonce":2612165542}]},"public_addr":"172.21.15.153:6801/2612165542","cluster_addr":"172.21.15.153:6805/2612165542","heartbeat_back_addr":"172.21.15.153:6813/2612165542","heartbeat_front_addr":"172.21.15.153:6809/2612165542","state":["exists","up"]},{"osd":1,"uuid":"6ef7e72e-dab4-4fbf-b7d4-e82b9d9fd3e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2249861047},{"type":"v1","addr":"172.21.15.153:6817","nonce":2249861047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2249861047},{"type":"v1","addr":"172.21.15.153:6819","nonce":2249861047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2249861047},{"type":"v1","addr":"172.21.15.153:6823","nonce":2249861047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2249861047},{"type":"v1","addr":"172.21.15.153:6821","nonce":2249861047}]},"public_addr":"172.21.15.153:6817/2249861047","cluster_addr":"172.21.15.153:6819/2249861047","heartbeat_back_addr":"172.21.15.153:6823/2249861047","heartbeat_front_addr":"172.21.15.153:6821/2249861047","state":["exists","up"]},{"osd":2,"uuid":"105fb43b-575f-4bf1-89b4-be8d155bfa92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3031001318},{"type":"v1","addr":"172.21.15.153:6804","nonce":3031001318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3031001318},{"type":"v1","addr":"172.21.15.153:6808","nonce":3031001318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3031001318},{"type":"v1","addr":"172.21.15.153:6815","nonce":3031001318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3031001318},{"type":"v1","addr":"172.21.15.153:6812","nonce":3031001318}]},"public_addr":"172.21.15.153:6804/3031001318","cluster_addr":"172.21.15.153:6808/3031001318","heartbeat_back_addr":"172.21.15.153:6815/3031001318","heartbeat_front_addr":"172.21.15.153:6812/3031001318","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T17:32:39.477 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-03-04T17:32:39.477 INFO:tasks.ceph:Creating RBD pool 2024-03-04T17:32:39.477 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster c2 osd pool create rbd 8 2024-03-04T17:32:39.816 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'rbd' created 2024-03-04T17:32:39.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-04T17:32:40.821 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-04T17:32:40.848 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2024-03-04T17:32:40.849 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2024-03-04T17:32:40.849 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2024-03-04T17:32:40.849 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2024-03-04T17:32:40.849 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-04T17:32:41.181 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:41.197 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":633132477},{"type":"v1","addr":"172.21.15.153:6825","nonce":633132477}]},"active_addr":"172.21.15.153:6825/633132477","active_change":"2024-03-04T17:32:35.924293+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3625066785}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":268949688}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":15667080}]}]} 2024-03-04T17:32:41.202 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2024-03-04T17:32:41.202 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2024-03-04T17:32:41.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-03-04T17:32:41.504 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:41.506 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":11,"fsid":"5f1d29fe-dcec-4834-91dc-1efe93fdf67a","created":"2024-03-04T17:32:31.222798+0000","modified":"2024-03-04T17:32:40.817108+0000","last_up_change":"2024-03-04T17:32:38.725891+0000","last_in_change":"2024-03-04T17:32:32.116983+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-03-04T17:32:39.814948+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-03-04T17:32:39.988516+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6c852b2f-e2ad-474b-88e4-d9e330c05f88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2612165542},{"type":"v1","addr":"172.21.15.153:6801","nonce":2612165542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":2612165542},{"type":"v1","addr":"172.21.15.153:6805","nonce":2612165542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6811","nonce":2612165542},{"type":"v1","addr":"172.21.15.153:6813","nonce":2612165542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":2612165542},{"type":"v1","addr":"172.21.15.153:6809","nonce":2612165542}]},"public_addr":"172.21.15.153:6801/2612165542","cluster_addr":"172.21.15.153:6805/2612165542","heartbeat_back_addr":"172.21.15.153:6813/2612165542","heartbeat_front_addr":"172.21.15.153:6809/2612165542","state":["exists","up"]},{"osd":1,"uuid":"6ef7e72e-dab4-4fbf-b7d4-e82b9d9fd3e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2249861047},{"type":"v1","addr":"172.21.15.153:6817","nonce":2249861047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2249861047},{"type":"v1","addr":"172.21.15.153:6819","nonce":2249861047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2249861047},{"type":"v1","addr":"172.21.15.153:6823","nonce":2249861047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2249861047},{"type":"v1","addr":"172.21.15.153:6821","nonce":2249861047}]},"public_addr":"172.21.15.153:6817/2249861047","cluster_addr":"172.21.15.153:6819/2249861047","heartbeat_back_addr":"172.21.15.153:6823/2249861047","heartbeat_front_addr":"172.21.15.153:6821/2249861047","state":["exists","up"]},{"osd":2,"uuid":"105fb43b-575f-4bf1-89b4-be8d155bfa92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3031001318},{"type":"v1","addr":"172.21.15.153:6804","nonce":3031001318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3031001318},{"type":"v1","addr":"172.21.15.153:6808","nonce":3031001318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3031001318},{"type":"v1","addr":"172.21.15.153:6815","nonce":3031001318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3031001318},{"type":"v1","addr":"172.21.15.153:6812","nonce":3031001318}]},"public_addr":"172.21.15.153:6804/3031001318","cluster_addr":"172.21.15.153:6808/3031001318","heartbeat_back_addr":"172.21.15.153:6815/3031001318","heartbeat_front_addr":"172.21.15.153:6812/3031001318","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T17:32:36.614797+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T17:32:36.907616+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T17:32:36.560247+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T17:32:41.522 INFO:tasks.ceph.ceph_manager.c2:all up! 2024-03-04T17:32:41.522 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-03-04T17:32:41.817 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:41.817 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":11,"fsid":"5f1d29fe-dcec-4834-91dc-1efe93fdf67a","created":"2024-03-04T17:32:31.222798+0000","modified":"2024-03-04T17:32:40.817108+0000","last_up_change":"2024-03-04T17:32:38.725891+0000","last_in_change":"2024-03-04T17:32:32.116983+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-03-04T17:32:39.814948+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-03-04T17:32:39.988516+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6c852b2f-e2ad-474b-88e4-d9e330c05f88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2612165542},{"type":"v1","addr":"172.21.15.153:6801","nonce":2612165542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":2612165542},{"type":"v1","addr":"172.21.15.153:6805","nonce":2612165542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6811","nonce":2612165542},{"type":"v1","addr":"172.21.15.153:6813","nonce":2612165542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":2612165542},{"type":"v1","addr":"172.21.15.153:6809","nonce":2612165542}]},"public_addr":"172.21.15.153:6801/2612165542","cluster_addr":"172.21.15.153:6805/2612165542","heartbeat_back_addr":"172.21.15.153:6813/2612165542","heartbeat_front_addr":"172.21.15.153:6809/2612165542","state":["exists","up"]},{"osd":1,"uuid":"6ef7e72e-dab4-4fbf-b7d4-e82b9d9fd3e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2249861047},{"type":"v1","addr":"172.21.15.153:6817","nonce":2249861047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2249861047},{"type":"v1","addr":"172.21.15.153:6819","nonce":2249861047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2249861047},{"type":"v1","addr":"172.21.15.153:6823","nonce":2249861047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2249861047},{"type":"v1","addr":"172.21.15.153:6821","nonce":2249861047}]},"public_addr":"172.21.15.153:6817/2249861047","cluster_addr":"172.21.15.153:6819/2249861047","heartbeat_back_addr":"172.21.15.153:6823/2249861047","heartbeat_front_addr":"172.21.15.153:6821/2249861047","state":["exists","up"]},{"osd":2,"uuid":"105fb43b-575f-4bf1-89b4-be8d155bfa92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3031001318},{"type":"v1","addr":"172.21.15.153:6804","nonce":3031001318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3031001318},{"type":"v1","addr":"172.21.15.153:6808","nonce":3031001318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3031001318},{"type":"v1","addr":"172.21.15.153:6815","nonce":3031001318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3031001318},{"type":"v1","addr":"172.21.15.153:6812","nonce":3031001318}]},"public_addr":"172.21.15.153:6804/3031001318","cluster_addr":"172.21.15.153:6808/3031001318","heartbeat_back_addr":"172.21.15.153:6815/3031001318","heartbeat_front_addr":"172.21.15.153:6812/3031001318","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T17:32:36.614797+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T17:32:36.907616+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T17:32:36.560247+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T17:32:41.838 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2024-03-04T17:32:41.838 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2024-03-04T17:32:41.839 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2024-03-04T17:32:42.007 INFO:teuthology.orchestra.run.smithi153.stdout:38654705667 2024-03-04T17:32:42.008 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-03-04T17:32:42.010 INFO:teuthology.orchestra.run.smithi153.stdout:38654705667 2024-03-04T17:32:42.011 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-03-04T17:32:42.072 INFO:teuthology.orchestra.run.smithi153.stdout:38654705668 2024-03-04T17:32:42.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-03-04T17:32:42.322 INFO:teuthology.orchestra.run.smithi153.stdout:38654705666 2024-03-04T17:32:42.344 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.1 2024-03-04T17:32:42.345 INFO:teuthology.orchestra.run.smithi153.stdout:38654705666 2024-03-04T17:32:42.362 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.2 2024-03-04T17:32:42.400 INFO:teuthology.orchestra.run.smithi153.stdout:38654705667 2024-03-04T17:32:42.416 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705667 for osd.0 2024-03-04T17:32:43.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-03-04T17:32:43.363 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-03-04T17:32:43.417 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-03-04T17:32:43.671 INFO:teuthology.orchestra.run.smithi153.stdout:38654705666 2024-03-04T17:32:43.677 INFO:teuthology.orchestra.run.smithi153.stdout:38654705666 2024-03-04T17:32:43.694 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.2 2024-03-04T17:32:43.700 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.1 2024-03-04T17:32:43.743 INFO:teuthology.orchestra.run.smithi153.stdout:38654705667 2024-03-04T17:32:43.759 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705667 for osd.0 2024-03-04T17:32:44.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-03-04T17:32:44.701 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-03-04T17:32:44.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-03-04T17:32:44.998 INFO:teuthology.orchestra.run.smithi153.stdout:38654705667 2024-03-04T17:32:45.021 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705667 for osd.2 2024-03-04T17:32:45.021 DEBUG:teuthology.parallel:result is None 2024-03-04T17:32:45.046 INFO:teuthology.orchestra.run.smithi153.stdout:38654705667 2024-03-04T17:32:45.063 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705667 for osd.1 2024-03-04T17:32:45.063 DEBUG:teuthology.parallel:result is None 2024-03-04T17:32:45.071 INFO:teuthology.orchestra.run.smithi153.stdout:38654705668 2024-03-04T17:32:45.088 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705668 for osd.0 2024-03-04T17:32:45.088 DEBUG:teuthology.parallel:result is None 2024-03-04T17:32:45.088 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-03-04T17:32:45.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-03-04T17:32:45.389 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:45.389 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-03-04T17:32:45.405 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-03-04T17:32:43.930662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81356,"kb_used_data":288,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314491444,"statfs":{"total":322122547200,"available":322039238656,"internally_reserved":0,"allocated":294912,"data_stored":71994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.101641"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.825968+0000","last_change":"2024-03-04T17:32:40.832635+0000","last_active":"2024-03-04T17:32:41.825968+0000","last_peered":"2024-03-04T17:32:41.825968+0000","last_clean":"2024-03-04T17:32:41.825968+0000","last_became_active":"2024-03-04T17:32:40.832052+0000","last_became_peered":"2024-03-04T17:32:40.832052+0000","last_unstale":"2024-03-04T17:32:41.825968+0000","last_undegraded":"2024-03-04T17:32:41.825968+0000","last_fullsized":"2024-03-04T17:32:41.825968+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:20:27.011319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.825374+0000","last_change":"2024-03-04T17:32:40.834953+0000","last_active":"2024-03-04T17:32:41.825374+0000","last_peered":"2024-03-04T17:32:41.825374+0000","last_clean":"2024-03-04T17:32:41.825374+0000","last_became_active":"2024-03-04T17:32:40.834671+0000","last_became_peered":"2024-03-04T17:32:40.834671+0000","last_unstale":"2024-03-04T17:32:41.825374+0000","last_undegraded":"2024-03-04T17:32:41.825374+0000","last_fullsized":"2024-03-04T17:32:41.825374+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:15:57.396091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.827501+0000","last_change":"2024-03-04T17:32:40.837451+0000","last_active":"2024-03-04T17:32:41.827501+0000","last_peered":"2024-03-04T17:32:41.827501+0000","last_clean":"2024-03-04T17:32:41.827501+0000","last_became_active":"2024-03-04T17:32:40.837083+0000","last_became_peered":"2024-03-04T17:32:40.837083+0000","last_unstale":"2024-03-04T17:32:41.827501+0000","last_undegraded":"2024-03-04T17:32:41.827501+0000","last_fullsized":"2024-03-04T17:32:41.827501+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:20:17.185582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.825350+0000","last_change":"2024-03-04T17:32:40.835218+0000","last_active":"2024-03-04T17:32:41.825350+0000","last_peered":"2024-03-04T17:32:41.825350+0000","last_clean":"2024-03-04T17:32:41.825350+0000","last_became_active":"2024-03-04T17:32:40.834742+0000","last_became_peered":"2024-03-04T17:32:40.834742+0000","last_unstale":"2024-03-04T17:32:41.825350+0000","last_undegraded":"2024-03-04T17:32:41.825350+0000","last_fullsized":"2024-03-04T17:32:41.825350+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:58:19.545163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'64","reported_seq":97,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.938005+0000","last_change":"2024-03-04T17:32:41.831289+0000","last_active":"2024-03-04T17:32:41.938005+0000","last_peered":"2024-03-04T17:32:41.938005+0000","last_clean":"2024-03-04T17:32:41.938005+0000","last_became_active":"2024-03-04T17:32:41.830153+0000","last_became_peered":"2024-03-04T17:32:41.830153+0000","last_unstale":"2024-03-04T17:32:41.938005+0000","last_undegraded":"2024-03-04T17:32:41.938005+0000","last_fullsized":"2024-03-04T17:32:41.938005+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:40.817108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:40.817108+0000","last_clean_scrub_stamp":"2024-03-04T17:32:40.817108+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:55.961919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.825589+0000","last_change":"2024-03-04T17:32:40.832413+0000","last_active":"2024-03-04T17:32:41.825589+0000","last_peered":"2024-03-04T17:32:41.825589+0000","last_clean":"2024-03-04T17:32:41.825589+0000","last_became_active":"2024-03-04T17:32:40.831996+0000","last_became_peered":"2024-03-04T17:32:40.831996+0000","last_unstale":"2024-03-04T17:32:41.825589+0000","last_undegraded":"2024-03-04T17:32:41.825589+0000","last_fullsized":"2024-03-04T17:32:41.825589+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:06:40.760741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-04T17:32:40.839605+0000","last_change":"2024-03-04T17:32:40.832329+0000","last_active":"2024-03-04T17:32:40.839605+0000","last_peered":"2024-03-04T17:32:40.839605+0000","last_clean":"2024-03-04T17:32:40.839605+0000","last_became_active":"2024-03-04T17:32:40.832017+0000","last_became_peered":"2024-03-04T17:32:40.832017+0000","last_unstale":"2024-03-04T17:32:40.839605+0000","last_undegraded":"2024-03-04T17:32:40.839605+0000","last_fullsized":"2024-03-04T17:32:40.839605+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:05.150446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.827366+0000","last_change":"2024-03-04T17:32:40.837668+0000","last_active":"2024-03-04T17:32:41.827366+0000","last_peered":"2024-03-04T17:32:41.827366+0000","last_clean":"2024-03-04T17:32:41.827366+0000","last_became_active":"2024-03-04T17:32:40.837174+0000","last_became_peered":"2024-03-04T17:32:40.837174+0000","last_unstale":"2024-03-04T17:32:41.827366+0000","last_undegraded":"2024-03-04T17:32:41.827366+0000","last_fullsized":"2024-03-04T17:32:41.827366+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:37:20.261444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.825307+0000","last_change":"2024-03-04T17:32:40.835352+0000","last_active":"2024-03-04T17:32:41.825307+0000","last_peered":"2024-03-04T17:32:41.825307+0000","last_clean":"2024-03-04T17:32:41.825307+0000","last_became_active":"2024-03-04T17:32:40.834889+0000","last_became_peered":"2024-03-04T17:32:40.834889+0000","last_unstale":"2024-03-04T17:32:41.825307+0000","last_undegraded":"2024-03-04T17:32:41.825307+0000","last_fullsized":"2024-03-04T17:32:41.825307+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:46:58.754716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925696,"data_stored":918560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830484,"statfs":{"total":107374182400,"available":107346415616,"internally_reserved":0,"allocated":98304,"data_stored":23998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":23998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830484,"statfs":{"total":107374182400,"available":107346415616,"internally_reserved":0,"allocated":98304,"data_stored":23998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T17:32:45.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-03-04T17:32:45.678 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:45.678 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-03-04T17:32:45.695 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-03-04T17:32:43.930662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81356,"kb_used_data":288,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314491444,"statfs":{"total":322122547200,"available":322039238656,"internally_reserved":0,"allocated":294912,"data_stored":71994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.101641"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.825968+0000","last_change":"2024-03-04T17:32:40.832635+0000","last_active":"2024-03-04T17:32:41.825968+0000","last_peered":"2024-03-04T17:32:41.825968+0000","last_clean":"2024-03-04T17:32:41.825968+0000","last_became_active":"2024-03-04T17:32:40.832052+0000","last_became_peered":"2024-03-04T17:32:40.832052+0000","last_unstale":"2024-03-04T17:32:41.825968+0000","last_undegraded":"2024-03-04T17:32:41.825968+0000","last_fullsized":"2024-03-04T17:32:41.825968+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:20:27.011319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.825374+0000","last_change":"2024-03-04T17:32:40.834953+0000","last_active":"2024-03-04T17:32:41.825374+0000","last_peered":"2024-03-04T17:32:41.825374+0000","last_clean":"2024-03-04T17:32:41.825374+0000","last_became_active":"2024-03-04T17:32:40.834671+0000","last_became_peered":"2024-03-04T17:32:40.834671+0000","last_unstale":"2024-03-04T17:32:41.825374+0000","last_undegraded":"2024-03-04T17:32:41.825374+0000","last_fullsized":"2024-03-04T17:32:41.825374+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:15:57.396091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.827501+0000","last_change":"2024-03-04T17:32:40.837451+0000","last_active":"2024-03-04T17:32:41.827501+0000","last_peered":"2024-03-04T17:32:41.827501+0000","last_clean":"2024-03-04T17:32:41.827501+0000","last_became_active":"2024-03-04T17:32:40.837083+0000","last_became_peered":"2024-03-04T17:32:40.837083+0000","last_unstale":"2024-03-04T17:32:41.827501+0000","last_undegraded":"2024-03-04T17:32:41.827501+0000","last_fullsized":"2024-03-04T17:32:41.827501+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:20:17.185582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.825350+0000","last_change":"2024-03-04T17:32:40.835218+0000","last_active":"2024-03-04T17:32:41.825350+0000","last_peered":"2024-03-04T17:32:41.825350+0000","last_clean":"2024-03-04T17:32:41.825350+0000","last_became_active":"2024-03-04T17:32:40.834742+0000","last_became_peered":"2024-03-04T17:32:40.834742+0000","last_unstale":"2024-03-04T17:32:41.825350+0000","last_undegraded":"2024-03-04T17:32:41.825350+0000","last_fullsized":"2024-03-04T17:32:41.825350+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:58:19.545163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'64","reported_seq":97,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.938005+0000","last_change":"2024-03-04T17:32:41.831289+0000","last_active":"2024-03-04T17:32:41.938005+0000","last_peered":"2024-03-04T17:32:41.938005+0000","last_clean":"2024-03-04T17:32:41.938005+0000","last_became_active":"2024-03-04T17:32:41.830153+0000","last_became_peered":"2024-03-04T17:32:41.830153+0000","last_unstale":"2024-03-04T17:32:41.938005+0000","last_undegraded":"2024-03-04T17:32:41.938005+0000","last_fullsized":"2024-03-04T17:32:41.938005+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:40.817108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:40.817108+0000","last_clean_scrub_stamp":"2024-03-04T17:32:40.817108+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:55.961919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.825589+0000","last_change":"2024-03-04T17:32:40.832413+0000","last_active":"2024-03-04T17:32:41.825589+0000","last_peered":"2024-03-04T17:32:41.825589+0000","last_clean":"2024-03-04T17:32:41.825589+0000","last_became_active":"2024-03-04T17:32:40.831996+0000","last_became_peered":"2024-03-04T17:32:40.831996+0000","last_unstale":"2024-03-04T17:32:41.825589+0000","last_undegraded":"2024-03-04T17:32:41.825589+0000","last_fullsized":"2024-03-04T17:32:41.825589+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:06:40.760741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-04T17:32:40.839605+0000","last_change":"2024-03-04T17:32:40.832329+0000","last_active":"2024-03-04T17:32:40.839605+0000","last_peered":"2024-03-04T17:32:40.839605+0000","last_clean":"2024-03-04T17:32:40.839605+0000","last_became_active":"2024-03-04T17:32:40.832017+0000","last_became_peered":"2024-03-04T17:32:40.832017+0000","last_unstale":"2024-03-04T17:32:40.839605+0000","last_undegraded":"2024-03-04T17:32:40.839605+0000","last_fullsized":"2024-03-04T17:32:40.839605+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:05.150446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.827366+0000","last_change":"2024-03-04T17:32:40.837668+0000","last_active":"2024-03-04T17:32:41.827366+0000","last_peered":"2024-03-04T17:32:41.827366+0000","last_clean":"2024-03-04T17:32:41.827366+0000","last_became_active":"2024-03-04T17:32:40.837174+0000","last_became_peered":"2024-03-04T17:32:40.837174+0000","last_unstale":"2024-03-04T17:32:41.827366+0000","last_undegraded":"2024-03-04T17:32:41.827366+0000","last_fullsized":"2024-03-04T17:32:41.827366+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:37:20.261444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T17:32:41.825307+0000","last_change":"2024-03-04T17:32:40.835352+0000","last_active":"2024-03-04T17:32:41.825307+0000","last_peered":"2024-03-04T17:32:41.825307+0000","last_clean":"2024-03-04T17:32:41.825307+0000","last_became_active":"2024-03-04T17:32:40.834889+0000","last_became_peered":"2024-03-04T17:32:40.834889+0000","last_unstale":"2024-03-04T17:32:41.825307+0000","last_undegraded":"2024-03-04T17:32:41.825307+0000","last_fullsized":"2024-03-04T17:32:41.825307+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T17:32:39.815081+0000","last_clean_scrub_stamp":"2024-03-04T17:32:39.815081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:46:58.754716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925696,"data_stored":918560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830484,"statfs":{"total":107374182400,"available":107346415616,"internally_reserved":0,"allocated":98304,"data_stored":23998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":23998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830484,"statfs":{"total":107374182400,"available":107346415616,"internally_reserved":0,"allocated":98304,"data_stored":23998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T17:32:45.696 INFO:tasks.ceph.ceph_manager.c2:clean! 2024-03-04T17:32:45.696 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2024-03-04T17:32:45.696 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2024-03-04T17:32:45.696 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2024-03-04T17:32:45.992 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-04T17:32:45.992 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T17:32:46.009 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2024-03-04T17:32:46.009 INFO:teuthology.run_tasks:Running task rgw... 2024-03-04T17:32:46.023 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2024-03-04T17:32:46.024 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2024-03-04T17:32:46.024 INFO:tasks.rgw:Creating data pools 2024-03-04T17:32:46.024 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2024-03-04T17:32:46.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-03-04T17:32:47.178 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'default.rgw.buckets.data' created 2024-03-04T17:32:47.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-03-04T17:32:48.183 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-03-04T17:32:48.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-03-04T17:32:49.186 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'default.rgw.buckets.index' created 2024-03-04T17:32:49.223 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-03-04T17:32:50.196 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-03-04T17:32:50.248 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2024-03-04T17:32:50.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-03-04T17:32:50.537 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'default.rgw.buckets.data' already exists 2024-03-04T17:32:50.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-03-04T17:32:51.212 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-03-04T17:32:51.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-03-04T17:32:51.525 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'default.rgw.buckets.index' already exists 2024-03-04T17:32:51.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-03-04T17:32:52.220 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-03-04T17:32:52.245 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2024-03-04T17:32:52.245 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-03-04T17:32:52.989 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'default.rgw.buckets.data' created 2024-03-04T17:32:53.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-03-04T17:32:53.999 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-03-04T17:32:54.037 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-03-04T17:32:55.007 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'default.rgw.buckets.index' created 2024-03-04T17:32:55.040 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-03-04T17:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-03-04T17:32:56.055 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2024-03-04T17:32:56.056 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-03-04T17:32:56.341 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'default.rgw.buckets.data' already exists 2024-03-04T17:32:56.360 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-03-04T17:32:57.013 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-03-04T17:32:57.041 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-03-04T17:32:57.339 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'default.rgw.buckets.index' already exists 2024-03-04T17:32:57.357 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-03-04T17:32:58.030 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-03-04T17:32:58.056 DEBUG:tasks.rgw:Pools created 2024-03-04T17:32:58.057 INFO:tasks.rgw:Configuring compression type = random 2024-03-04T17:32:58.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2024-03-04T17:32:58.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2024-03-04T17:32:58.057 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2024-03-04T17:32:58.117 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:58.117 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:59.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.313+0000 7f31f9011980 20 rados->read ofs=0 len=0 2024-03-04T17:32:59.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.313+0000 7f31f9011980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:32:59.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.313+0000 7f31f9011980 20 realm 2024-03-04T17:32:59.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.313+0000 7f31f9011980 20 rados->read ofs=0 len=0 2024-03-04T17:32:59.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.313+0000 7f31f9011980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:32:59.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.313+0000 7f31f9011980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-04T17:32:59.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.313+0000 7f31f9011980 20 rados->read ofs=0 len=0 2024-03-04T17:32:59.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.313+0000 7f31f9011980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:32:59.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.313+0000 7f31f9011980 20 rados->read ofs=0 len=0 2024-03-04T17:32:59.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.313+0000 7f31f9011980 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T17:32:59.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.313+0000 7f31f9011980 20 rados->read ofs=0 len=0 2024-03-04T17:32:59.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.313+0000 7f31f9011980 20 rados_obj.operate() r=0 bl.length=875 2024-03-04T17:32:59.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.313+0000 7f31f9011980 20 searching for the correct realm 2024-03-04T17:32:59.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-04T17:32:59.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 RGWRados::pool_iterate: got zonegroup_info.5115bf61-7b35-4d01-b0d5-c52ef066fed3 2024-03-04T17:32:59.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 RGWRados::pool_iterate: got default.zone. 2024-03-04T17:32:59.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 RGWRados::pool_iterate: got zone_info.507111a3-5efa-4367-8b59-070a341c8a1f 2024-03-04T17:32:59.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-04T17:32:59.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-04T17:32:59.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 rados->read ofs=0 len=0 2024-03-04T17:32:59.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:32:59.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 rados->read ofs=0 len=0 2024-03-04T17:32:59.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T17:32:59.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 rados->read ofs=0 len=0 2024-03-04T17:32:59.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 rados_obj.operate() r=0 bl.length=398 2024-03-04T17:32:59.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 zone default found 2024-03-04T17:32:59.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 4 Realm: () 2024-03-04T17:32:59.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 4 ZoneGroup: default (5115bf61-7b35-4d01-b0d5-c52ef066fed3) 2024-03-04T17:32:59.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 4 Zone: default (507111a3-5efa-4367-8b59-070a341c8a1f) 2024-03-04T17:32:59.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-04T17:32:59.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 zonegroup default 2024-03-04T17:32:59.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.325+0000 7f31f9011980 20 rados->read ofs=0 len=0 2024-03-04T17:32:59.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.329+0000 7f31f9011980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:32:59.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:32:59.329+0000 7f31f9011980 20 rados->read ofs=0 len=0 2024-03-04T17:33:01.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:01.277+0000 7f31f9011980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:01.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:01.277+0000 7f31f9011980 20 rados->read ofs=0 len=0 2024-03-04T17:33:01.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:01.277+0000 7f31f9011980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:01.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:01.277+0000 7f31f9011980 20 started sync module instance, tier type = 2024-03-04T17:33:01.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:01.277+0000 7f31f9011980 20 started zone id=507111a3-5efa-4367-8b59-070a341c8a1f (name=default) with tier type = 2024-03-04T17:33:03.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 add_watcher() i=0 2024-03-04T17:33:03.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 add_watcher() i=1 2024-03-04T17:33:03.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 add_watcher() i=2 2024-03-04T17:33:03.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 add_watcher() i=3 2024-03-04T17:33:03.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 add_watcher() i=4 2024-03-04T17:33:03.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 add_watcher() i=5 2024-03-04T17:33:03.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 add_watcher() i=6 2024-03-04T17:33:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 add_watcher() i=7 2024-03-04T17:33:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 2 all 8 watchers are set, enabling cache 2024-03-04T17:33:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-03-04T17:33:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-03-04T17:33:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T17:33:03.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-03-04T17:33:03.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-03-04T17:33:03.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T17:33:03.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-03-04T17:33:03.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-03-04T17:33:03.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.341+0000 7f31f9011980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T17:33:03.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.345+0000 7f31f9011980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-03-04T17:33:03.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.345+0000 7f31f9011980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-03-04T17:33:03.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.345+0000 7f31f9011980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T17:33:03.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.345+0000 7f31f9011980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-03-04T17:33:03.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.345+0000 7f31f9011980 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1211 entering 2024-03-04T17:33:03.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.345+0000 7f31f9011980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T17:33:03.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.353+0000 7f30a97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T17:33:03.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.353+0000 7f31f9011980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T17:33:03.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:03.353+0000 7f31f9011980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T17:33:06.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.289+0000 7f31f9011980 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-03-04T17:33:06.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.289+0000 7f31f9011980 5 note: GC not initialized 2024-03-04T17:33:06.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.301+0000 7f3013fff640 20 reqs_thread_entry: start 2024-03-04T17:33:06.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.301+0000 7f31f9011980 20 init_complete bucket index max shards: 11 2024-03-04T17:33:06.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.301+0000 7f31f9011980 20 Filter name: none 2024-03-04T17:33:06.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.301+0000 7f3011ffb640 20 reqs_thread_entry: start 2024-03-04T17:33:06.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.329+0000 7f30a97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T17:33:06.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.333+0000 7f31f9011980 20 remove_watcher() i=0 2024-03-04T17:33:06.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.333+0000 7f31f9011980 2 removed watcher, disabling cache 2024-03-04T17:33:06.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.333+0000 7f31f9011980 20 remove_watcher() i=1 2024-03-04T17:33:06.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.333+0000 7f31f9011980 20 remove_watcher() i=2 2024-03-04T17:33:06.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.337+0000 7f31f9011980 20 remove_watcher() i=3 2024-03-04T17:33:06.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.337+0000 7f31f9011980 20 remove_watcher() i=4 2024-03-04T17:33:06.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.337+0000 7f31f9011980 20 remove_watcher() i=5 2024-03-04T17:33:06.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.341+0000 7f31f9011980 20 remove_watcher() i=6 2024-03-04T17:33:06.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.341+0000 7f31f9011980 20 remove_watcher() i=7 2024-03-04T17:33:06.359 INFO:teuthology.orchestra.run.smithi136.stdout:[] 2024-03-04T17:33:06.359 DEBUG:tasks.util.rgw: json result: [] 2024-03-04T17:33:06.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T17:33:06.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T17:33:06.360 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-03-04T17:33:06.400 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:06.400 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:06.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.425+0000 7fa17ea44980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T17:33:06.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.425+0000 7fa17ea44980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T17:33:06.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.429+0000 7f9fea7fc640 20 reqs_thread_entry: start 2024-03-04T17:33:06.447 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"507111a3-5efa-4367-8b59-070a341c8a1f","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2024-03-04T17:33:06.447 DEBUG:tasks.util.rgw: json result: {'id': '507111a3-5efa-4367-8b59-070a341c8a1f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-03-04T17:33:06.447 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2024-03-04T17:33:06.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2024-03-04T17:33:06.448 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2024-03-04T17:33:06.521 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:06.550 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:06.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.553+0000 7f47b10c0980 20 rados->read ofs=0 len=0 2024-03-04T17:33:06.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.553+0000 7f47b10c0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:06.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.553+0000 7f47b10c0980 20 realm 2024-03-04T17:33:06.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.553+0000 7f47b10c0980 20 rados->read ofs=0 len=0 2024-03-04T17:33:06.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.553+0000 7f47b10c0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:06.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.553+0000 7f47b10c0980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-04T17:33:06.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.553+0000 7f47b10c0980 20 rados->read ofs=0 len=0 2024-03-04T17:33:06.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.553+0000 7f47b10c0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:06.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.553+0000 7f47b10c0980 20 rados->read ofs=0 len=0 2024-03-04T17:33:06.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.553+0000 7f47b10c0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T17:33:06.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.553+0000 7f47b10c0980 20 rados->read ofs=0 len=0 2024-03-04T17:33:06.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.553+0000 7f47b10c0980 20 rados_obj.operate() r=0 bl.length=891 2024-03-04T17:33:06.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.553+0000 7f47b10c0980 20 searching for the correct realm 2024-03-04T17:33:06.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.561+0000 7f47b10c0980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-04T17:33:06.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.561+0000 7f47b10c0980 20 RGWRados::pool_iterate: got zonegroup_info.5115bf61-7b35-4d01-b0d5-c52ef066fed3 2024-03-04T17:33:06.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.561+0000 7f47b10c0980 20 RGWRados::pool_iterate: got default.zone. 2024-03-04T17:33:06.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.561+0000 7f47b10c0980 20 RGWRados::pool_iterate: got zone_info.507111a3-5efa-4367-8b59-070a341c8a1f 2024-03-04T17:33:06.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.561+0000 7f47b10c0980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-04T17:33:06.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.561+0000 7f47b10c0980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-04T17:33:06.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.561+0000 7f47b10c0980 20 rados->read ofs=0 len=0 2024-03-04T17:33:06.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.561+0000 7f47b10c0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:06.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.561+0000 7f47b10c0980 20 rados->read ofs=0 len=0 2024-03-04T17:33:06.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T17:33:06.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 20 rados->read ofs=0 len=0 2024-03-04T17:33:06.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 20 rados_obj.operate() r=0 bl.length=398 2024-03-04T17:33:06.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 20 zone default found 2024-03-04T17:33:06.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 4 Realm: () 2024-03-04T17:33:06.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 4 ZoneGroup: default (5115bf61-7b35-4d01-b0d5-c52ef066fed3) 2024-03-04T17:33:06.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 4 Zone: default (507111a3-5efa-4367-8b59-070a341c8a1f) 2024-03-04T17:33:06.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-04T17:33:06.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 20 zonegroup default 2024-03-04T17:33:06.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 20 rados->read ofs=0 len=0 2024-03-04T17:33:06.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:06.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 20 rados->read ofs=0 len=0 2024-03-04T17:33:06.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:06.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 20 rados->read ofs=0 len=0 2024-03-04T17:33:06.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:06.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 20 started sync module instance, tier type = 2024-03-04T17:33:06.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.565+0000 7f47b10c0980 20 started zone id=507111a3-5efa-4367-8b59-070a341c8a1f (name=default) with tier type = 2024-03-04T17:33:06.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.577+0000 7f47b10c0980 20 add_watcher() i=0 2024-03-04T17:33:06.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.577+0000 7f47b10c0980 20 add_watcher() i=1 2024-03-04T17:33:06.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.577+0000 7f47b10c0980 20 add_watcher() i=2 2024-03-04T17:33:06.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.577+0000 7f47b10c0980 20 add_watcher() i=3 2024-03-04T17:33:06.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.577+0000 7f47b10c0980 20 add_watcher() i=4 2024-03-04T17:33:06.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.577+0000 7f47b10c0980 20 add_watcher() i=5 2024-03-04T17:33:06.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.577+0000 7f47b10c0980 20 add_watcher() i=6 2024-03-04T17:33:06.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.577+0000 7f47b10c0980 20 add_watcher() i=7 2024-03-04T17:33:06.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.577+0000 7f47b10c0980 2 all 8 watchers are set, enabling cache 2024-03-04T17:33:06.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.581+0000 7f46657e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T17:33:06.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.581+0000 7f47b10c0980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T17:33:06.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.581+0000 7f47b10c0980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T17:33:06.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.581+0000 7f47b10c0980 5 note: GC not initialized 2024-03-04T17:33:06.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.581+0000 7f45dffff640 20 reqs_thread_entry: start 2024-03-04T17:33:06.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.581+0000 7f47b10c0980 20 init_complete bucket index max shards: 11 2024-03-04T17:33:06.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.581+0000 7f47b10c0980 20 Filter name: none 2024-03-04T17:33:06.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.581+0000 7f45ddffb640 20 reqs_thread_entry: start 2024-03-04T17:33:06.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.589+0000 7f46657e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T17:33:06.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.593+0000 7f47b10c0980 20 remove_watcher() i=0 2024-03-04T17:33:06.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.593+0000 7f47b10c0980 2 removed watcher, disabling cache 2024-03-04T17:33:06.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.593+0000 7f47b10c0980 20 remove_watcher() i=1 2024-03-04T17:33:06.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.593+0000 7f47b10c0980 20 remove_watcher() i=2 2024-03-04T17:33:06.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.593+0000 7f47b10c0980 20 remove_watcher() i=3 2024-03-04T17:33:06.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.597+0000 7f47b10c0980 20 remove_watcher() i=4 2024-03-04T17:33:06.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.597+0000 7f47b10c0980 20 remove_watcher() i=5 2024-03-04T17:33:06.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.597+0000 7f47b10c0980 20 remove_watcher() i=6 2024-03-04T17:33:06.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.597+0000 7f47b10c0980 20 remove_watcher() i=7 2024-03-04T17:33:06.617 INFO:teuthology.orchestra.run.smithi136.stdout:[] 2024-03-04T17:33:06.617 DEBUG:tasks.util.rgw: json result: [] 2024-03-04T17:33:06.618 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T17:33:06.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T17:33:06.618 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-03-04T17:33:06.694 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:06.694 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:06.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.717+0000 7f10b8294980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T17:33:06.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.717+0000 7f10b8294980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T17:33:06.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:06.721+0000 7f0f78fe1640 20 reqs_thread_entry: start 2024-03-04T17:33:06.741 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"507111a3-5efa-4367-8b59-070a341c8a1f","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2024-03-04T17:33:06.742 DEBUG:tasks.util.rgw: json result: {'id': '507111a3-5efa-4367-8b59-070a341c8a1f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-03-04T17:33:06.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2024-03-04T17:33:06.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2024-03-04T17:33:06.743 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2024-03-04T17:33:06.885 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:06.886 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:08.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.149+0000 7f26752fd980 20 rados->read ofs=0 len=0 2024-03-04T17:33:08.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.149+0000 7f26752fd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:08.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.149+0000 7f26752fd980 20 realm 2024-03-04T17:33:08.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.149+0000 7f26752fd980 20 rados->read ofs=0 len=0 2024-03-04T17:33:08.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.149+0000 7f26752fd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:08.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.149+0000 7f26752fd980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-04T17:33:08.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.149+0000 7f26752fd980 20 rados->read ofs=0 len=0 2024-03-04T17:33:08.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.149+0000 7f26752fd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:08.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.149+0000 7f26752fd980 20 rados->read ofs=0 len=0 2024-03-04T17:33:08.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.149+0000 7f26752fd980 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T17:33:08.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.149+0000 7f26752fd980 20 rados->read ofs=0 len=0 2024-03-04T17:33:08.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.149+0000 7f26752fd980 20 rados_obj.operate() r=0 bl.length=875 2024-03-04T17:33:08.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.149+0000 7f26752fd980 20 searching for the correct realm 2024-03-04T17:33:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 RGWRados::pool_iterate: got zonegroup_info.be061770-ad07-43f4-8436-fd9c6ba14763 2024-03-04T17:33:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-04T17:33:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 RGWRados::pool_iterate: got zone_info.d4796dd9-b51d-4fc8-b394-749e777f8124 2024-03-04T17:33:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 RGWRados::pool_iterate: got default.zone. 2024-03-04T17:33:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-04T17:33:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-04T17:33:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 rados->read ofs=0 len=0 2024-03-04T17:33:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 rados->read ofs=0 len=0 2024-03-04T17:33:08.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T17:33:08.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 rados->read ofs=0 len=0 2024-03-04T17:33:08.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 rados_obj.operate() r=0 bl.length=398 2024-03-04T17:33:08.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 zone default found 2024-03-04T17:33:08.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 4 Realm: () 2024-03-04T17:33:08.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 4 ZoneGroup: default (be061770-ad07-43f4-8436-fd9c6ba14763) 2024-03-04T17:33:08.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 4 Zone: default (d4796dd9-b51d-4fc8-b394-749e777f8124) 2024-03-04T17:33:08.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-04T17:33:08.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 zonegroup default 2024-03-04T17:33:08.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.161+0000 7f26752fd980 20 rados->read ofs=0 len=0 2024-03-04T17:33:08.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.165+0000 7f26752fd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:08.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:08.165+0000 7f26752fd980 20 rados->read ofs=0 len=0 2024-03-04T17:33:10.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:10.105+0000 7f26752fd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:10.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:10.105+0000 7f26752fd980 20 rados->read ofs=0 len=0 2024-03-04T17:33:10.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:10.113+0000 7f26752fd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:10.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:10.113+0000 7f26752fd980 20 started sync module instance, tier type = 2024-03-04T17:33:10.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:10.113+0000 7f26752fd980 20 started zone id=d4796dd9-b51d-4fc8-b394-749e777f8124 (name=default) with tier type = 2024-03-04T17:33:12.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.169+0000 7f26752fd980 20 add_watcher() i=0 2024-03-04T17:33:12.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.169+0000 7f26752fd980 20 add_watcher() i=1 2024-03-04T17:33:12.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.169+0000 7f26752fd980 20 add_watcher() i=2 2024-03-04T17:33:12.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.169+0000 7f26752fd980 20 add_watcher() i=3 2024-03-04T17:33:12.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.169+0000 7f26752fd980 20 add_watcher() i=4 2024-03-04T17:33:12.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.169+0000 7f26752fd980 20 add_watcher() i=5 2024-03-04T17:33:12.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.169+0000 7f26752fd980 20 add_watcher() i=6 2024-03-04T17:33:12.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.173+0000 7f26752fd980 20 add_watcher() i=7 2024-03-04T17:33:12.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.173+0000 7f26752fd980 2 all 8 watchers are set, enabling cache 2024-03-04T17:33:12.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.173+0000 7f26752fd980 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-03-04T17:33:12.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.173+0000 7f26752fd980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-03-04T17:33:12.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.173+0000 7f26752fd980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T17:33:12.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.173+0000 7f26752fd980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-03-04T17:33:12.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.173+0000 7f26752fd980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-03-04T17:33:12.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.173+0000 7f26752fd980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T17:33:12.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.177+0000 7f26752fd980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-03-04T17:33:12.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.177+0000 7f26752fd980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-03-04T17:33:12.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.177+0000 7f26752fd980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T17:33:12.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.177+0000 7f26752fd980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-03-04T17:33:12.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.177+0000 7f26752fd980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-03-04T17:33:12.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.177+0000 7f26752fd980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T17:33:12.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.177+0000 7f26752fd980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-03-04T17:33:12.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.177+0000 7f26752fd980 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1211 entering 2024-03-04T17:33:12.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.177+0000 7f26752fd980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T17:33:12.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.189+0000 7f2525fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T17:33:12.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.189+0000 7f26752fd980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T17:33:12.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:12.189+0000 7f26752fd980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T17:33:15.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.105+0000 7f26752fd980 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-03-04T17:33:15.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.105+0000 7f26752fd980 5 note: GC not initialized 2024-03-04T17:33:15.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.113+0000 7f2524fe1640 20 reqs_thread_entry: start 2024-03-04T17:33:15.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.113+0000 7f26752fd980 20 init_complete bucket index max shards: 11 2024-03-04T17:33:15.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.113+0000 7f26752fd980 20 Filter name: none 2024-03-04T17:33:15.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.113+0000 7f251dffb640 20 reqs_thread_entry: start 2024-03-04T17:33:15.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.137+0000 7f2525fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T17:33:15.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.137+0000 7f26752fd980 20 remove_watcher() i=0 2024-03-04T17:33:15.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.137+0000 7f26752fd980 2 removed watcher, disabling cache 2024-03-04T17:33:15.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.141+0000 7f26752fd980 20 remove_watcher() i=1 2024-03-04T17:33:15.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.141+0000 7f26752fd980 20 remove_watcher() i=2 2024-03-04T17:33:15.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.145+0000 7f26752fd980 20 remove_watcher() i=3 2024-03-04T17:33:15.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.149+0000 7f26752fd980 20 remove_watcher() i=4 2024-03-04T17:33:15.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.149+0000 7f26752fd980 20 remove_watcher() i=5 2024-03-04T17:33:15.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.149+0000 7f26752fd980 20 remove_watcher() i=6 2024-03-04T17:33:15.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.153+0000 7f26752fd980 20 remove_watcher() i=7 2024-03-04T17:33:15.169 INFO:teuthology.orchestra.run.smithi153.stdout:[] 2024-03-04T17:33:15.169 DEBUG:tasks.util.rgw: json result: [] 2024-03-04T17:33:15.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T17:33:15.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T17:33:15.170 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-03-04T17:33:15.209 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:15.209 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:15.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.233+0000 7f1f4c23b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T17:33:15.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.233+0000 7f1f4c23b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T17:33:15.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.237+0000 7f1d7ffff640 20 reqs_thread_entry: start 2024-03-04T17:33:15.258 INFO:teuthology.orchestra.run.smithi153.stdout:{"id":"d4796dd9-b51d-4fc8-b394-749e777f8124","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2024-03-04T17:33:15.258 DEBUG:tasks.util.rgw: json result: {'id': 'd4796dd9-b51d-4fc8-b394-749e777f8124', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-03-04T17:33:15.258 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2024-03-04T17:33:15.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2024-03-04T17:33:15.259 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2024-03-04T17:33:15.334 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:15.335 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:15.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.365+0000 7f59bc29b980 20 rados->read ofs=0 len=0 2024-03-04T17:33:15.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.365+0000 7f59bc29b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:15.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.365+0000 7f59bc29b980 20 realm 2024-03-04T17:33:15.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.365+0000 7f59bc29b980 20 rados->read ofs=0 len=0 2024-03-04T17:33:15.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.365+0000 7f59bc29b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:15.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.365+0000 7f59bc29b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-04T17:33:15.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.365+0000 7f59bc29b980 20 rados->read ofs=0 len=0 2024-03-04T17:33:15.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.365+0000 7f59bc29b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:15.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.365+0000 7f59bc29b980 20 rados->read ofs=0 len=0 2024-03-04T17:33:15.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.369+0000 7f59bc29b980 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T17:33:15.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.369+0000 7f59bc29b980 20 rados->read ofs=0 len=0 2024-03-04T17:33:15.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.369+0000 7f59bc29b980 20 rados_obj.operate() r=0 bl.length=891 2024-03-04T17:33:15.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.369+0000 7f59bc29b980 20 searching for the correct realm 2024-03-04T17:33:15.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 RGWRados::pool_iterate: got zonegroup_info.be061770-ad07-43f4-8436-fd9c6ba14763 2024-03-04T17:33:15.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-04T17:33:15.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 RGWRados::pool_iterate: got zone_info.d4796dd9-b51d-4fc8-b394-749e777f8124 2024-03-04T17:33:15.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 RGWRados::pool_iterate: got default.zone. 2024-03-04T17:33:15.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-04T17:33:15.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-04T17:33:15.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 rados->read ofs=0 len=0 2024-03-04T17:33:15.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:15.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 rados->read ofs=0 len=0 2024-03-04T17:33:15.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T17:33:15.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 rados->read ofs=0 len=0 2024-03-04T17:33:15.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 rados_obj.operate() r=0 bl.length=398 2024-03-04T17:33:15.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 zone default found 2024-03-04T17:33:15.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 4 Realm: () 2024-03-04T17:33:15.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 4 ZoneGroup: default (be061770-ad07-43f4-8436-fd9c6ba14763) 2024-03-04T17:33:15.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 4 Zone: default (d4796dd9-b51d-4fc8-b394-749e777f8124) 2024-03-04T17:33:15.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-04T17:33:15.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 zonegroup default 2024-03-04T17:33:15.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 rados->read ofs=0 len=0 2024-03-04T17:33:15.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:15.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 rados->read ofs=0 len=0 2024-03-04T17:33:15.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:15.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 rados->read ofs=0 len=0 2024-03-04T17:33:15.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T17:33:15.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 started sync module instance, tier type = 2024-03-04T17:33:15.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.377+0000 7f59bc29b980 20 started zone id=d4796dd9-b51d-4fc8-b394-749e777f8124 (name=default) with tier type = 2024-03-04T17:33:15.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.409+0000 7f59bc29b980 20 add_watcher() i=0 2024-03-04T17:33:15.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.409+0000 7f59bc29b980 20 add_watcher() i=1 2024-03-04T17:33:15.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.409+0000 7f59bc29b980 20 add_watcher() i=2 2024-03-04T17:33:15.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.409+0000 7f59bc29b980 20 add_watcher() i=3 2024-03-04T17:33:15.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.409+0000 7f59bc29b980 20 add_watcher() i=4 2024-03-04T17:33:15.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.409+0000 7f59bc29b980 20 add_watcher() i=5 2024-03-04T17:33:15.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.409+0000 7f59bc29b980 20 add_watcher() i=6 2024-03-04T17:33:15.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.413+0000 7f59bc29b980 20 add_watcher() i=7 2024-03-04T17:33:15.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.413+0000 7f59bc29b980 2 all 8 watchers are set, enabling cache 2024-03-04T17:33:15.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.417+0000 7f586b7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T17:33:15.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.417+0000 7f59bc29b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T17:33:15.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.417+0000 7f59bc29b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T17:33:15.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.417+0000 7f59bc29b980 5 note: GC not initialized 2024-03-04T17:33:15.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.417+0000 7f586a7fc640 20 reqs_thread_entry: start 2024-03-04T17:33:15.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.417+0000 7f59bc29b980 20 init_complete bucket index max shards: 11 2024-03-04T17:33:15.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.417+0000 7f59bc29b980 20 Filter name: none 2024-03-04T17:33:15.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.417+0000 7f57dbfff640 20 reqs_thread_entry: start 2024-03-04T17:33:15.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.425+0000 7f586b7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T17:33:15.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.433+0000 7f59bc29b980 20 remove_watcher() i=0 2024-03-04T17:33:15.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.433+0000 7f59bc29b980 2 removed watcher, disabling cache 2024-03-04T17:33:15.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.433+0000 7f59bc29b980 20 remove_watcher() i=1 2024-03-04T17:33:15.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.437+0000 7f59bc29b980 20 remove_watcher() i=2 2024-03-04T17:33:15.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.437+0000 7f59bc29b980 20 remove_watcher() i=3 2024-03-04T17:33:15.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.441+0000 7f59bc29b980 20 remove_watcher() i=4 2024-03-04T17:33:15.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.441+0000 7f59bc29b980 20 remove_watcher() i=5 2024-03-04T17:33:15.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.445+0000 7f59bc29b980 20 remove_watcher() i=6 2024-03-04T17:33:15.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.445+0000 7f59bc29b980 20 remove_watcher() i=7 2024-03-04T17:33:15.466 INFO:teuthology.orchestra.run.smithi153.stdout:[] 2024-03-04T17:33:15.466 DEBUG:tasks.util.rgw: json result: [] 2024-03-04T17:33:15.466 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T17:33:15.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T17:33:15.466 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-03-04T17:33:15.550 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:15.550 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:15.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.577+0000 7f79294e9980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T17:33:15.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.577+0000 7f79294e9980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T17:33:15.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:15.581+0000 7f77e4ff9640 20 reqs_thread_entry: start 2024-03-04T17:33:15.600 INFO:teuthology.orchestra.run.smithi153.stdout:{"id":"d4796dd9-b51d-4fc8-b394-749e777f8124","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2024-03-04T17:33:15.600 DEBUG:tasks.util.rgw: json result: {'id': 'd4796dd9-b51d-4fc8-b394-749e777f8124', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-03-04T17:33:15.600 INFO:tasks.rgw:Starting rgw... 2024-03-04T17:33:15.600 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2024-03-04T17:33:15.601 INFO:tasks.rgw:Using beast as radosgw frontend 2024-03-04T17:33:15.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo echo -n http://smithi136.front.sepia.ceph.com:8000 | sudo tee /home/ubuntu/cephtest/url_file 2024-03-04T17:33:15.617 INFO:teuthology.orchestra.run.smithi136.stdout:http://smithi136.front.sepia.ceph.com:8000 2024-03-04T17:33:15.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-03-04T17:33:15.675 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-03-04T17:33:15.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-03-04T17:33:15.721 INFO:tasks.rgw.c1.client.0:Started 2024-03-04T17:33:15.721 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2024-03-04T17:33:15.721 INFO:tasks.rgw:Using beast as radosgw frontend 2024-03-04T17:33:15.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo echo -n http://smithi136.front.sepia.ceph.com:8001 | sudo tee /home/ubuntu/cephtest/url_file 2024-03-04T17:33:15.736 INFO:teuthology.orchestra.run.smithi136.stdout:http://smithi136.front.sepia.ceph.com:8001 2024-03-04T17:33:15.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-03-04T17:33:15.790 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-03-04T17:33:15.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-03-04T17:33:15.841 INFO:tasks.rgw.c1.client.1:Started 2024-03-04T17:33:15.842 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2024-03-04T17:33:15.842 INFO:tasks.rgw:Using beast as radosgw frontend 2024-03-04T17:33:15.842 DEBUG:teuthology.orchestra.run.smithi153:> sudo echo -n http://smithi153.front.sepia.ceph.com:8000 | sudo tee /home/ubuntu/cephtest/url_file 2024-03-04T17:33:15.861 INFO:teuthology.orchestra.run.smithi153.stdout:http://smithi153.front.sepia.ceph.com:8000 2024-03-04T17:33:15.861 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-03-04T17:33:15.914 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-03-04T17:33:15.915 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-03-04T17:33:15.960 INFO:tasks.rgw.c2.client.0:Started 2024-03-04T17:33:15.960 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2024-03-04T17:33:15.960 INFO:tasks.rgw:Using beast as radosgw frontend 2024-03-04T17:33:15.960 DEBUG:teuthology.orchestra.run.smithi153:> sudo echo -n http://smithi153.front.sepia.ceph.com:8001 | sudo tee /home/ubuntu/cephtest/url_file 2024-03-04T17:33:15.975 INFO:teuthology.orchestra.run.smithi153.stdout:http://smithi153.front.sepia.ceph.com:8001 2024-03-04T17:33:15.975 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-03-04T17:33:16.029 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-03-04T17:33:16.029 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-03-04T17:33:16.071 INFO:tasks.rgw.c2.client.1:Started 2024-03-04T17:33:16.072 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi136.front.sepia.ceph.com:8000/ 2024-03-04T17:33:16.072 DEBUG:teuthology.orchestra.run.smithi136:> curl http://smithi136.front.sepia.ceph.com:8000/ 2024-03-04T17:33:16.093 INFO:teuthology.orchestra.run.smithi136.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T17:33:16.093 INFO:teuthology.orchestra.run.smithi136.stderr: Dload Upload Total Spent Left Speed 2024-03-04T17:33:16.095 INFO:teuthology.orchestra.run.smithi136.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 99k 0 --:--:-- --:--:-- --:--:-- 104k 2024-03-04T17:33:16.096 INFO:teuthology.orchestra.run.smithi136.stdout:anonymous 2024-03-04T17:33:16.096 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi136.front.sepia.ceph.com:8001/ 2024-03-04T17:33:16.096 DEBUG:teuthology.orchestra.run.smithi136:> curl http://smithi136.front.sepia.ceph.com:8001/ 2024-03-04T17:33:16.149 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-03-04T17:33:16.149 INFO:teuthology.orchestra.run.smithi136.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T17:33:16.149 INFO:teuthology.orchestra.run.smithi136.stderr: Dload Upload Total Spent Left Speed 2024-03-04T17:33:16.150 INFO:teuthology.orchestra.run.smithi136.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-03-04T17:33:16.150 INFO:teuthology.orchestra.run.smithi136.stderr:curl: (7) Failed to connect to smithi136.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2024-03-04T17:33:17.151 DEBUG:teuthology.orchestra.run.smithi136:> curl http://smithi136.front.sepia.ceph.com:8001/ 2024-03-04T17:33:17.162 INFO:teuthology.orchestra.run.smithi136.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T17:33:17.163 INFO:teuthology.orchestra.run.smithi136.stderr: Dload Upload Total Spent Left Speed 2024-03-04T17:33:17.166 INFO:teuthology.orchestra.run.smithi136.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 64535 0 --:--:-- --:--:-- --:--:-- 71333 2024-03-04T17:33:17.166 INFO:teuthology.orchestra.run.smithi136.stdout:anonymous 2024-03-04T17:33:17.167 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi153.front.sepia.ceph.com:8000/ 2024-03-04T17:33:17.167 DEBUG:teuthology.orchestra.run.smithi153:> curl http://smithi153.front.sepia.ceph.com:8000/ 2024-03-04T17:33:17.188 INFO:teuthology.orchestra.run.smithi153.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T17:33:17.188 INFO:teuthology.orchestra.run.smithi153.stderr: Dload Upload Total Spent Left Speed 2024-03-04T17:33:17.190 INFO:teuthology.orchestra.run.smithi153.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 85429 0 --:--:-- --:--:-- --:--:-- 104k 2024-03-04T17:33:17.191 INFO:teuthology.orchestra.run.smithi153.stdout:anonymous 2024-03-04T17:33:17.191 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi153.front.sepia.ceph.com:8001/ 2024-03-04T17:33:17.191 DEBUG:teuthology.orchestra.run.smithi153:> curl http://smithi153.front.sepia.ceph.com:8001/ 2024-03-04T17:33:17.244 INFO:teuthology.orchestra.run.smithi153.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T17:33:17.245 INFO:teuthology.orchestra.run.smithi153.stderr: Dload Upload Total Spent Left Speed 2024-03-04T17:33:17.246 INFO:teuthology.orchestra.run.smithi153.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 79142 0 --:--:-- --:--:-- --:--:-- 104k 2024-03-04T17:33:17.247 INFO:teuthology.orchestra.run.smithi153.stdout:anonymous 2024-03-04T17:33:17.248 INFO:teuthology.run_tasks:Running task rgw-multisite... 2024-03-04T17:33:17.475 INFO:tasks.rgw_multisite:creating realm.. 2024-03-04T17:33:17.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:17.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:17.476 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:33:17.514 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:17.514 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:17.571 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","name":"test-realm","current_period":"aa7942d8-051d-40ca-b86f-359c9c0301cc","epoch":1} 2024-03-04T17:33:17.571 INFO:tasks.rgw_multisite:creating master zonegroup.. 2024-03-04T17:33:17.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:17.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:17.572 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:33:17.648 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:17.648 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:17.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:17.680+0000 7f7df89f8980 -1 old format 2024-03-04T17:33:17.699 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":[]}],"default_placement":"default-placement","realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","sync_policy":{"groups":[]},"enabled_features":["resharding"]} 2024-03-04T17:33:17.699 INFO:tasks.rgw_multisite:creating master zone.. 2024-03-04T17:33:17.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '1THAB257QDW2GNAH', '--secret', 'QoUA0kEYOw3e9ZPsidmlofuWIy510GDX', '--endpoints', 'http://smithi136.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:17.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '1THAB257QDW2GNAH', '--secret', 'QoUA0kEYOw3e9ZPsidmlofuWIy510GDX', '--endpoints', 'http://smithi136.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:17.700 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key 1THAB257QDW2GNAH --secret QoUA0kEYOw3e9ZPsidmlofuWIy510GDX --endpoints http://smithi136.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:33:17.785 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:17.785 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:17.818 INFO:teuthology.orchestra.run.smithi136.stderr:NOTICE: set zone's realm_id=fcff5afe-947d-4c2e-8449-fe8f8ae33dc5 2024-03-04T17:33:17.844 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"d96636ac-054d-40e3-90f3-e3912da784b2","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"1THAB257QDW2GNAH","secret_key":"QoUA0kEYOw3e9ZPsidmlofuWIy510GDX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","notif_pool":"test-zone1.rgw.log:notif"} 2024-03-04T17:33:17.844 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2024-03-04T17:33:18.303 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'test-zone1.rgw.buckets.data' created 2024-03-04T17:33:18.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2024-03-04T17:33:19.305 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2024-03-04T17:33:19.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:19.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:19.339 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:33:19.388 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:19.388 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:19.433 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"d96636ac-054d-40e3-90f3-e3912da784b2","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"1THAB257QDW2GNAH","secret_key":"QoUA0kEYOw3e9ZPsidmlofuWIy510GDX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","notif_pool":"test-zone1.rgw.log:notif"} 2024-03-04T17:33:19.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:19.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:19.434 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:33:19.466 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:19.466 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:19.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:19.500+0000 7f64a58f3980 0 period (aa7942d8-051d-40ca-b86f-359c9c0301cc does not have zone d96636ac-054d-40e3-90f3-e3912da784b2 configured 2024-03-04T17:33:26.494 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"f6a09d88-afdd-40f4-a159-dda085c373df","epoch":1,"predecessor_uuid":"aa7942d8-051d-40ca-b86f-359c9c0301cc","sync_status":[],"period_map":{"id":"f6a09d88-afdd-40f4-a159-dda085c373df","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","zones":[{"id":"d96636ac-054d-40e3-90f3-e3912da784b2","name":"test-zone1","endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"d96636ac-054d-40e3-90f3-e3912da784b2","val":2093114680}]},"master_zonegroup":"test-zonegroup","master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","realm_epoch":2} 2024-03-04T17:33:26.494 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-03-04T17:33:26.495 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T17:33:26.536 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T17:33:26.536+0000 7ff896cad640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 29314) UID: 0 2024-03-04T17:33:26.536 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T17:33:26.536+0000 7ff89a4f5a40 -1 shutting down 2024-03-04T17:33:31.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-03-04T17:33:32.596 INFO:tasks.rgw.c1.client.0:Stopped 2024-03-04T17:33:32.597 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-03-04T17:33:32.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-03-04T17:33:32.600 INFO:tasks.rgw.c1.client.0:Started 2024-03-04T17:33:32.600 DEBUG:teuthology.orchestra.run.smithi136:> curl http://smithi136.front.sepia.ceph.com:8000 2024-03-04T17:33:32.612 INFO:teuthology.orchestra.run.smithi136.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T17:33:32.612 INFO:teuthology.orchestra.run.smithi136.stderr: Dload Upload Total Spent Left Speed 2024-03-04T17:33:32.612 INFO:teuthology.orchestra.run.smithi136.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-03-04T17:33:32.612 INFO:teuthology.orchestra.run.smithi136.stderr:curl: (7) Failed to connect to smithi136.front.sepia.ceph.com port 8000 after 0 ms: Connection refused 2024-03-04T17:33:32.613 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-03-04T17:33:33.614 DEBUG:teuthology.orchestra.run.smithi136:> curl http://smithi136.front.sepia.ceph.com:8000 2024-03-04T17:33:33.626 INFO:teuthology.orchestra.run.smithi136.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T17:33:33.626 INFO:teuthology.orchestra.run.smithi136.stderr: Dload Upload Total Spent Left Speed 2024-03-04T17:33:33.629 INFO:teuthology.orchestra.run.smithi136.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 64264 0 --:--:-- --:--:-- --:--:-- 71333 2024-03-04T17:33:33.630 INFO:teuthology.orchestra.run.smithi136.stdout:anonymous 2024-03-04T17:33:33.630 INFO:tasks.rgw_multisite:creating admin user.. 2024-03-04T17:33:33.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '1THAB257QDW2GNAH', '--secret', 'QoUA0kEYOw3e9ZPsidmlofuWIy510GDX', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:33.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '1THAB257QDW2GNAH', '--secret', 'QoUA0kEYOw3e9ZPsidmlofuWIy510GDX', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:33.630 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 1THAB257QDW2GNAH --secret QoUA0kEYOw3e9ZPsidmlofuWIy510GDX --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:33:33.721 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:33.721 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:33.776 INFO:teuthology.orchestra.run.smithi136.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"1THAB257QDW2GNAH","secret_key":"QoUA0kEYOw3e9ZPsidmlofuWIy510GDX","active":true}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":true,"default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2024-03-04T17:33:33.796 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-03-04T17:33:33.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi136.front.sepia.ceph.com:8000', '--access-key', '1THAB257QDW2GNAH', '--secret', 'QoUA0kEYOw3e9ZPsidmlofuWIy510GDX', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:33.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi136.front.sepia.ceph.com:8000', '--access-key', '1THAB257QDW2GNAH', '--secret', 'QoUA0kEYOw3e9ZPsidmlofuWIy510GDX', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:33.797 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi136.front.sepia.ceph.com:8000 --access-key 1THAB257QDW2GNAH --secret QoUA0kEYOw3e9ZPsidmlofuWIy510GDX --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T17:33:33.843 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:33.843 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:33.916 INFO:teuthology.orchestra.run.smithi153.stdout:{"id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","name":"test-realm","current_period":"f6a09d88-afdd-40f4-a159-dda085c373df","epoch":2} 2024-03-04T17:33:33.916 INFO:tasks.rgw_multisite:creating zone.. 2024-03-04T17:33:33.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '1THAB257QDW2GNAH', '--secret', 'QoUA0kEYOw3e9ZPsidmlofuWIy510GDX', '--endpoints', 'http://smithi153.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:33.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '1THAB257QDW2GNAH', '--secret', 'QoUA0kEYOw3e9ZPsidmlofuWIy510GDX', '--endpoints', 'http://smithi153.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:33.917 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key 1THAB257QDW2GNAH --secret QoUA0kEYOw3e9ZPsidmlofuWIy510GDX --endpoints http://smithi153.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T17:33:33.956 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:33.956 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:33.991 INFO:teuthology.orchestra.run.smithi153.stderr:NOTICE: set zone's realm_id=fcff5afe-947d-4c2e-8449-fe8f8ae33dc5 2024-03-04T17:33:34.024 INFO:teuthology.orchestra.run.smithi153.stdout:{"id":"185c722f-6c40-48e4-8855-bf605b5498c7","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"1THAB257QDW2GNAH","secret_key":"QoUA0kEYOw3e9ZPsidmlofuWIy510GDX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","notif_pool":"test-zone2.rgw.log:notif"} 2024-03-04T17:33:34.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2024-03-04T17:33:35.138 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'test-zone2.rgw.buckets.data' created 2024-03-04T17:33:35.173 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2024-03-04T17:33:36.146 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2024-03-04T17:33:36.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:36.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:36.177 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T17:33:36.229 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:36.230 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:36.273 INFO:teuthology.orchestra.run.smithi153.stdout:{"id":"185c722f-6c40-48e4-8855-bf605b5498c7","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"1THAB257QDW2GNAH","secret_key":"QoUA0kEYOw3e9ZPsidmlofuWIy510GDX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","notif_pool":"test-zone2.rgw.log:notif"} 2024-03-04T17:33:36.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:36.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:36.274 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T17:33:36.304 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:36.305 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:36.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:33:36.332+0000 7fca9b946980 0 period (f6a09d88-afdd-40f4-a159-dda085c373df does not have zone 185c722f-6c40-48e4-8855-bf605b5498c7 configured 2024-03-04T17:33:37.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-03-04T17:33:43.250 INFO:teuthology.orchestra.run.smithi153.stderr:Sending period to new master zone d96636ac-054d-40e3-90f3-e3912da784b2 2024-03-04T17:33:43.336 INFO:teuthology.orchestra.run.smithi153.stdout:{"id":"f6a09d88-afdd-40f4-a159-dda085c373df","epoch":2,"predecessor_uuid":"aa7942d8-051d-40ca-b86f-359c9c0301cc","sync_status":[],"period_map":{"id":"f6a09d88-afdd-40f4-a159-dda085c373df","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","zones":[{"id":"185c722f-6c40-48e4-8855-bf605b5498c7","name":"test-zone2","endpoints":["http://smithi153.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"d96636ac-054d-40e3-90f3-e3912da784b2","name":"test-zone1","endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"185c722f-6c40-48e4-8855-bf605b5498c7","val":1962731440},{"key":"d96636ac-054d-40e3-90f3-e3912da784b2","val":2093114680}]},"master_zonegroup":"test-zonegroup","master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","realm_epoch":2} 2024-03-04T17:33:43.337 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-03-04T17:33:43.337 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T17:33:43.379 INFO:tasks.rgw.c2.client.0.smithi153.stdout:2024-03-04T17:33:43.376+0000 7fd0e4a9c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 29288) UID: 0 2024-03-04T17:33:43.380 INFO:tasks.rgw.c2.client.0.smithi153.stdout:2024-03-04T17:33:43.376+0000 7fd0e82e4a40 -1 shutting down 2024-03-04T17:33:48.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-03-04T17:33:49.439 INFO:tasks.rgw.c2.client.0:Stopped 2024-03-04T17:33:49.439 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-03-04T17:33:49.439 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-03-04T17:33:49.443 INFO:tasks.rgw.c2.client.0:Started 2024-03-04T17:33:49.443 DEBUG:teuthology.orchestra.run.smithi153:> curl http://smithi153.front.sepia.ceph.com:8000 2024-03-04T17:33:49.455 INFO:teuthology.orchestra.run.smithi153.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T17:33:49.455 INFO:teuthology.orchestra.run.smithi153.stderr: Dload Upload Total Spent Left Speed 2024-03-04T17:33:49.455 INFO:teuthology.orchestra.run.smithi153.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-03-04T17:33:49.455 INFO:teuthology.orchestra.run.smithi153.stderr:curl: (7) Failed to connect to smithi153.front.sepia.ceph.com port 8000 after 0 ms: Connection refused 2024-03-04T17:33:49.456 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-03-04T17:33:50.457 DEBUG:teuthology.orchestra.run.smithi153:> curl http://smithi153.front.sepia.ceph.com:8000 2024-03-04T17:33:50.469 INFO:teuthology.orchestra.run.smithi153.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T17:33:50.470 INFO:teuthology.orchestra.run.smithi153.stderr: Dload Upload Total Spent Left Speed 2024-03-04T17:33:50.472 INFO:teuthology.orchestra.run.smithi153.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 66542 0 --:--:-- --:--:-- --:--:-- 71333 2024-03-04T17:33:50.473 INFO:teuthology.orchestra.run.smithi153.stdout:anonymous 2024-03-04T17:33:50.473 INFO:tasks.rgw_multisite:creating zone.. 2024-03-04T17:33:50.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1THAB257QDW2GNAH', '--secret', 'QoUA0kEYOw3e9ZPsidmlofuWIy510GDX', '--endpoints', 'http://smithi136.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:50.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1THAB257QDW2GNAH', '--secret', 'QoUA0kEYOw3e9ZPsidmlofuWIy510GDX', '--endpoints', 'http://smithi136.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:50.474 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 1THAB257QDW2GNAH --secret QoUA0kEYOw3e9ZPsidmlofuWIy510GDX --endpoints http://smithi136.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:33:50.518 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:50.518 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:50.550 INFO:teuthology.orchestra.run.smithi136.stderr:NOTICE: set zone's realm_id=fcff5afe-947d-4c2e-8449-fe8f8ae33dc5 2024-03-04T17:33:50.577 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"c146b41a-1ed1-469f-afe6-2dddc3577925","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"1THAB257QDW2GNAH","secret_key":"QoUA0kEYOw3e9ZPsidmlofuWIy510GDX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","notif_pool":"test-zone3.rgw.log:notif"} 2024-03-04T17:33:50.578 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2024-03-04T17:33:51.206 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'test-zone3.rgw.buckets.data' created 2024-03-04T17:33:51.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2024-03-04T17:33:52.230 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2024-03-04T17:33:52.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:52.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:52.272 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:33:52.321 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:52.322 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:52.369 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"c146b41a-1ed1-469f-afe6-2dddc3577925","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"1THAB257QDW2GNAH","secret_key":"QoUA0kEYOw3e9ZPsidmlofuWIy510GDX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","notif_pool":"test-zone3.rgw.log:notif"} 2024-03-04T17:33:52.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:52.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:33:52.370 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:33:52.401 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:52.401 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:52.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:33:52.435+0000 7f6e86387980 0 period (f6a09d88-afdd-40f4-a159-dda085c373df does not have zone c146b41a-1ed1-469f-afe6-2dddc3577925 configured 2024-03-04T17:33:54.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-03-04T17:33:59.348 INFO:teuthology.orchestra.run.smithi136.stderr:Sending period to new master zone d96636ac-054d-40e3-90f3-e3912da784b2 2024-03-04T17:33:59.400 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"f6a09d88-afdd-40f4-a159-dda085c373df","epoch":3,"predecessor_uuid":"aa7942d8-051d-40ca-b86f-359c9c0301cc","sync_status":[],"period_map":{"id":"f6a09d88-afdd-40f4-a159-dda085c373df","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","zones":[{"id":"185c722f-6c40-48e4-8855-bf605b5498c7","name":"test-zone2","endpoints":["http://smithi153.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"c146b41a-1ed1-469f-afe6-2dddc3577925","name":"test-zone3","endpoints":["http://smithi136.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"d96636ac-054d-40e3-90f3-e3912da784b2","name":"test-zone1","endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"185c722f-6c40-48e4-8855-bf605b5498c7","val":1962731440},{"key":"c146b41a-1ed1-469f-afe6-2dddc3577925","val":3710957901},{"key":"d96636ac-054d-40e3-90f3-e3912da784b2","val":2093114680}]},"master_zonegroup":"test-zonegroup","master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","realm_epoch":2} 2024-03-04T17:33:59.400 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-03-04T17:33:59.400 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T17:33:59.444 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T17:33:59.443+0000 7f5df3641640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 29911) UID: 0 2024-03-04T17:33:59.444 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T17:33:59.443+0000 7f5df6e89a40 -1 shutting down 2024-03-04T17:34:00.296 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-03-04T17:34:05.502 INFO:tasks.rgw.c1.client.1:Stopped 2024-03-04T17:34:05.502 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-03-04T17:34:05.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-03-04T17:34:05.506 INFO:tasks.rgw.c1.client.1:Started 2024-03-04T17:34:05.506 DEBUG:teuthology.orchestra.run.smithi136:> curl http://smithi136.front.sepia.ceph.com:8001 2024-03-04T17:34:05.517 INFO:teuthology.orchestra.run.smithi136.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T17:34:05.518 INFO:teuthology.orchestra.run.smithi136.stderr: Dload Upload Total Spent Left Speed 2024-03-04T17:34:05.518 INFO:teuthology.orchestra.run.smithi136.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-03-04T17:34:05.519 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-03-04T17:34:05.519 INFO:teuthology.orchestra.run.smithi136.stderr:curl: (7) Failed to connect to smithi136.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2024-03-04T17:34:06.002 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-03-04T17:34:06.520 DEBUG:teuthology.orchestra.run.smithi136:> curl http://smithi136.front.sepia.ceph.com:8001 2024-03-04T17:34:06.533 INFO:teuthology.orchestra.run.smithi136.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T17:34:06.533 INFO:teuthology.orchestra.run.smithi136.stderr: Dload Upload Total Spent Left Speed 2024-03-04T17:34:06.536 INFO:teuthology.orchestra.run.smithi136.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 67253 0 --:--:-- --:--:-- --:--:-- 71333 2024-03-04T17:34:06.536 INFO:teuthology.orchestra.run.smithi136.stdout:anonymous 2024-03-04T17:34:06.537 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2024-03-04T17:34:06.537 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2024-03-04T17:34:06.552 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi 2024-03-04T17:34:06.615 INFO:tasks.rgw_multisite_tests:creating test user.. 2024-03-04T17:34:06.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:34:06.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:34:06.616 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:34:06.665 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:06.666 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:06.725 INFO:teuthology.orchestra.run.smithi136.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"HBI3JS1W1675M40GMG1A","secret_key":"3v8xhnZmrxveMksKKC7WC3ep597MnFkLFNPdhitl","active":true}],"swift_keys":[],"caps":[{"type":"roles","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2024-03-04T17:34:06.746 INFO:rgw_multi.tests:meta checkpoint 2024-03-04T17:34:06.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:06.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:06.747 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:06.779 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:06.779 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:06.831 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:34:06.852 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:34:06.853 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:34:06.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:06.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:06.853 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:06.896 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:06.896 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:06.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:06.944+0000 7fa94ed2e980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:06.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:06.948+0000 7fa94ed2e980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:06.975 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:06Z"} 2024-03-04T17:34:06.975 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:34:11.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:11.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:11.976 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:12.019 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:12.019 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:12.072+0000 7f375045e980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:12.072+0000 7f375045e980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:12.100 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:12Z"} 2024-03-04T17:34:12.101 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:34:17.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:17.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:17.103 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:17.147 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:17.147 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:17.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:17.196+0000 7f656de6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:17.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:17.200+0000 7f656de6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:17.225 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:17Z"} 2024-03-04T17:34:17.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:34:22.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:22.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:22.228 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:22.272 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:22.272 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:22.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:22.320+0000 7ff3d17d0980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:22.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:22.320+0000 7ff3d17d0980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:22.349 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:22Z"} 2024-03-04T17:34:22.349 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:34:27.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:27.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:27.351 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:27.443+0000 7ff93c8a4980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:27.443+0000 7ff93c8a4980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:27.471 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:27Z"} 2024-03-04T17:34:27.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:34:32.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:32.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:32.473 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:32.517 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:32.517 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:32.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:32.563+0000 7fcb6b327980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:32.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:32.563+0000 7fcb6b327980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:32.591 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:32Z"} 2024-03-04T17:34:32.592 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:34:37.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:37.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:37.593 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:37.635 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:37.636 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:37.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:37.687+0000 7fc7e213c980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:37.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:37.687+0000 7fc7e213c980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:37.717 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:37Z"} 2024-03-04T17:34:37.718 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:34:42.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:42.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:42.719 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:42.762 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:42.763 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:42.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:42.811+0000 7f789bfb6980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:42.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:42.815+0000 7f789bfb6980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:42.842 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:42Z"} 2024-03-04T17:34:42.842 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:34:47.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:47.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:47.843 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:47.886 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:47.886 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:47.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:47.935+0000 7f6048c78980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:47.935+0000 7f6048c78980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:47.965 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:47Z"} 2024-03-04T17:34:47.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:34:52.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:52.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:52.967 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:53.010 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:53.010 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:53.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:53.059+0000 7fd29008c980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:53.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:53.059+0000 7fd29008c980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:53.086 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:53Z"} 2024-03-04T17:34:53.086 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:34:58.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:58.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:58.088 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:58.132 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:58.132 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:58.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:58.179+0000 7faaa02cb980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:58.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:34:58.183+0000 7faaa02cb980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:58.211 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:58Z"} 2024-03-04T17:34:58.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:35:03.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:03.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:03.213 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:03.256 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:03.256 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:03.307+0000 7f880bcd2980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:03.307+0000 7f880bcd2980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:03.335 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:03Z"} 2024-03-04T17:35:03.335 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:35:08.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:08.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:08.337 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:08.379 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:08.379 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:08.427+0000 7fb3f511a980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:08.427+0000 7fb3f511a980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:08.457 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:08Z"} 2024-03-04T17:35:08.457 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:35:13.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:13.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:13.458 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:13.501 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:13.501 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:13.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:13.547+0000 7f150cc5b980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:13.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:13.547+0000 7f150cc5b980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:13.576 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:13Z"} 2024-03-04T17:35:13.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:35:18.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:18.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:18.578 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:18.621 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:18.621 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:18.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:18.675+0000 7fe5a9760980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:18.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:18.675+0000 7fe5a9760980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:18.699 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:18Z"} 2024-03-04T17:35:18.700 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:35:23.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:23.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:23.701 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:23.743 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:23.744 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:23.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:23.795+0000 7ff0a96f4980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:23.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:23.795+0000 7ff0a96f4980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:23.821 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:23Z"} 2024-03-04T17:35:23.821 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:35:28.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:28.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:28.822 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:28.865 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:28.865 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:28.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:28.918+0000 7f04be1b0980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:28.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:28.918+0000 7f04be1b0980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:28.945 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:28Z"} 2024-03-04T17:35:28.946 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:35:33.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:33.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:33.947 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:33.990 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:33.990 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:34.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:34.046+0000 7fa8eb6a2980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:34.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:34.046+0000 7fa8eb6a2980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:34.074 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:34Z"} 2024-03-04T17:35:34.074 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:35:39.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:39.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:39.076 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:39.119 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:39.119 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:39.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:39.170+0000 7fb39f72f980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:39.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:39.170+0000 7fb39f72f980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:39.195 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:39Z"} 2024-03-04T17:35:39.196 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:35:44.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:44.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:44.197 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:44.243 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:44.243 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:44.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:44.298+0000 7f957ec4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:44.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:44.298+0000 7f957ec4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:44.325 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:44Z"} 2024-03-04T17:35:44.326 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:35:49.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:49.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:49.327 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:49.370 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:49.370 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:49.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:49.422+0000 7f88dbfb0980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:49.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:49.422+0000 7f88dbfb0980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:49.449 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:49Z"} 2024-03-04T17:35:49.449 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:35:54.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:54.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:54.450 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:54.492 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:54.492 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:54.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:54.542+0000 7f7c8a670980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:54.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:54.546+0000 7f7c8a670980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:54.570 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:54Z"} 2024-03-04T17:35:54.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:35:59.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:59.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:59.572 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:59.614 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:59.614 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:59.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:59.666+0000 7efe9a623980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:59.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:35:59.666+0000 7efe9a623980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:59.692 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:59Z"} 2024-03-04T17:35:59.693 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573646.723863_31.1 target= 2024-03-04T17:36:04.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:04.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:04.694 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:04.736 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:04.736 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:04.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:04.786+0000 7fc4ff6ee980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:04.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:04.786+0000 7fc4ff6ee980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:04.813 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:04Z"} 2024-03-04T17:36:04.813 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:36:04.813 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:36:04.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:04.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:04.814 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:04.857 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:04.857 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:04.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:04.915+0000 7fbe23ec9980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:04.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:04.915+0000 7fbe23ec9980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:04.940 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:04Z"} 2024-03-04T17:36:04.940 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:36:04.940 INFO:rgw_multi.tests:meta checkpoint 2024-03-04T17:36:04.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:04.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:04.941 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:05.018 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:05.018 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:05.073 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:36:05.095 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:36:05.095 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:36:05.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:05.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:05.096 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:05.139 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:05.139 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:05.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:05.190+0000 7fb7f9e66980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:05.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:05.190+0000 7fb7f9e66980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:05.217 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:05Z"} 2024-03-04T17:36:05.217 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:36:05.217 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:36:05.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:05.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:05.218 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:05.261 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:05.261 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:05.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:05.315+0000 7fe0a9200980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:05.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:05.315+0000 7fe0a9200980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:05.341 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:05Z"} 2024-03-04T17:36:05.341 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:36:05.341 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2024-03-04T17:36:05.405 DEBUG:boto:Using access key provided by client. 2024-03-04T17:36:05.405 DEBUG:boto:Using secret key provided by client. 2024-03-04T17:36:05.406 DEBUG:boto:Using access key provided by client. 2024-03-04T17:36:05.406 DEBUG:boto:Using secret key provided by client. 2024-03-04T17:36:05.406 DEBUG:boto:Using access key provided by client. 2024-03-04T17:36:05.406 DEBUG:boto:Using secret key provided by client. 2024-03-04T17:36:05.407 DEBUG:boto:Using access key provided by client. 2024-03-04T17:36:05.407 DEBUG:boto:Using secret key provided by client. 2024-03-04T17:36:05.407 DEBUG:boto:Using access key provided by client. 2024-03-04T17:36:05.407 DEBUG:boto:Using secret key provided by client. 2024-03-04T17:36:05.408 DEBUG:boto:Using access key provided by client. 2024-03-04T17:36:05.408 DEBUG:boto:Using secret key provided by client. 2024-03-04T17:36:05.408 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-1 2024-03-04T17:36:05.408 DEBUG:boto:path=/ppfahn-1/ 2024-03-04T17:36:05.408 DEBUG:boto:auth_path=/ppfahn-1/ 2024-03-04T17:36:05.408 DEBUG:boto:Method: PUT 2024-03-04T17:36:05.409 DEBUG:boto:Path: /ppfahn-1/ 2024-03-04T17:36:05.409 DEBUG:boto:Data: 2024-03-04T17:36:05.409 DEBUG:boto:Headers: {} 2024-03-04T17:36:05.409 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:05.409 DEBUG:boto:Port: 8000 2024-03-04T17:36:05.409 DEBUG:boto:Params: {} 2024-03-04T17:36:05.409 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:36:05.409 DEBUG:boto:Token: None 2024-03-04T17:36:05.409 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:05 GMT /ppfahn-1/ 2024-03-04T17:36:05.410 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U22meN+RzDNPfImf0ulZt1JNcb4= 2024-03-04T17:36:05.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U22meN+RzDNPfImf0ulZt1JNcb4=', 'Content-Length': '0'} 2024-03-04T17:36:08.449 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002c0a53230387a196-0065e60685-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:08.449 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-2 2024-03-04T17:36:08.450 DEBUG:boto:path=/ppfahn-2/ 2024-03-04T17:36:08.450 DEBUG:boto:auth_path=/ppfahn-2/ 2024-03-04T17:36:08.450 DEBUG:boto:Method: PUT 2024-03-04T17:36:08.450 DEBUG:boto:Path: /ppfahn-2/ 2024-03-04T17:36:08.450 DEBUG:boto:Data: 2024-03-04T17:36:08.450 DEBUG:boto:Headers: {} 2024-03-04T17:36:08.450 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:08.450 DEBUG:boto:Port: 8000 2024-03-04T17:36:08.450 DEBUG:boto:Params: {} 2024-03-04T17:36:08.450 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:36:08.451 DEBUG:boto:Token: None 2024-03-04T17:36:08.451 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:08 GMT /ppfahn-2/ 2024-03-04T17:36:08.451 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uvG4oK8fz6u5jHvljghX044Vet8= 2024-03-04T17:36:08.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uvG4oK8fz6u5jHvljghX044Vet8=', 'Content-Length': '0'} 2024-03-04T17:36:11.217 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000daa8bdbc8de1bec3-0065e60688-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:11.217 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-3 2024-03-04T17:36:11.217 DEBUG:boto:path=/ppfahn-3/ 2024-03-04T17:36:11.217 DEBUG:boto:auth_path=/ppfahn-3/ 2024-03-04T17:36:11.217 DEBUG:boto:Method: PUT 2024-03-04T17:36:11.217 DEBUG:boto:Path: /ppfahn-3/ 2024-03-04T17:36:11.218 DEBUG:boto:Data: 2024-03-04T17:36:11.218 DEBUG:boto:Headers: {} 2024-03-04T17:36:11.218 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:11.218 DEBUG:boto:Port: 8001 2024-03-04T17:36:11.218 DEBUG:boto:Params: {} 2024-03-04T17:36:11.218 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T17:36:11.218 DEBUG:boto:Token: None 2024-03-04T17:36:11.218 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:11 GMT /ppfahn-3/ 2024-03-04T17:36:11.218 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QeKALNqXTWQB2gY1io83U8DgUjQ= 2024-03-04T17:36:11.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QeKALNqXTWQB2gY1io83U8DgUjQ=', 'Content-Length': '0'} 2024-03-04T17:36:11.278 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000265d0bb8b3e5777a-0065e6068b-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:11.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:11.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:11.279 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:11.319 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:11.319 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:11.373 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:36:11.401 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573771.249847_47.1","last_update":"2024-03-04T17:36:11.249847Z"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:36:11.402 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573771.249847_47.1', 3: ''} 2024-03-04T17:36:11.402 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:36:11.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:11.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:11.402 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:11.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:11.514+0000 7f91f8432980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:11.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:11.518+0000 7f91f8432980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:11.542 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573771.249847_47.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:11.249847Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:11Z"} 2024-03-04T17:36:11.543 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573771.249847_47.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:11.249847Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:11Z"} 2024-03-04T17:36:11.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573771.249847_47.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:11.249847Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:11.543 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573771.249847_47.1', 3: ''} 2024-03-04T17:36:11.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573771.249847_47.1', 3: ''} 2024-03-04T17:36:11.543 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:36:11.544 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:36:11.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:11.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:11.544 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:11.585 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:11.585 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:11.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:11.639+0000 7fddb1fb7980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:11.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:11.639+0000 7fddb1fb7980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:11.665 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573768.433494_36.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:08.433494Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:11Z"} 2024-03-04T17:36:11.666 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573768.433494_36.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:08.433494Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:11Z"} 2024-03-04T17:36:11.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573768.433494_36.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:08.433494Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:11.666 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573771.249847_47.1', 3: ''} 2024-03-04T17:36:11.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573768.433494_36.1', 3: ''} 2024-03-04T17:36:11.666 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1709573771.249847_47.1 target=1_1709573768.433494_36.1 2024-03-04T17:36:16.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:16.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:16.668 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:16.711 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:16.712 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:16.763+0000 7f302a2f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:16.763+0000 7f302a2f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:16.788 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573771.249847_47.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:11.249847Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:16Z"} 2024-03-04T17:36:16.788 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573771.249847_47.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:11.249847Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:16Z"} 2024-03-04T17:36:16.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573771.249847_47.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:11.249847Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:16.789 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573771.249847_47.1', 3: ''} 2024-03-04T17:36:16.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573771.249847_47.1', 3: ''} 2024-03-04T17:36:16.789 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:36:16.789 DEBUG:boto:path=/ppfahn-1/ 2024-03-04T17:36:16.789 DEBUG:boto:auth_path=/ppfahn-1/ 2024-03-04T17:36:16.790 DEBUG:boto:Method: HEAD 2024-03-04T17:36:16.790 DEBUG:boto:Path: /ppfahn-1/ 2024-03-04T17:36:16.790 DEBUG:boto:Data: 2024-03-04T17:36:16.790 DEBUG:boto:Headers: {} 2024-03-04T17:36:16.790 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:16.790 DEBUG:boto:Port: 8000 2024-03-04T17:36:16.790 DEBUG:boto:Params: {} 2024-03-04T17:36:16.790 DEBUG:boto:Token: None 2024-03-04T17:36:16.790 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:16 GMT /ppfahn-1/ 2024-03-04T17:36:16.791 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M5Js8kEHduxtxtQLVsWDkh2ux7E= 2024-03-04T17:36:16.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M5Js8kEHduxtxtQLVsWDkh2ux7E=', 'Content-Length': '0'} 2024-03-04T17:36:16.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b67f907007de4f7-0065e60690-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:16.793 DEBUG:boto:path=/ppfahn-2/ 2024-03-04T17:36:16.793 DEBUG:boto:auth_path=/ppfahn-2/ 2024-03-04T17:36:16.793 DEBUG:boto:Method: HEAD 2024-03-04T17:36:16.793 DEBUG:boto:Path: /ppfahn-2/ 2024-03-04T17:36:16.794 DEBUG:boto:Data: 2024-03-04T17:36:16.794 DEBUG:boto:Headers: {} 2024-03-04T17:36:16.794 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:16.794 DEBUG:boto:Port: 8000 2024-03-04T17:36:16.794 DEBUG:boto:Params: {} 2024-03-04T17:36:16.794 DEBUG:boto:Token: None 2024-03-04T17:36:16.794 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:16 GMT /ppfahn-2/ 2024-03-04T17:36:16.794 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NRhhQS1Jca1+pUiiQ5AWdpg3Cxc= 2024-03-04T17:36:16.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NRhhQS1Jca1+pUiiQ5AWdpg3Cxc=', 'Content-Length': '0'} 2024-03-04T17:36:16.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfa84cc525f2656f-0065e60690-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:16.797 DEBUG:boto:path=/ppfahn-3/ 2024-03-04T17:36:16.797 DEBUG:boto:auth_path=/ppfahn-3/ 2024-03-04T17:36:16.797 DEBUG:boto:Method: HEAD 2024-03-04T17:36:16.797 DEBUG:boto:Path: /ppfahn-3/ 2024-03-04T17:36:16.797 DEBUG:boto:Data: 2024-03-04T17:36:16.797 DEBUG:boto:Headers: {} 2024-03-04T17:36:16.797 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:16.797 DEBUG:boto:Port: 8000 2024-03-04T17:36:16.797 DEBUG:boto:Params: {} 2024-03-04T17:36:16.798 DEBUG:boto:Token: None 2024-03-04T17:36:16.798 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:16 GMT /ppfahn-3/ 2024-03-04T17:36:16.798 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6Pi8PlwaO4pOyr4zILiTkM/q4jE= 2024-03-04T17:36:16.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6Pi8PlwaO4pOyr4zILiTkM/q4jE=', 'Content-Length': '0'} 2024-03-04T17:36:16.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031d4fd7be5487041-0065e60690-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:16.800 DEBUG:boto:path=/ppfahn-1/ 2024-03-04T17:36:16.800 DEBUG:boto:auth_path=/ppfahn-1/ 2024-03-04T17:36:16.800 DEBUG:boto:Method: HEAD 2024-03-04T17:36:16.801 DEBUG:boto:Path: /ppfahn-1/ 2024-03-04T17:36:16.801 DEBUG:boto:Data: 2024-03-04T17:36:16.801 DEBUG:boto:Headers: {} 2024-03-04T17:36:16.801 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:16.801 DEBUG:boto:Port: 8000 2024-03-04T17:36:16.801 DEBUG:boto:Params: {} 2024-03-04T17:36:16.801 DEBUG:boto:Token: None 2024-03-04T17:36:16.801 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:16 GMT /ppfahn-1/ 2024-03-04T17:36:16.802 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M5Js8kEHduxtxtQLVsWDkh2ux7E= 2024-03-04T17:36:16.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M5Js8kEHduxtxtQLVsWDkh2ux7E=', 'Content-Length': '0'} 2024-03-04T17:36:16.806 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae8bdf4a07b4299e-0065e60690-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:16.806 DEBUG:boto:path=/ppfahn-2/ 2024-03-04T17:36:16.806 DEBUG:boto:auth_path=/ppfahn-2/ 2024-03-04T17:36:16.807 DEBUG:boto:Method: HEAD 2024-03-04T17:36:16.807 DEBUG:boto:Path: /ppfahn-2/ 2024-03-04T17:36:16.807 DEBUG:boto:Data: 2024-03-04T17:36:16.807 DEBUG:boto:Headers: {} 2024-03-04T17:36:16.807 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:16.807 DEBUG:boto:Port: 8000 2024-03-04T17:36:16.807 DEBUG:boto:Params: {} 2024-03-04T17:36:16.807 DEBUG:boto:Token: None 2024-03-04T17:36:16.807 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:16 GMT /ppfahn-2/ 2024-03-04T17:36:16.808 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NRhhQS1Jca1+pUiiQ5AWdpg3Cxc= 2024-03-04T17:36:16.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NRhhQS1Jca1+pUiiQ5AWdpg3Cxc=', 'Content-Length': '0'} 2024-03-04T17:36:16.812 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003686cc293d70fbb9-0065e60690-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:16.812 DEBUG:boto:path=/ppfahn-3/ 2024-03-04T17:36:16.812 DEBUG:boto:auth_path=/ppfahn-3/ 2024-03-04T17:36:16.812 DEBUG:boto:Method: HEAD 2024-03-04T17:36:16.812 DEBUG:boto:Path: /ppfahn-3/ 2024-03-04T17:36:16.812 DEBUG:boto:Data: 2024-03-04T17:36:16.812 DEBUG:boto:Headers: {} 2024-03-04T17:36:16.812 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:16.812 DEBUG:boto:Port: 8000 2024-03-04T17:36:16.813 DEBUG:boto:Params: {} 2024-03-04T17:36:16.813 DEBUG:boto:Token: None 2024-03-04T17:36:16.813 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:16 GMT /ppfahn-3/ 2024-03-04T17:36:16.813 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6Pi8PlwaO4pOyr4zILiTkM/q4jE= 2024-03-04T17:36:16.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6Pi8PlwaO4pOyr4zILiTkM/q4jE=', 'Content-Length': '0'} 2024-03-04T17:36:16.817 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ccb4362b63f3dee-0065e60690-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:16.817 DEBUG:boto:path=/ppfahn-1/ 2024-03-04T17:36:16.817 DEBUG:boto:auth_path=/ppfahn-1/ 2024-03-04T17:36:16.817 DEBUG:boto:Method: HEAD 2024-03-04T17:36:16.818 DEBUG:boto:Path: /ppfahn-1/ 2024-03-04T17:36:16.818 DEBUG:boto:Data: 2024-03-04T17:36:16.818 DEBUG:boto:Headers: {} 2024-03-04T17:36:16.818 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:16.818 DEBUG:boto:Port: 8001 2024-03-04T17:36:16.818 DEBUG:boto:Params: {} 2024-03-04T17:36:16.818 DEBUG:boto:Token: None 2024-03-04T17:36:16.818 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:16 GMT /ppfahn-1/ 2024-03-04T17:36:16.818 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M5Js8kEHduxtxtQLVsWDkh2ux7E= 2024-03-04T17:36:16.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M5Js8kEHduxtxtQLVsWDkh2ux7E=', 'Content-Length': '0'} 2024-03-04T17:36:16.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bced081ee7af04f0-0065e60690-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:16.821 DEBUG:boto:path=/ppfahn-2/ 2024-03-04T17:36:16.821 DEBUG:boto:auth_path=/ppfahn-2/ 2024-03-04T17:36:16.821 DEBUG:boto:Method: HEAD 2024-03-04T17:36:16.821 DEBUG:boto:Path: /ppfahn-2/ 2024-03-04T17:36:16.821 DEBUG:boto:Data: 2024-03-04T17:36:16.822 DEBUG:boto:Headers: {} 2024-03-04T17:36:16.822 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:16.822 DEBUG:boto:Port: 8001 2024-03-04T17:36:16.822 DEBUG:boto:Params: {} 2024-03-04T17:36:16.822 DEBUG:boto:Token: None 2024-03-04T17:36:16.822 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:16 GMT /ppfahn-2/ 2024-03-04T17:36:16.822 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NRhhQS1Jca1+pUiiQ5AWdpg3Cxc= 2024-03-04T17:36:16.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NRhhQS1Jca1+pUiiQ5AWdpg3Cxc=', 'Content-Length': '0'} 2024-03-04T17:36:16.825 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b252654235a4cf2d-0065e60690-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:16.825 DEBUG:boto:path=/ppfahn-3/ 2024-03-04T17:36:16.826 DEBUG:boto:auth_path=/ppfahn-3/ 2024-03-04T17:36:16.826 DEBUG:boto:Method: HEAD 2024-03-04T17:36:16.826 DEBUG:boto:Path: /ppfahn-3/ 2024-03-04T17:36:16.826 DEBUG:boto:Data: 2024-03-04T17:36:16.826 DEBUG:boto:Headers: {} 2024-03-04T17:36:16.826 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:16.826 DEBUG:boto:Port: 8001 2024-03-04T17:36:16.826 DEBUG:boto:Params: {} 2024-03-04T17:36:16.826 DEBUG:boto:Token: None 2024-03-04T17:36:16.826 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:16 GMT /ppfahn-3/ 2024-03-04T17:36:16.827 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6Pi8PlwaO4pOyr4zILiTkM/q4jE= 2024-03-04T17:36:16.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6Pi8PlwaO4pOyr4zILiTkM/q4jE=', 'Content-Length': '0'} 2024-03-04T17:36:16.830 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051f1458ba655d778-0065e60690-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:16.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2024-03-04T17:36:16.830 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-4 2024-03-04T17:36:16.831 DEBUG:boto:path=/ppfahn-4/ 2024-03-04T17:36:16.831 DEBUG:boto:auth_path=/ppfahn-4/ 2024-03-04T17:36:16.831 DEBUG:boto:Method: PUT 2024-03-04T17:36:16.831 DEBUG:boto:Path: /ppfahn-4/ 2024-03-04T17:36:16.831 DEBUG:boto:Data: 2024-03-04T17:36:16.831 DEBUG:boto:Headers: {} 2024-03-04T17:36:16.831 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:16.831 DEBUG:boto:Port: 8000 2024-03-04T17:36:16.831 DEBUG:boto:Params: {} 2024-03-04T17:36:16.831 DEBUG:boto:Token: None 2024-03-04T17:36:16.831 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:16 GMT /ppfahn-4/ 2024-03-04T17:36:16.832 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:X3iuwpjoHVLWOK/FhrSeO/Tm03Q= 2024-03-04T17:36:16.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:X3iuwpjoHVLWOK/FhrSeO/Tm03Q=', 'Content-Length': '0'} 2024-03-04T17:36:16.855 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000025bdc475046352d4-0065e60690-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:16.855 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-5 2024-03-04T17:36:16.855 DEBUG:boto:path=/ppfahn-5/ 2024-03-04T17:36:16.856 DEBUG:boto:auth_path=/ppfahn-5/ 2024-03-04T17:36:16.856 DEBUG:boto:Method: PUT 2024-03-04T17:36:16.856 DEBUG:boto:Path: /ppfahn-5/ 2024-03-04T17:36:16.856 DEBUG:boto:Data: 2024-03-04T17:36:16.856 DEBUG:boto:Headers: {} 2024-03-04T17:36:16.856 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:16.856 DEBUG:boto:Port: 8000 2024-03-04T17:36:16.856 DEBUG:boto:Params: {} 2024-03-04T17:36:16.856 DEBUG:boto:Token: None 2024-03-04T17:36:16.856 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:16 GMT /ppfahn-5/ 2024-03-04T17:36:16.857 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8op77U5EK802QTeYgRzz59dC+aE= 2024-03-04T17:36:16.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8op77U5EK802QTeYgRzz59dC+aE=', 'Content-Length': '0'} 2024-03-04T17:36:16.905 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008d13bcf0e1664cea-0065e60690-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:16.905 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-6 2024-03-04T17:36:16.905 DEBUG:boto:path=/ppfahn-6/ 2024-03-04T17:36:16.906 DEBUG:boto:auth_path=/ppfahn-6/ 2024-03-04T17:36:16.906 DEBUG:boto:Method: PUT 2024-03-04T17:36:16.906 DEBUG:boto:Path: /ppfahn-6/ 2024-03-04T17:36:16.906 DEBUG:boto:Data: 2024-03-04T17:36:16.906 DEBUG:boto:Headers: {} 2024-03-04T17:36:16.906 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:16.907 DEBUG:boto:Port: 8001 2024-03-04T17:36:16.907 DEBUG:boto:Params: {} 2024-03-04T17:36:16.907 DEBUG:boto:Token: None 2024-03-04T17:36:16.907 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:16 GMT /ppfahn-6/ 2024-03-04T17:36:16.907 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GEhqQydq+9id5RRSo4cSpej904M= 2024-03-04T17:36:16.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GEhqQydq+9id5RRSo4cSpej904M=', 'Content-Length': '0'} 2024-03-04T17:36:16.954 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000faa122f3677a881a-0065e60690-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:16.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:16.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:16.955 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:16.994 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:16.994 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:17.051 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:36:17.085 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573776.934148_59.1","last_update":"2024-03-04T17:36:16.934148Z"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:36:17.085 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573776.934148_59.1', 3: ''} 2024-03-04T17:36:17.085 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:36:17.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:17.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:17.086 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:17.132 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:17.132 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:17.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:17.182+0000 7f8ab1f03980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:17.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:17.182+0000 7f8ab1f03980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:17.208 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573776.934148_59.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:16.934148Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:17Z"} 2024-03-04T17:36:17.208 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573776.934148_59.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:16.934148Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:17Z"} 2024-03-04T17:36:17.209 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573776.934148_59.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:16.934148Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:17.209 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573776.934148_59.1', 3: ''} 2024-03-04T17:36:17.209 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573776.934148_59.1', 3: ''} 2024-03-04T17:36:17.209 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:36:17.209 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:36:17.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:17.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:17.210 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:17.254 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:17.254 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:17.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:17.303+0000 7f5c665fa980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:17.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:17.307+0000 7f5c665fa980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:17.332 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573776.934148_59.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:16.934148Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:17Z"} 2024-03-04T17:36:17.333 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573776.934148_59.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:16.934148Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:17Z"} 2024-03-04T17:36:17.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573776.934148_59.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:16.934148Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:17.333 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573776.934148_59.1', 3: ''} 2024-03-04T17:36:17.333 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573776.934148_59.1', 3: ''} 2024-03-04T17:36:17.333 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:36:17.333 DEBUG:boto:path=/ppfahn-4/ 2024-03-04T17:36:17.333 DEBUG:boto:auth_path=/ppfahn-4/ 2024-03-04T17:36:17.333 DEBUG:boto:Method: HEAD 2024-03-04T17:36:17.334 DEBUG:boto:Path: /ppfahn-4/ 2024-03-04T17:36:17.334 DEBUG:boto:Data: 2024-03-04T17:36:17.334 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.334 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:17.334 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.334 DEBUG:boto:Params: {} 2024-03-04T17:36:17.334 DEBUG:boto:Token: None 2024-03-04T17:36:17.334 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-4/ 2024-03-04T17:36:17.334 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:g7c8RLZWsiKosuN3jufx21Jfa1U= 2024-03-04T17:36:17.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:g7c8RLZWsiKosuN3jufx21Jfa1U=', 'Content-Length': '0'} 2024-03-04T17:36:17.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039ec35651d9b1e8d-0065e60691-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.337 DEBUG:boto:path=/ppfahn-5/ 2024-03-04T17:36:17.337 DEBUG:boto:auth_path=/ppfahn-5/ 2024-03-04T17:36:17.337 DEBUG:boto:Method: HEAD 2024-03-04T17:36:17.337 DEBUG:boto:Path: /ppfahn-5/ 2024-03-04T17:36:17.337 DEBUG:boto:Data: 2024-03-04T17:36:17.337 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.337 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:17.338 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.338 DEBUG:boto:Params: {} 2024-03-04T17:36:17.338 DEBUG:boto:Token: None 2024-03-04T17:36:17.338 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-5/ 2024-03-04T17:36:17.338 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8MG0mDeUTrujwfNldrw+iHwgezk= 2024-03-04T17:36:17.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8MG0mDeUTrujwfNldrw+iHwgezk=', 'Content-Length': '0'} 2024-03-04T17:36:17.340 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3cda747960facf8-0065e60691-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.340 DEBUG:boto:path=/ppfahn-6/ 2024-03-04T17:36:17.340 DEBUG:boto:auth_path=/ppfahn-6/ 2024-03-04T17:36:17.340 DEBUG:boto:Method: HEAD 2024-03-04T17:36:17.340 DEBUG:boto:Path: /ppfahn-6/ 2024-03-04T17:36:17.340 DEBUG:boto:Data: 2024-03-04T17:36:17.341 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.341 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:17.341 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.341 DEBUG:boto:Params: {} 2024-03-04T17:36:17.341 DEBUG:boto:Token: None 2024-03-04T17:36:17.341 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-6/ 2024-03-04T17:36:17.341 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oEkdSTPXeCza2FHQJWUUVjUp0yw= 2024-03-04T17:36:17.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oEkdSTPXeCza2FHQJWUUVjUp0yw=', 'Content-Length': '0'} 2024-03-04T17:36:17.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5108991cd3147b6-0065e60691-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.343 DEBUG:boto:path=/ppfahn-4/ 2024-03-04T17:36:17.343 DEBUG:boto:auth_path=/ppfahn-4/ 2024-03-04T17:36:17.344 DEBUG:boto:Method: HEAD 2024-03-04T17:36:17.344 DEBUG:boto:Path: /ppfahn-4/ 2024-03-04T17:36:17.344 DEBUG:boto:Data: 2024-03-04T17:36:17.344 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.344 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:17.344 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.344 DEBUG:boto:Params: {} 2024-03-04T17:36:17.344 DEBUG:boto:Token: None 2024-03-04T17:36:17.344 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-4/ 2024-03-04T17:36:17.344 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:g7c8RLZWsiKosuN3jufx21Jfa1U= 2024-03-04T17:36:17.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:g7c8RLZWsiKosuN3jufx21Jfa1U=', 'Content-Length': '0'} 2024-03-04T17:36:17.349 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007fd006bfe597fed8-0065e60691-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.349 DEBUG:boto:path=/ppfahn-5/ 2024-03-04T17:36:17.349 DEBUG:boto:auth_path=/ppfahn-5/ 2024-03-04T17:36:17.349 DEBUG:boto:Method: HEAD 2024-03-04T17:36:17.349 DEBUG:boto:Path: /ppfahn-5/ 2024-03-04T17:36:17.349 DEBUG:boto:Data: 2024-03-04T17:36:17.349 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.349 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:17.349 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.349 DEBUG:boto:Params: {} 2024-03-04T17:36:17.350 DEBUG:boto:Token: None 2024-03-04T17:36:17.350 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-5/ 2024-03-04T17:36:17.350 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8MG0mDeUTrujwfNldrw+iHwgezk= 2024-03-04T17:36:17.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8MG0mDeUTrujwfNldrw+iHwgezk=', 'Content-Length': '0'} 2024-03-04T17:36:17.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd6458e0987fa37f-0065e60691-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.354 DEBUG:boto:path=/ppfahn-6/ 2024-03-04T17:36:17.354 DEBUG:boto:auth_path=/ppfahn-6/ 2024-03-04T17:36:17.354 DEBUG:boto:Method: HEAD 2024-03-04T17:36:17.354 DEBUG:boto:Path: /ppfahn-6/ 2024-03-04T17:36:17.354 DEBUG:boto:Data: 2024-03-04T17:36:17.355 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.355 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:17.355 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.355 DEBUG:boto:Params: {} 2024-03-04T17:36:17.355 DEBUG:boto:Token: None 2024-03-04T17:36:17.355 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-6/ 2024-03-04T17:36:17.355 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oEkdSTPXeCza2FHQJWUUVjUp0yw= 2024-03-04T17:36:17.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oEkdSTPXeCza2FHQJWUUVjUp0yw=', 'Content-Length': '0'} 2024-03-04T17:36:17.359 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018645f7f69e23406-0065e60691-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.359 DEBUG:boto:path=/ppfahn-4/ 2024-03-04T17:36:17.359 DEBUG:boto:auth_path=/ppfahn-4/ 2024-03-04T17:36:17.359 DEBUG:boto:Method: HEAD 2024-03-04T17:36:17.359 DEBUG:boto:Path: /ppfahn-4/ 2024-03-04T17:36:17.359 DEBUG:boto:Data: 2024-03-04T17:36:17.360 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.360 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:17.360 DEBUG:boto:Port: 8001 2024-03-04T17:36:17.360 DEBUG:boto:Params: {} 2024-03-04T17:36:17.360 DEBUG:boto:Token: None 2024-03-04T17:36:17.360 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-4/ 2024-03-04T17:36:17.360 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:g7c8RLZWsiKosuN3jufx21Jfa1U= 2024-03-04T17:36:17.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:g7c8RLZWsiKosuN3jufx21Jfa1U=', 'Content-Length': '0'} 2024-03-04T17:36:17.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054da19fba25892d4-0065e60691-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.363 DEBUG:boto:path=/ppfahn-5/ 2024-03-04T17:36:17.364 DEBUG:boto:auth_path=/ppfahn-5/ 2024-03-04T17:36:17.364 DEBUG:boto:Method: HEAD 2024-03-04T17:36:17.364 DEBUG:boto:Path: /ppfahn-5/ 2024-03-04T17:36:17.364 DEBUG:boto:Data: 2024-03-04T17:36:17.364 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.364 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:17.364 DEBUG:boto:Port: 8001 2024-03-04T17:36:17.364 DEBUG:boto:Params: {} 2024-03-04T17:36:17.364 DEBUG:boto:Token: None 2024-03-04T17:36:17.365 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-5/ 2024-03-04T17:36:17.365 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8MG0mDeUTrujwfNldrw+iHwgezk= 2024-03-04T17:36:17.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8MG0mDeUTrujwfNldrw+iHwgezk=', 'Content-Length': '0'} 2024-03-04T17:36:17.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a628019480622a0c-0065e60691-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.367 DEBUG:boto:path=/ppfahn-6/ 2024-03-04T17:36:17.367 DEBUG:boto:auth_path=/ppfahn-6/ 2024-03-04T17:36:17.367 DEBUG:boto:Method: HEAD 2024-03-04T17:36:17.367 DEBUG:boto:Path: /ppfahn-6/ 2024-03-04T17:36:17.367 DEBUG:boto:Data: 2024-03-04T17:36:17.368 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.368 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:17.368 DEBUG:boto:Port: 8001 2024-03-04T17:36:17.368 DEBUG:boto:Params: {} 2024-03-04T17:36:17.368 DEBUG:boto:Token: None 2024-03-04T17:36:17.368 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-6/ 2024-03-04T17:36:17.368 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oEkdSTPXeCza2FHQJWUUVjUp0yw= 2024-03-04T17:36:17.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oEkdSTPXeCza2FHQJWUUVjUp0yw=', 'Content-Length': '0'} 2024-03-04T17:36:17.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9e5244bfdd3346e-0065e60691-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.372 DEBUG:boto:path=/ppfahn-4/ 2024-03-04T17:36:17.372 DEBUG:boto:auth_path=/ppfahn-4/ 2024-03-04T17:36:17.372 DEBUG:boto:Method: PUT 2024-03-04T17:36:17.372 DEBUG:boto:Path: /ppfahn-4/ 2024-03-04T17:36:17.372 DEBUG:boto:Data: 2024-03-04T17:36:17.372 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.372 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:17.372 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.372 DEBUG:boto:Params: {} 2024-03-04T17:36:17.373 DEBUG:boto:Token: None 2024-03-04T17:36:17.373 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-4/ 2024-03-04T17:36:17.373 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:awoJmW/JRAy0DkXhbgIVgfemcSM= 2024-03-04T17:36:17.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:awoJmW/JRAy0DkXhbgIVgfemcSM=', 'Content-Length': '0'} 2024-03-04T17:36:17.395 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000378c7e3bf335ed9e-0065e60691-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.395 DEBUG:boto:path=/ppfahn-5/ 2024-03-04T17:36:17.395 DEBUG:boto:auth_path=/ppfahn-5/ 2024-03-04T17:36:17.395 DEBUG:boto:Method: PUT 2024-03-04T17:36:17.396 DEBUG:boto:Path: /ppfahn-5/ 2024-03-04T17:36:17.396 DEBUG:boto:Data: 2024-03-04T17:36:17.396 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.396 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:17.396 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.396 DEBUG:boto:Params: {} 2024-03-04T17:36:17.396 DEBUG:boto:Token: None 2024-03-04T17:36:17.396 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-5/ 2024-03-04T17:36:17.396 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iBPbdNMjlFxYZ71jQJrxeVZvaAc= 2024-03-04T17:36:17.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iBPbdNMjlFxYZ71jQJrxeVZvaAc=', 'Content-Length': '0'} 2024-03-04T17:36:17.426 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c642f33911d3d554-0065e60691-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.426 DEBUG:boto:path=/ppfahn-6/ 2024-03-04T17:36:17.426 DEBUG:boto:auth_path=/ppfahn-6/ 2024-03-04T17:36:17.426 DEBUG:boto:Method: PUT 2024-03-04T17:36:17.426 DEBUG:boto:Path: /ppfahn-6/ 2024-03-04T17:36:17.426 DEBUG:boto:Data: 2024-03-04T17:36:17.426 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.426 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:17.426 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.427 DEBUG:boto:Params: {} 2024-03-04T17:36:17.427 DEBUG:boto:Token: None 2024-03-04T17:36:17.427 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-6/ 2024-03-04T17:36:17.427 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tSh2H7zhjBJSNIH4QOdhpJ90Ebk= 2024-03-04T17:36:17.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tSh2H7zhjBJSNIH4QOdhpJ90Ebk=', 'Content-Length': '0'} 2024-03-04T17:36:17.452 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b80868bf800b0fd0-0065e60691-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.452 DEBUG:boto:path=/ppfahn-4/ 2024-03-04T17:36:17.452 DEBUG:boto:auth_path=/ppfahn-4/ 2024-03-04T17:36:17.453 DEBUG:boto:Method: PUT 2024-03-04T17:36:17.453 DEBUG:boto:Path: /ppfahn-4/ 2024-03-04T17:36:17.453 DEBUG:boto:Data: 2024-03-04T17:36:17.453 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.453 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:17.453 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.453 DEBUG:boto:Params: {} 2024-03-04T17:36:17.453 DEBUG:boto:Token: None 2024-03-04T17:36:17.453 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-4/ 2024-03-04T17:36:17.454 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:awoJmW/JRAy0DkXhbgIVgfemcSM= 2024-03-04T17:36:17.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:awoJmW/JRAy0DkXhbgIVgfemcSM=', 'Content-Length': '0'} 2024-03-04T17:36:17.498 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a9506a141789a0a4-0065e60691-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.499 DEBUG:boto:path=/ppfahn-5/ 2024-03-04T17:36:17.499 DEBUG:boto:auth_path=/ppfahn-5/ 2024-03-04T17:36:17.499 DEBUG:boto:Method: PUT 2024-03-04T17:36:17.499 DEBUG:boto:Path: /ppfahn-5/ 2024-03-04T17:36:17.499 DEBUG:boto:Data: 2024-03-04T17:36:17.499 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.499 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:17.499 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.499 DEBUG:boto:Params: {} 2024-03-04T17:36:17.499 DEBUG:boto:Token: None 2024-03-04T17:36:17.500 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-5/ 2024-03-04T17:36:17.500 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iBPbdNMjlFxYZ71jQJrxeVZvaAc= 2024-03-04T17:36:17.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iBPbdNMjlFxYZ71jQJrxeVZvaAc=', 'Content-Length': '0'} 2024-03-04T17:36:17.592 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009e568d598f5c57f9-0065e60691-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.592 DEBUG:boto:path=/ppfahn-6/ 2024-03-04T17:36:17.593 DEBUG:boto:auth_path=/ppfahn-6/ 2024-03-04T17:36:17.593 DEBUG:boto:Method: PUT 2024-03-04T17:36:17.593 DEBUG:boto:Path: /ppfahn-6/ 2024-03-04T17:36:17.593 DEBUG:boto:Data: 2024-03-04T17:36:17.593 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.593 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:17.593 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.593 DEBUG:boto:Params: {} 2024-03-04T17:36:17.593 DEBUG:boto:Token: None 2024-03-04T17:36:17.594 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-6/ 2024-03-04T17:36:17.594 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tSh2H7zhjBJSNIH4QOdhpJ90Ebk= 2024-03-04T17:36:17.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tSh2H7zhjBJSNIH4QOdhpJ90Ebk=', 'Content-Length': '0'} 2024-03-04T17:36:17.704 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005258805f17471385-0065e60691-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.705 DEBUG:boto:path=/ppfahn-4/ 2024-03-04T17:36:17.705 DEBUG:boto:auth_path=/ppfahn-4/ 2024-03-04T17:36:17.705 DEBUG:boto:Method: PUT 2024-03-04T17:36:17.705 DEBUG:boto:Path: /ppfahn-4/ 2024-03-04T17:36:17.705 DEBUG:boto:Data: 2024-03-04T17:36:17.705 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.705 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:17.705 DEBUG:boto:Port: 8001 2024-03-04T17:36:17.706 DEBUG:boto:Params: {} 2024-03-04T17:36:17.706 DEBUG:boto:Token: None 2024-03-04T17:36:17.706 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-4/ 2024-03-04T17:36:17.706 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:awoJmW/JRAy0DkXhbgIVgfemcSM= 2024-03-04T17:36:17.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:awoJmW/JRAy0DkXhbgIVgfemcSM=', 'Content-Length': '0'} 2024-03-04T17:36:17.772 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c2fb0d905485f41c-0065e60691-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.772 DEBUG:boto:path=/ppfahn-5/ 2024-03-04T17:36:17.773 DEBUG:boto:auth_path=/ppfahn-5/ 2024-03-04T17:36:17.773 DEBUG:boto:Method: PUT 2024-03-04T17:36:17.773 DEBUG:boto:Path: /ppfahn-5/ 2024-03-04T17:36:17.773 DEBUG:boto:Data: 2024-03-04T17:36:17.773 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.773 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:17.773 DEBUG:boto:Port: 8001 2024-03-04T17:36:17.773 DEBUG:boto:Params: {} 2024-03-04T17:36:17.773 DEBUG:boto:Token: None 2024-03-04T17:36:17.774 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-5/ 2024-03-04T17:36:17.774 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iBPbdNMjlFxYZ71jQJrxeVZvaAc= 2024-03-04T17:36:17.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iBPbdNMjlFxYZ71jQJrxeVZvaAc=', 'Content-Length': '0'} 2024-03-04T17:36:17.879 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f4370d77c3d21953-0065e60691-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.880 DEBUG:boto:path=/ppfahn-6/ 2024-03-04T17:36:17.880 DEBUG:boto:auth_path=/ppfahn-6/ 2024-03-04T17:36:17.880 DEBUG:boto:Method: PUT 2024-03-04T17:36:17.880 DEBUG:boto:Path: /ppfahn-6/ 2024-03-04T17:36:17.880 DEBUG:boto:Data: 2024-03-04T17:36:17.880 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.880 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:17.880 DEBUG:boto:Port: 8001 2024-03-04T17:36:17.880 DEBUG:boto:Params: {} 2024-03-04T17:36:17.881 DEBUG:boto:Token: None 2024-03-04T17:36:17.881 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-6/ 2024-03-04T17:36:17.881 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tSh2H7zhjBJSNIH4QOdhpJ90Ebk= 2024-03-04T17:36:17.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tSh2H7zhjBJSNIH4QOdhpJ90Ebk=', 'Content-Length': '0'} 2024-03-04T17:36:17.986 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008f78813622f65a50-0065e60691-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.987 DEBUG:boto:path=/ppfahn-4/ 2024-03-04T17:36:17.987 DEBUG:boto:auth_path=/ppfahn-4/ 2024-03-04T17:36:17.987 DEBUG:boto:Method: HEAD 2024-03-04T17:36:17.987 DEBUG:boto:Path: /ppfahn-4/ 2024-03-04T17:36:17.987 DEBUG:boto:Data: 2024-03-04T17:36:17.987 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.987 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:17.987 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.987 DEBUG:boto:Params: {} 2024-03-04T17:36:17.987 DEBUG:boto:Token: None 2024-03-04T17:36:17.987 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-4/ 2024-03-04T17:36:17.988 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:g7c8RLZWsiKosuN3jufx21Jfa1U= 2024-03-04T17:36:17.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:g7c8RLZWsiKosuN3jufx21Jfa1U=', 'Content-Length': '0'} 2024-03-04T17:36:17.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050c8f91fb1947b10-0065e60691-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:17.994 DEBUG:boto:path=/ppfahn-5/ 2024-03-04T17:36:17.994 DEBUG:boto:auth_path=/ppfahn-5/ 2024-03-04T17:36:17.994 DEBUG:boto:Method: HEAD 2024-03-04T17:36:17.994 DEBUG:boto:Path: /ppfahn-5/ 2024-03-04T17:36:17.994 DEBUG:boto:Data: 2024-03-04T17:36:17.994 DEBUG:boto:Headers: {} 2024-03-04T17:36:17.994 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:17.994 DEBUG:boto:Port: 8000 2024-03-04T17:36:17.994 DEBUG:boto:Params: {} 2024-03-04T17:36:17.995 DEBUG:boto:Token: None 2024-03-04T17:36:17.995 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:17 GMT /ppfahn-5/ 2024-03-04T17:36:17.995 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8MG0mDeUTrujwfNldrw+iHwgezk= 2024-03-04T17:36:17.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8MG0mDeUTrujwfNldrw+iHwgezk=', 'Content-Length': '0'} 2024-03-04T17:36:18.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d667d91f1e46f32-0065e60691-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:18.000 DEBUG:boto:path=/ppfahn-6/ 2024-03-04T17:36:18.000 DEBUG:boto:auth_path=/ppfahn-6/ 2024-03-04T17:36:18.000 DEBUG:boto:Method: HEAD 2024-03-04T17:36:18.000 DEBUG:boto:Path: /ppfahn-6/ 2024-03-04T17:36:18.000 DEBUG:boto:Data: 2024-03-04T17:36:18.000 DEBUG:boto:Headers: {} 2024-03-04T17:36:18.001 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:18.001 DEBUG:boto:Port: 8000 2024-03-04T17:36:18.001 DEBUG:boto:Params: {} 2024-03-04T17:36:18.001 DEBUG:boto:Token: None 2024-03-04T17:36:18.001 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:18 GMT /ppfahn-6/ 2024-03-04T17:36:18.001 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:STASBrio5OA+pH/hWbqflzgn9T4= 2024-03-04T17:36:18.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:STASBrio5OA+pH/hWbqflzgn9T4=', 'Content-Length': '0'} 2024-03-04T17:36:18.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000546b161eb3479f26-0065e60691-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:18.006 DEBUG:boto:path=/ppfahn-4/ 2024-03-04T17:36:18.006 DEBUG:boto:auth_path=/ppfahn-4/ 2024-03-04T17:36:18.007 DEBUG:boto:Method: HEAD 2024-03-04T17:36:18.007 DEBUG:boto:Path: /ppfahn-4/ 2024-03-04T17:36:18.007 DEBUG:boto:Data: 2024-03-04T17:36:18.007 DEBUG:boto:Headers: {} 2024-03-04T17:36:18.007 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:18.007 DEBUG:boto:Port: 8000 2024-03-04T17:36:18.007 DEBUG:boto:Params: {} 2024-03-04T17:36:18.007 DEBUG:boto:Token: None 2024-03-04T17:36:18.007 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:18 GMT /ppfahn-4/ 2024-03-04T17:36:18.007 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:drGIA0NH5gYtGCiwRF0tXqE6Itw= 2024-03-04T17:36:18.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:drGIA0NH5gYtGCiwRF0tXqE6Itw=', 'Content-Length': '0'} 2024-03-04T17:36:18.013 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db8fa9c0fb68467d-0065e60692-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:18.013 DEBUG:boto:path=/ppfahn-5/ 2024-03-04T17:36:18.013 DEBUG:boto:auth_path=/ppfahn-5/ 2024-03-04T17:36:18.013 DEBUG:boto:Method: HEAD 2024-03-04T17:36:18.013 DEBUG:boto:Path: /ppfahn-5/ 2024-03-04T17:36:18.014 DEBUG:boto:Data: 2024-03-04T17:36:18.014 DEBUG:boto:Headers: {} 2024-03-04T17:36:18.014 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:18.014 DEBUG:boto:Port: 8000 2024-03-04T17:36:18.014 DEBUG:boto:Params: {} 2024-03-04T17:36:18.014 DEBUG:boto:Token: None 2024-03-04T17:36:18.014 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:18 GMT /ppfahn-5/ 2024-03-04T17:36:18.014 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZYp6OY0B3hcnYagz80gD4OMJsn0= 2024-03-04T17:36:18.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZYp6OY0B3hcnYagz80gD4OMJsn0=', 'Content-Length': '0'} 2024-03-04T17:36:18.020 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df6dc4bbc16a8803-0065e60692-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:18.020 DEBUG:boto:path=/ppfahn-6/ 2024-03-04T17:36:18.020 DEBUG:boto:auth_path=/ppfahn-6/ 2024-03-04T17:36:18.020 DEBUG:boto:Method: HEAD 2024-03-04T17:36:18.020 DEBUG:boto:Path: /ppfahn-6/ 2024-03-04T17:36:18.020 DEBUG:boto:Data: 2024-03-04T17:36:18.020 DEBUG:boto:Headers: {} 2024-03-04T17:36:18.020 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:18.020 DEBUG:boto:Port: 8000 2024-03-04T17:36:18.020 DEBUG:boto:Params: {} 2024-03-04T17:36:18.021 DEBUG:boto:Token: None 2024-03-04T17:36:18.021 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:18 GMT /ppfahn-6/ 2024-03-04T17:36:18.021 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:STASBrio5OA+pH/hWbqflzgn9T4= 2024-03-04T17:36:18.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:STASBrio5OA+pH/hWbqflzgn9T4=', 'Content-Length': '0'} 2024-03-04T17:36:18.026 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f761e1ab18c98c50-0065e60692-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:18.026 DEBUG:boto:path=/ppfahn-4/ 2024-03-04T17:36:18.026 DEBUG:boto:auth_path=/ppfahn-4/ 2024-03-04T17:36:18.026 DEBUG:boto:Method: HEAD 2024-03-04T17:36:18.026 DEBUG:boto:Path: /ppfahn-4/ 2024-03-04T17:36:18.026 DEBUG:boto:Data: 2024-03-04T17:36:18.027 DEBUG:boto:Headers: {} 2024-03-04T17:36:18.027 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:18.027 DEBUG:boto:Port: 8001 2024-03-04T17:36:18.027 DEBUG:boto:Params: {} 2024-03-04T17:36:18.027 DEBUG:boto:Token: None 2024-03-04T17:36:18.027 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:18 GMT /ppfahn-4/ 2024-03-04T17:36:18.027 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:drGIA0NH5gYtGCiwRF0tXqE6Itw= 2024-03-04T17:36:18.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:drGIA0NH5gYtGCiwRF0tXqE6Itw=', 'Content-Length': '0'} 2024-03-04T17:36:18.035 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058283b2a11e8f929-0065e60692-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:18.035 DEBUG:boto:path=/ppfahn-5/ 2024-03-04T17:36:18.035 DEBUG:boto:auth_path=/ppfahn-5/ 2024-03-04T17:36:18.035 DEBUG:boto:Method: HEAD 2024-03-04T17:36:18.035 DEBUG:boto:Path: /ppfahn-5/ 2024-03-04T17:36:18.036 DEBUG:boto:Data: 2024-03-04T17:36:18.036 DEBUG:boto:Headers: {} 2024-03-04T17:36:18.036 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:18.036 DEBUG:boto:Port: 8001 2024-03-04T17:36:18.036 DEBUG:boto:Params: {} 2024-03-04T17:36:18.036 DEBUG:boto:Token: None 2024-03-04T17:36:18.036 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:18 GMT /ppfahn-5/ 2024-03-04T17:36:18.036 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZYp6OY0B3hcnYagz80gD4OMJsn0= 2024-03-04T17:36:18.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZYp6OY0B3hcnYagz80gD4OMJsn0=', 'Content-Length': '0'} 2024-03-04T17:36:18.041 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063c6412ecaab5186-0065e60692-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:18.041 DEBUG:boto:path=/ppfahn-6/ 2024-03-04T17:36:18.041 DEBUG:boto:auth_path=/ppfahn-6/ 2024-03-04T17:36:18.041 DEBUG:boto:Method: HEAD 2024-03-04T17:36:18.042 DEBUG:boto:Path: /ppfahn-6/ 2024-03-04T17:36:18.042 DEBUG:boto:Data: 2024-03-04T17:36:18.042 DEBUG:boto:Headers: {} 2024-03-04T17:36:18.042 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:18.042 DEBUG:boto:Port: 8001 2024-03-04T17:36:18.042 DEBUG:boto:Params: {} 2024-03-04T17:36:18.042 DEBUG:boto:Token: None 2024-03-04T17:36:18.042 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:18 GMT /ppfahn-6/ 2024-03-04T17:36:18.042 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:STASBrio5OA+pH/hWbqflzgn9T4= 2024-03-04T17:36:18.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:STASBrio5OA+pH/hWbqflzgn9T4=', 'Content-Length': '0'} 2024-03-04T17:36:18.047 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9f5256333028bca-0065e60692-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:18.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:18.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:18.048 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:18.092 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:18.092 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:18.150 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:36:18.174 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573777.910443_95.1","last_update":"2024-03-04T17:36:17.910443Z"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:36:18.174 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573777.910443_95.1', 3: ''} 2024-03-04T17:36:18.174 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:36:18.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:18.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:18.174 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:18.218 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:18.218 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:18.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:18.270+0000 7f0d96ed5980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:18.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:18.270+0000 7f0d96ed5980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:18.297 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573777.438404_70.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:17.438404Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:18Z"} 2024-03-04T17:36:18.298 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573777.438404_70.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:17.438404Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:18Z"} 2024-03-04T17:36:18.298 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573777.438404_70.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:17.438404Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:18.298 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573777.910443_95.1', 3: ''} 2024-03-04T17:36:18.298 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573777.438404_70.1', 3: ''} 2024-03-04T17:36:18.298 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1709573777.910443_95.1 target=1_1709573777.438404_70.1 2024-03-04T17:36:23.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:23.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:23.300 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:23.342 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:23.342 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:23.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:23.394+0000 7efd2fcff980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:23.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:23.394+0000 7efd2fcff980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:23.420 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573777.910443_95.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:17.910443Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:23Z"} 2024-03-04T17:36:23.421 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573777.910443_95.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:17.910443Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:23Z"} 2024-03-04T17:36:23.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573777.910443_95.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:17.910443Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:23.421 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573777.910443_95.1', 3: ''} 2024-03-04T17:36:23.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573777.910443_95.1', 3: ''} 2024-03-04T17:36:23.421 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:36:23.421 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:36:23.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:23.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:23.422 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:23.464 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:23.464 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:23.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:23.515+0000 7f67f6345980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:23.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:23.515+0000 7f67f6345980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:23.542 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573777.910443_95.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:17.910443Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:23Z"} 2024-03-04T17:36:23.543 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573777.910443_95.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:17.910443Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:23Z"} 2024-03-04T17:36:23.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573777.910443_95.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:17.910443Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:23.543 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573777.910443_95.1', 3: ''} 2024-03-04T17:36:23.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573777.910443_95.1', 3: ''} 2024-03-04T17:36:23.543 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:36:23.543 DEBUG:boto:path=/ppfahn-4/ 2024-03-04T17:36:23.543 DEBUG:boto:auth_path=/ppfahn-4/ 2024-03-04T17:36:23.544 DEBUG:boto:Method: HEAD 2024-03-04T17:36:23.544 DEBUG:boto:Path: /ppfahn-4/ 2024-03-04T17:36:23.544 DEBUG:boto:Data: 2024-03-04T17:36:23.544 DEBUG:boto:Headers: {} 2024-03-04T17:36:23.544 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:23.544 DEBUG:boto:Port: 8000 2024-03-04T17:36:23.544 DEBUG:boto:Params: {} 2024-03-04T17:36:23.544 DEBUG:boto:Token: None 2024-03-04T17:36:23.545 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:23 GMT /ppfahn-4/ 2024-03-04T17:36:23.545 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ABHZRA6oPPoVWaibgZ4QN7xIOUg= 2024-03-04T17:36:23.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ABHZRA6oPPoVWaibgZ4QN7xIOUg=', 'Content-Length': '0'} 2024-03-04T17:36:23.547 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da9c0ae28ab07db2-0065e60697-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:23.547 DEBUG:boto:path=/ppfahn-5/ 2024-03-04T17:36:23.548 DEBUG:boto:auth_path=/ppfahn-5/ 2024-03-04T17:36:23.548 DEBUG:boto:Method: HEAD 2024-03-04T17:36:23.548 DEBUG:boto:Path: /ppfahn-5/ 2024-03-04T17:36:23.548 DEBUG:boto:Data: 2024-03-04T17:36:23.548 DEBUG:boto:Headers: {} 2024-03-04T17:36:23.548 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:23.548 DEBUG:boto:Port: 8000 2024-03-04T17:36:23.548 DEBUG:boto:Params: {} 2024-03-04T17:36:23.548 DEBUG:boto:Token: None 2024-03-04T17:36:23.548 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:23 GMT /ppfahn-5/ 2024-03-04T17:36:23.549 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hxIqSL8aUguVh5BaIXZum9rZE4A= 2024-03-04T17:36:23.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hxIqSL8aUguVh5BaIXZum9rZE4A=', 'Content-Length': '0'} 2024-03-04T17:36:23.551 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005c596bc47400de9-0065e60697-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:23.551 DEBUG:boto:path=/ppfahn-6/ 2024-03-04T17:36:23.551 DEBUG:boto:auth_path=/ppfahn-6/ 2024-03-04T17:36:23.551 DEBUG:boto:Method: HEAD 2024-03-04T17:36:23.551 DEBUG:boto:Path: /ppfahn-6/ 2024-03-04T17:36:23.551 DEBUG:boto:Data: 2024-03-04T17:36:23.551 DEBUG:boto:Headers: {} 2024-03-04T17:36:23.551 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:23.551 DEBUG:boto:Port: 8000 2024-03-04T17:36:23.551 DEBUG:boto:Params: {} 2024-03-04T17:36:23.552 DEBUG:boto:Token: None 2024-03-04T17:36:23.552 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:23 GMT /ppfahn-6/ 2024-03-04T17:36:23.552 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rGK5jD+9DO1csi5uG4xOSEe0bRU= 2024-03-04T17:36:23.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rGK5jD+9DO1csi5uG4xOSEe0bRU=', 'Content-Length': '0'} 2024-03-04T17:36:23.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008efba6ac9b0ad191-0065e60697-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:23.554 DEBUG:boto:path=/ppfahn-4/ 2024-03-04T17:36:23.554 DEBUG:boto:auth_path=/ppfahn-4/ 2024-03-04T17:36:23.554 DEBUG:boto:Method: HEAD 2024-03-04T17:36:23.555 DEBUG:boto:Path: /ppfahn-4/ 2024-03-04T17:36:23.555 DEBUG:boto:Data: 2024-03-04T17:36:23.555 DEBUG:boto:Headers: {} 2024-03-04T17:36:23.555 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:23.555 DEBUG:boto:Port: 8000 2024-03-04T17:36:23.555 DEBUG:boto:Params: {} 2024-03-04T17:36:23.555 DEBUG:boto:Token: None 2024-03-04T17:36:23.555 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:23 GMT /ppfahn-4/ 2024-03-04T17:36:23.555 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ABHZRA6oPPoVWaibgZ4QN7xIOUg= 2024-03-04T17:36:23.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ABHZRA6oPPoVWaibgZ4QN7xIOUg=', 'Content-Length': '0'} 2024-03-04T17:36:23.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012e52b7ebb80e4be-0065e60697-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:23.560 DEBUG:boto:path=/ppfahn-5/ 2024-03-04T17:36:23.560 DEBUG:boto:auth_path=/ppfahn-5/ 2024-03-04T17:36:23.560 DEBUG:boto:Method: HEAD 2024-03-04T17:36:23.560 DEBUG:boto:Path: /ppfahn-5/ 2024-03-04T17:36:23.560 DEBUG:boto:Data: 2024-03-04T17:36:23.560 DEBUG:boto:Headers: {} 2024-03-04T17:36:23.561 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:23.561 DEBUG:boto:Port: 8000 2024-03-04T17:36:23.561 DEBUG:boto:Params: {} 2024-03-04T17:36:23.561 DEBUG:boto:Token: None 2024-03-04T17:36:23.561 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:23 GMT /ppfahn-5/ 2024-03-04T17:36:23.561 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hxIqSL8aUguVh5BaIXZum9rZE4A= 2024-03-04T17:36:23.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hxIqSL8aUguVh5BaIXZum9rZE4A=', 'Content-Length': '0'} 2024-03-04T17:36:23.565 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000293e923370e74afd-0065e60697-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:23.565 DEBUG:boto:path=/ppfahn-6/ 2024-03-04T17:36:23.565 DEBUG:boto:auth_path=/ppfahn-6/ 2024-03-04T17:36:23.565 DEBUG:boto:Method: HEAD 2024-03-04T17:36:23.565 DEBUG:boto:Path: /ppfahn-6/ 2024-03-04T17:36:23.566 DEBUG:boto:Data: 2024-03-04T17:36:23.566 DEBUG:boto:Headers: {} 2024-03-04T17:36:23.566 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:23.566 DEBUG:boto:Port: 8000 2024-03-04T17:36:23.566 DEBUG:boto:Params: {} 2024-03-04T17:36:23.566 DEBUG:boto:Token: None 2024-03-04T17:36:23.566 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:23 GMT /ppfahn-6/ 2024-03-04T17:36:23.566 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rGK5jD+9DO1csi5uG4xOSEe0bRU= 2024-03-04T17:36:23.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rGK5jD+9DO1csi5uG4xOSEe0bRU=', 'Content-Length': '0'} 2024-03-04T17:36:23.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed9813a380061d9b-0065e60697-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:23.570 DEBUG:boto:path=/ppfahn-4/ 2024-03-04T17:36:23.570 DEBUG:boto:auth_path=/ppfahn-4/ 2024-03-04T17:36:23.570 DEBUG:boto:Method: HEAD 2024-03-04T17:36:23.570 DEBUG:boto:Path: /ppfahn-4/ 2024-03-04T17:36:23.570 DEBUG:boto:Data: 2024-03-04T17:36:23.570 DEBUG:boto:Headers: {} 2024-03-04T17:36:23.570 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:23.570 DEBUG:boto:Port: 8001 2024-03-04T17:36:23.571 DEBUG:boto:Params: {} 2024-03-04T17:36:23.571 DEBUG:boto:Token: None 2024-03-04T17:36:23.571 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:23 GMT /ppfahn-4/ 2024-03-04T17:36:23.571 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ABHZRA6oPPoVWaibgZ4QN7xIOUg= 2024-03-04T17:36:23.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ABHZRA6oPPoVWaibgZ4QN7xIOUg=', 'Content-Length': '0'} 2024-03-04T17:36:23.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e63f8da593de60a6-0065e60697-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:23.573 DEBUG:boto:path=/ppfahn-5/ 2024-03-04T17:36:23.573 DEBUG:boto:auth_path=/ppfahn-5/ 2024-03-04T17:36:23.574 DEBUG:boto:Method: HEAD 2024-03-04T17:36:23.574 DEBUG:boto:Path: /ppfahn-5/ 2024-03-04T17:36:23.574 DEBUG:boto:Data: 2024-03-04T17:36:23.574 DEBUG:boto:Headers: {} 2024-03-04T17:36:23.574 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:23.574 DEBUG:boto:Port: 8001 2024-03-04T17:36:23.574 DEBUG:boto:Params: {} 2024-03-04T17:36:23.574 DEBUG:boto:Token: None 2024-03-04T17:36:23.574 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:23 GMT /ppfahn-5/ 2024-03-04T17:36:23.574 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hxIqSL8aUguVh5BaIXZum9rZE4A= 2024-03-04T17:36:23.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hxIqSL8aUguVh5BaIXZum9rZE4A=', 'Content-Length': '0'} 2024-03-04T17:36:23.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c4a386588508c18-0065e60697-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:23.577 DEBUG:boto:path=/ppfahn-6/ 2024-03-04T17:36:23.577 DEBUG:boto:auth_path=/ppfahn-6/ 2024-03-04T17:36:23.577 DEBUG:boto:Method: HEAD 2024-03-04T17:36:23.577 DEBUG:boto:Path: /ppfahn-6/ 2024-03-04T17:36:23.577 DEBUG:boto:Data: 2024-03-04T17:36:23.578 DEBUG:boto:Headers: {} 2024-03-04T17:36:23.578 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:23.578 DEBUG:boto:Port: 8001 2024-03-04T17:36:23.578 DEBUG:boto:Params: {} 2024-03-04T17:36:23.578 DEBUG:boto:Token: None 2024-03-04T17:36:23.578 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:23 GMT /ppfahn-6/ 2024-03-04T17:36:23.578 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rGK5jD+9DO1csi5uG4xOSEe0bRU= 2024-03-04T17:36:23.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rGK5jD+9DO1csi5uG4xOSEe0bRU=', 'Content-Length': '0'} 2024-03-04T17:36:23.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001477263762ffc92f-0065e60697-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:23.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2024-03-04T17:36:23.581 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-7 2024-03-04T17:36:23.581 DEBUG:boto:path=/ppfahn-7/ 2024-03-04T17:36:23.581 DEBUG:boto:auth_path=/ppfahn-7/ 2024-03-04T17:36:23.581 DEBUG:boto:Method: PUT 2024-03-04T17:36:23.581 DEBUG:boto:Path: /ppfahn-7/ 2024-03-04T17:36:23.581 DEBUG:boto:Data: 2024-03-04T17:36:23.581 DEBUG:boto:Headers: {} 2024-03-04T17:36:23.582 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:23.582 DEBUG:boto:Port: 8000 2024-03-04T17:36:23.582 DEBUG:boto:Params: {} 2024-03-04T17:36:23.582 DEBUG:boto:Token: None 2024-03-04T17:36:23.582 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:23 GMT /ppfahn-7/ 2024-03-04T17:36:23.582 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ttapETC8nu12ovqnVZGhIBsbWdQ= 2024-03-04T17:36:23.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ttapETC8nu12ovqnVZGhIBsbWdQ=', 'Content-Length': '0'} 2024-03-04T17:36:23.606 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d906840a9176f9a2-0065e60697-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:23.606 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-8 2024-03-04T17:36:23.606 DEBUG:boto:path=/ppfahn-8/ 2024-03-04T17:36:23.606 DEBUG:boto:auth_path=/ppfahn-8/ 2024-03-04T17:36:23.606 DEBUG:boto:Method: PUT 2024-03-04T17:36:23.606 DEBUG:boto:Path: /ppfahn-8/ 2024-03-04T17:36:23.606 DEBUG:boto:Data: 2024-03-04T17:36:23.607 DEBUG:boto:Headers: {} 2024-03-04T17:36:23.607 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:23.607 DEBUG:boto:Port: 8000 2024-03-04T17:36:23.607 DEBUG:boto:Params: {} 2024-03-04T17:36:23.607 DEBUG:boto:Token: None 2024-03-04T17:36:23.607 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:23 GMT /ppfahn-8/ 2024-03-04T17:36:23.607 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RAurcvI8qpGoUMseAbEC7q9Uyp0= 2024-03-04T17:36:23.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RAurcvI8qpGoUMseAbEC7q9Uyp0=', 'Content-Length': '0'} 2024-03-04T17:36:23.651 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000de6843f7a4ef024f-0065e60697-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:23.651 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-9 2024-03-04T17:36:23.651 DEBUG:boto:path=/ppfahn-9/ 2024-03-04T17:36:23.651 DEBUG:boto:auth_path=/ppfahn-9/ 2024-03-04T17:36:23.651 DEBUG:boto:Method: PUT 2024-03-04T17:36:23.651 DEBUG:boto:Path: /ppfahn-9/ 2024-03-04T17:36:23.651 DEBUG:boto:Data: 2024-03-04T17:36:23.652 DEBUG:boto:Headers: {} 2024-03-04T17:36:23.652 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:23.652 DEBUG:boto:Port: 8001 2024-03-04T17:36:23.652 DEBUG:boto:Params: {} 2024-03-04T17:36:23.652 DEBUG:boto:Token: None 2024-03-04T17:36:23.652 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:23 GMT /ppfahn-9/ 2024-03-04T17:36:23.652 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sIYsjHHRNC73qtTxKvPUFrvSm0E= 2024-03-04T17:36:23.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sIYsjHHRNC73qtTxKvPUFrvSm0E=', 'Content-Length': '0'} 2024-03-04T17:36:23.687 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002bf0413b24d4bc6b-0065e60697-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:23.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:23.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:23.687 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:23.731 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:23.731 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:23.787 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:36:23.809 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573783.672748_107.1","last_update":"2024-03-04T17:36:23.672748Z"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:36:23.809 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573783.672748_107.1', 3: ''} 2024-03-04T17:36:23.809 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:36:23.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:23.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:23.810 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:23.853 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:23.853 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:23.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:23.910+0000 7f3133a51980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:23.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:23.910+0000 7f3133a51980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:23.936 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573783.672748_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:23.672748Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:23Z"} 2024-03-04T17:36:23.936 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573783.672748_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:23.672748Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:23Z"} 2024-03-04T17:36:23.936 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573783.672748_107.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:23.672748Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:23.936 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573783.672748_107.1', 3: ''} 2024-03-04T17:36:23.936 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573783.672748_107.1', 3: ''} 2024-03-04T17:36:23.936 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:36:23.937 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:36:23.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:23.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:23.937 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:23.978 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:23.979 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:24.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:24.031+0000 7f973d1f4980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:24.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:24.031+0000 7f973d1f4980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:24.056 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573783.663486_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:23.663486Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:24Z"} 2024-03-04T17:36:24.057 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573783.663486_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:23.663486Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:24Z"} 2024-03-04T17:36:24.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573783.663486_104.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:23.663486Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:24.057 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573783.672748_107.1', 3: ''} 2024-03-04T17:36:24.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573783.663486_104.1', 3: ''} 2024-03-04T17:36:24.057 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1709573783.672748_107.1 target=1_1709573783.663486_104.1 2024-03-04T17:36:29.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:29.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:29.058 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:29.101 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:29.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:29.151+0000 7f225734c980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:29.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:29.151+0000 7f225734c980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:29.176 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573783.672748_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:23.672748Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:29Z"} 2024-03-04T17:36:29.177 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573783.672748_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:23.672748Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:29Z"} 2024-03-04T17:36:29.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573783.672748_107.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:23.672748Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:29.177 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573783.672748_107.1', 3: ''} 2024-03-04T17:36:29.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573783.672748_107.1', 3: ''} 2024-03-04T17:36:29.178 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:36:29.178 DEBUG:boto:path=/ppfahn-7/ 2024-03-04T17:36:29.178 DEBUG:boto:auth_path=/ppfahn-7/ 2024-03-04T17:36:29.178 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.178 DEBUG:boto:Path: /ppfahn-7/ 2024-03-04T17:36:29.178 DEBUG:boto:Data: 2024-03-04T17:36:29.178 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.178 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:29.178 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.178 DEBUG:boto:Params: {} 2024-03-04T17:36:29.179 DEBUG:boto:Token: None 2024-03-04T17:36:29.179 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-7/ 2024-03-04T17:36:29.179 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EMxbTMh9/tVdZuzAtooZUj4nEhU= 2024-03-04T17:36:29.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EMxbTMh9/tVdZuzAtooZUj4nEhU=', 'Content-Length': '0'} 2024-03-04T17:36:29.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f58b4aa5419d75d-0065e6069d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.182 DEBUG:boto:path=/ppfahn-8/ 2024-03-04T17:36:29.182 DEBUG:boto:auth_path=/ppfahn-8/ 2024-03-04T17:36:29.182 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.182 DEBUG:boto:Path: /ppfahn-8/ 2024-03-04T17:36:29.182 DEBUG:boto:Data: 2024-03-04T17:36:29.182 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.182 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:29.182 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.182 DEBUG:boto:Params: {} 2024-03-04T17:36:29.183 DEBUG:boto:Token: None 2024-03-04T17:36:29.183 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-8/ 2024-03-04T17:36:29.183 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V1bWKtAkCvl2TX8UTPAM056K9hM= 2024-03-04T17:36:29.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V1bWKtAkCvl2TX8UTPAM056K9hM=', 'Content-Length': '0'} 2024-03-04T17:36:29.185 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050daf0a8e8716562-0065e6069d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.185 DEBUG:boto:path=/ppfahn-9/ 2024-03-04T17:36:29.185 DEBUG:boto:auth_path=/ppfahn-9/ 2024-03-04T17:36:29.185 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.185 DEBUG:boto:Path: /ppfahn-9/ 2024-03-04T17:36:29.185 DEBUG:boto:Data: 2024-03-04T17:36:29.185 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.185 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:29.185 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.185 DEBUG:boto:Params: {} 2024-03-04T17:36:29.186 DEBUG:boto:Token: None 2024-03-04T17:36:29.186 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-9/ 2024-03-04T17:36:29.186 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jSvsjk65rB7aOphhm/GeYrU5cjQ= 2024-03-04T17:36:29.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jSvsjk65rB7aOphhm/GeYrU5cjQ=', 'Content-Length': '0'} 2024-03-04T17:36:29.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f374a85011dd62fb-0065e6069d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.188 DEBUG:boto:path=/ppfahn-7/ 2024-03-04T17:36:29.188 DEBUG:boto:auth_path=/ppfahn-7/ 2024-03-04T17:36:29.189 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.189 DEBUG:boto:Path: /ppfahn-7/ 2024-03-04T17:36:29.189 DEBUG:boto:Data: 2024-03-04T17:36:29.189 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.189 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:29.189 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.189 DEBUG:boto:Params: {} 2024-03-04T17:36:29.189 DEBUG:boto:Token: None 2024-03-04T17:36:29.189 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-7/ 2024-03-04T17:36:29.190 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EMxbTMh9/tVdZuzAtooZUj4nEhU= 2024-03-04T17:36:29.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EMxbTMh9/tVdZuzAtooZUj4nEhU=', 'Content-Length': '0'} 2024-03-04T17:36:29.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002dca2ac9b0562d02-0065e6069d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.194 DEBUG:boto:path=/ppfahn-8/ 2024-03-04T17:36:29.195 DEBUG:boto:auth_path=/ppfahn-8/ 2024-03-04T17:36:29.195 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.195 DEBUG:boto:Path: /ppfahn-8/ 2024-03-04T17:36:29.195 DEBUG:boto:Data: 2024-03-04T17:36:29.195 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.195 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:29.195 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.195 DEBUG:boto:Params: {} 2024-03-04T17:36:29.195 DEBUG:boto:Token: None 2024-03-04T17:36:29.195 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-8/ 2024-03-04T17:36:29.196 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V1bWKtAkCvl2TX8UTPAM056K9hM= 2024-03-04T17:36:29.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V1bWKtAkCvl2TX8UTPAM056K9hM=', 'Content-Length': '0'} 2024-03-04T17:36:29.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000185b70b2d51a8760-0065e6069d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.200 DEBUG:boto:path=/ppfahn-9/ 2024-03-04T17:36:29.200 DEBUG:boto:auth_path=/ppfahn-9/ 2024-03-04T17:36:29.200 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.200 DEBUG:boto:Path: /ppfahn-9/ 2024-03-04T17:36:29.200 DEBUG:boto:Data: 2024-03-04T17:36:29.200 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.200 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:29.201 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.201 DEBUG:boto:Params: {} 2024-03-04T17:36:29.201 DEBUG:boto:Token: None 2024-03-04T17:36:29.201 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-9/ 2024-03-04T17:36:29.201 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jSvsjk65rB7aOphhm/GeYrU5cjQ= 2024-03-04T17:36:29.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jSvsjk65rB7aOphhm/GeYrU5cjQ=', 'Content-Length': '0'} 2024-03-04T17:36:29.205 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e6d2471c5335039-0065e6069d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.205 DEBUG:boto:path=/ppfahn-7/ 2024-03-04T17:36:29.205 DEBUG:boto:auth_path=/ppfahn-7/ 2024-03-04T17:36:29.206 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.206 DEBUG:boto:Path: /ppfahn-7/ 2024-03-04T17:36:29.206 DEBUG:boto:Data: 2024-03-04T17:36:29.206 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.206 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:29.206 DEBUG:boto:Port: 8001 2024-03-04T17:36:29.206 DEBUG:boto:Params: {} 2024-03-04T17:36:29.206 DEBUG:boto:Token: None 2024-03-04T17:36:29.206 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-7/ 2024-03-04T17:36:29.206 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EMxbTMh9/tVdZuzAtooZUj4nEhU= 2024-03-04T17:36:29.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EMxbTMh9/tVdZuzAtooZUj4nEhU=', 'Content-Length': '0'} 2024-03-04T17:36:29.209 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000278218307dfc7b57-0065e6069d-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.209 DEBUG:boto:path=/ppfahn-8/ 2024-03-04T17:36:29.209 DEBUG:boto:auth_path=/ppfahn-8/ 2024-03-04T17:36:29.209 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.210 DEBUG:boto:Path: /ppfahn-8/ 2024-03-04T17:36:29.210 DEBUG:boto:Data: 2024-03-04T17:36:29.210 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.210 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:29.210 DEBUG:boto:Port: 8001 2024-03-04T17:36:29.210 DEBUG:boto:Params: {} 2024-03-04T17:36:29.210 DEBUG:boto:Token: None 2024-03-04T17:36:29.210 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-8/ 2024-03-04T17:36:29.210 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V1bWKtAkCvl2TX8UTPAM056K9hM= 2024-03-04T17:36:29.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V1bWKtAkCvl2TX8UTPAM056K9hM=', 'Content-Length': '0'} 2024-03-04T17:36:29.213 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007079660ea252a74d-0065e6069d-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.213 DEBUG:boto:path=/ppfahn-9/ 2024-03-04T17:36:29.213 DEBUG:boto:auth_path=/ppfahn-9/ 2024-03-04T17:36:29.213 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.213 DEBUG:boto:Path: /ppfahn-9/ 2024-03-04T17:36:29.213 DEBUG:boto:Data: 2024-03-04T17:36:29.213 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.213 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:29.213 DEBUG:boto:Port: 8001 2024-03-04T17:36:29.213 DEBUG:boto:Params: {} 2024-03-04T17:36:29.214 DEBUG:boto:Token: None 2024-03-04T17:36:29.214 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-9/ 2024-03-04T17:36:29.214 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jSvsjk65rB7aOphhm/GeYrU5cjQ= 2024-03-04T17:36:29.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jSvsjk65rB7aOphhm/GeYrU5cjQ=', 'Content-Length': '0'} 2024-03-04T17:36:29.216 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a83cdc45c65c79e-0065e6069d-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.216 DEBUG:boto:path=/ppfahn-7/ 2024-03-04T17:36:29.216 DEBUG:boto:auth_path=/ppfahn-7/ 2024-03-04T17:36:29.216 DEBUG:boto:Method: DELETE 2024-03-04T17:36:29.216 DEBUG:boto:Path: /ppfahn-7/ 2024-03-04T17:36:29.216 DEBUG:boto:Data: 2024-03-04T17:36:29.217 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.217 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:29.217 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.217 DEBUG:boto:Params: {} 2024-03-04T17:36:29.217 DEBUG:boto:Token: None 2024-03-04T17:36:29.217 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-7/ 2024-03-04T17:36:29.217 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U8GoPhTraGStbQEJXAUm5s5qwnA= 2024-03-04T17:36:29.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U8GoPhTraGStbQEJXAUm5s5qwnA=', 'Content-Length': '0'} 2024-03-04T17:36:29.242 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000abf168463c26b3ca-0065e6069d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.242 DEBUG:boto:path=/ppfahn-8/ 2024-03-04T17:36:29.242 DEBUG:boto:auth_path=/ppfahn-8/ 2024-03-04T17:36:29.242 DEBUG:boto:Method: DELETE 2024-03-04T17:36:29.242 DEBUG:boto:Path: /ppfahn-8/ 2024-03-04T17:36:29.243 DEBUG:boto:Data: 2024-03-04T17:36:29.243 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.243 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:29.243 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.243 DEBUG:boto:Params: {} 2024-03-04T17:36:29.243 DEBUG:boto:Token: None 2024-03-04T17:36:29.243 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-8/ 2024-03-04T17:36:29.243 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MVUKY+ApCqG0p0KD9KsxK/zVTok= 2024-03-04T17:36:29.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MVUKY+ApCqG0p0KD9KsxK/zVTok=', 'Content-Length': '0'} 2024-03-04T17:36:29.345 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000b3c6309262288c6-0065e6069d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.345 DEBUG:boto:path=/ppfahn-9/ 2024-03-04T17:36:29.346 DEBUG:boto:auth_path=/ppfahn-9/ 2024-03-04T17:36:29.346 DEBUG:boto:Method: DELETE 2024-03-04T17:36:29.346 DEBUG:boto:Path: /ppfahn-9/ 2024-03-04T17:36:29.346 DEBUG:boto:Data: 2024-03-04T17:36:29.346 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.346 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:29.346 DEBUG:boto:Port: 8001 2024-03-04T17:36:29.346 DEBUG:boto:Params: {} 2024-03-04T17:36:29.346 DEBUG:boto:Token: None 2024-03-04T17:36:29.347 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-9/ 2024-03-04T17:36:29.347 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:szgVkZU7THGJlhQ+27lWPSaRuVo= 2024-03-04T17:36:29.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:szgVkZU7THGJlhQ+27lWPSaRuVo=', 'Content-Length': '0'} 2024-03-04T17:36:29.463 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006cd02e3a7bd70444-0065e6069d-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:29.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:29.464 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:29.503 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:29.503 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:29.559 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:36:29.582 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:36:29.582 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: ''} 2024-03-04T17:36:29.582 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:36:29.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:29.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:29.583 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:29.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:29.677+0000 7fa40a904980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:29.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:29.681+0000 7fa40a904980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:29.706 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:29Z"} 2024-03-04T17:36:29.706 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:29Z"} 2024-03-04T17:36:29.707 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:29.707 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: ''} 2024-03-04T17:36:29.707 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: ''} 2024-03-04T17:36:29.707 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:36:29.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:36:29.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:29.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:29.708 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:29.750 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:29.750 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:29.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:29.803+0000 7fec89539980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:29.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:29.803+0000 7fec89539980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:29.829 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:29Z"} 2024-03-04T17:36:29.829 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:29Z"} 2024-03-04T17:36:29.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:29.829 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: ''} 2024-03-04T17:36:29.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: ''} 2024-03-04T17:36:29.829 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:36:29.830 DEBUG:boto:path=/ppfahn-7/ 2024-03-04T17:36:29.830 DEBUG:boto:auth_path=/ppfahn-7/ 2024-03-04T17:36:29.830 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.830 DEBUG:boto:Path: /ppfahn-7/ 2024-03-04T17:36:29.830 DEBUG:boto:Data: 2024-03-04T17:36:29.830 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.830 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:29.830 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.830 DEBUG:boto:Params: {} 2024-03-04T17:36:29.831 DEBUG:boto:Token: None 2024-03-04T17:36:29.831 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-7/ 2024-03-04T17:36:29.831 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EMxbTMh9/tVdZuzAtooZUj4nEhU= 2024-03-04T17:36:29.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EMxbTMh9/tVdZuzAtooZUj4nEhU=', 'Content-Length': '0'} 2024-03-04T17:36:29.833 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000b503a0acae47bfdf-0065e6069d-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.833 DEBUG:boto:path=/ppfahn-8/ 2024-03-04T17:36:29.833 DEBUG:boto:auth_path=/ppfahn-8/ 2024-03-04T17:36:29.833 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.833 DEBUG:boto:Path: /ppfahn-8/ 2024-03-04T17:36:29.833 DEBUG:boto:Data: 2024-03-04T17:36:29.834 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.834 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:29.834 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.834 DEBUG:boto:Params: {} 2024-03-04T17:36:29.834 DEBUG:boto:Token: None 2024-03-04T17:36:29.834 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-8/ 2024-03-04T17:36:29.834 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V1bWKtAkCvl2TX8UTPAM056K9hM= 2024-03-04T17:36:29.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V1bWKtAkCvl2TX8UTPAM056K9hM=', 'Content-Length': '0'} 2024-03-04T17:36:29.835 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000edde8e63a535d963-0065e6069d-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.836 DEBUG:boto:path=/ppfahn-9/ 2024-03-04T17:36:29.836 DEBUG:boto:auth_path=/ppfahn-9/ 2024-03-04T17:36:29.836 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.836 DEBUG:boto:Path: /ppfahn-9/ 2024-03-04T17:36:29.836 DEBUG:boto:Data: 2024-03-04T17:36:29.836 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.836 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:29.836 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.836 DEBUG:boto:Params: {} 2024-03-04T17:36:29.836 DEBUG:boto:Token: None 2024-03-04T17:36:29.837 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-9/ 2024-03-04T17:36:29.837 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jSvsjk65rB7aOphhm/GeYrU5cjQ= 2024-03-04T17:36:29.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jSvsjk65rB7aOphhm/GeYrU5cjQ=', 'Content-Length': '0'} 2024-03-04T17:36:29.838 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx0000013715d9ea1ad80e8-0065e6069d-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.838 DEBUG:boto:path=/ppfahn-7/ 2024-03-04T17:36:29.838 DEBUG:boto:auth_path=/ppfahn-7/ 2024-03-04T17:36:29.838 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.838 DEBUG:boto:Path: /ppfahn-7/ 2024-03-04T17:36:29.839 DEBUG:boto:Data: 2024-03-04T17:36:29.839 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.839 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:29.839 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.839 DEBUG:boto:Params: {} 2024-03-04T17:36:29.839 DEBUG:boto:Token: None 2024-03-04T17:36:29.839 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-7/ 2024-03-04T17:36:29.839 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EMxbTMh9/tVdZuzAtooZUj4nEhU= 2024-03-04T17:36:29.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EMxbTMh9/tVdZuzAtooZUj4nEhU=', 'Content-Length': '0'} 2024-03-04T17:36:29.842 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx000005dc3af013d1fc1b5-0065e6069d-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.842 DEBUG:boto:path=/ppfahn-8/ 2024-03-04T17:36:29.842 DEBUG:boto:auth_path=/ppfahn-8/ 2024-03-04T17:36:29.842 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.842 DEBUG:boto:Path: /ppfahn-8/ 2024-03-04T17:36:29.843 DEBUG:boto:Data: 2024-03-04T17:36:29.843 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.843 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:29.843 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.843 DEBUG:boto:Params: {} 2024-03-04T17:36:29.843 DEBUG:boto:Token: None 2024-03-04T17:36:29.843 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-8/ 2024-03-04T17:36:29.843 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V1bWKtAkCvl2TX8UTPAM056K9hM= 2024-03-04T17:36:29.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V1bWKtAkCvl2TX8UTPAM056K9hM=', 'Content-Length': '0'} 2024-03-04T17:36:29.845 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000cb5df11cdb8cea79-0065e6069d-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.845 DEBUG:boto:path=/ppfahn-9/ 2024-03-04T17:36:29.845 DEBUG:boto:auth_path=/ppfahn-9/ 2024-03-04T17:36:29.845 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.845 DEBUG:boto:Path: /ppfahn-9/ 2024-03-04T17:36:29.846 DEBUG:boto:Data: 2024-03-04T17:36:29.846 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.846 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:29.846 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.846 DEBUG:boto:Params: {} 2024-03-04T17:36:29.846 DEBUG:boto:Token: None 2024-03-04T17:36:29.846 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-9/ 2024-03-04T17:36:29.846 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jSvsjk65rB7aOphhm/GeYrU5cjQ= 2024-03-04T17:36:29.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jSvsjk65rB7aOphhm/GeYrU5cjQ=', 'Content-Length': '0'} 2024-03-04T17:36:29.848 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000421c5bb19f1fa037-0065e6069d-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.849 DEBUG:boto:path=/ppfahn-7/ 2024-03-04T17:36:29.849 DEBUG:boto:auth_path=/ppfahn-7/ 2024-03-04T17:36:29.849 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.849 DEBUG:boto:Path: /ppfahn-7/ 2024-03-04T17:36:29.849 DEBUG:boto:Data: 2024-03-04T17:36:29.849 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.849 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:29.849 DEBUG:boto:Port: 8001 2024-03-04T17:36:29.850 DEBUG:boto:Params: {} 2024-03-04T17:36:29.850 DEBUG:boto:Token: None 2024-03-04T17:36:29.850 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-7/ 2024-03-04T17:36:29.850 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EMxbTMh9/tVdZuzAtooZUj4nEhU= 2024-03-04T17:36:29.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EMxbTMh9/tVdZuzAtooZUj4nEhU=', 'Content-Length': '0'} 2024-03-04T17:36:29.852 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx000007fffc11edfc27b7f-0065e6069d-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.852 DEBUG:boto:path=/ppfahn-8/ 2024-03-04T17:36:29.852 DEBUG:boto:auth_path=/ppfahn-8/ 2024-03-04T17:36:29.852 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.852 DEBUG:boto:Path: /ppfahn-8/ 2024-03-04T17:36:29.852 DEBUG:boto:Data: 2024-03-04T17:36:29.852 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.852 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:29.852 DEBUG:boto:Port: 8001 2024-03-04T17:36:29.853 DEBUG:boto:Params: {} 2024-03-04T17:36:29.853 DEBUG:boto:Token: None 2024-03-04T17:36:29.853 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-8/ 2024-03-04T17:36:29.853 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V1bWKtAkCvl2TX8UTPAM056K9hM= 2024-03-04T17:36:29.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V1bWKtAkCvl2TX8UTPAM056K9hM=', 'Content-Length': '0'} 2024-03-04T17:36:29.854 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000c771b84d58320a52-0065e6069d-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.855 DEBUG:boto:path=/ppfahn-9/ 2024-03-04T17:36:29.855 DEBUG:boto:auth_path=/ppfahn-9/ 2024-03-04T17:36:29.855 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.855 DEBUG:boto:Path: /ppfahn-9/ 2024-03-04T17:36:29.855 DEBUG:boto:Data: 2024-03-04T17:36:29.855 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.855 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:29.855 DEBUG:boto:Port: 8001 2024-03-04T17:36:29.855 DEBUG:boto:Params: {} 2024-03-04T17:36:29.856 DEBUG:boto:Token: None 2024-03-04T17:36:29.856 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-9/ 2024-03-04T17:36:29.856 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jSvsjk65rB7aOphhm/GeYrU5cjQ= 2024-03-04T17:36:29.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jSvsjk65rB7aOphhm/GeYrU5cjQ=', 'Content-Length': '0'} 2024-03-04T17:36:29.857 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000136ab89c3951c84e-0065e6069d-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2024-03-04T17:36:29.858 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-10 2024-03-04T17:36:29.858 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:29.858 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:29.858 DEBUG:boto:Method: PUT 2024-03-04T17:36:29.858 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:36:29.858 DEBUG:boto:Data: 2024-03-04T17:36:29.858 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.859 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:29.859 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.859 DEBUG:boto:Params: {} 2024-03-04T17:36:29.859 DEBUG:boto:Token: None 2024-03-04T17:36:29.859 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-10/ 2024-03-04T17:36:29.859 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YE3wR+NyOYOgQvxq7Q9agCTtspA= 2024-03-04T17:36:29.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YE3wR+NyOYOgQvxq7Q9agCTtspA=', 'Content-Length': '0'} 2024-03-04T17:36:29.880 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e1c35d16bb1f25b6-0065e6069d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.880 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-11 2024-03-04T17:36:29.880 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:36:29.880 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:36:29.880 DEBUG:boto:Method: PUT 2024-03-04T17:36:29.880 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:36:29.881 DEBUG:boto:Data: 2024-03-04T17:36:29.881 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.881 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:29.881 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.881 DEBUG:boto:Params: {} 2024-03-04T17:36:29.881 DEBUG:boto:Token: None 2024-03-04T17:36:29.881 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-11/ 2024-03-04T17:36:29.881 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8zv7T63L0qPjG/kibS31ddc7oCo= 2024-03-04T17:36:29.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8zv7T63L0qPjG/kibS31ddc7oCo=', 'Content-Length': '0'} 2024-03-04T17:36:29.934 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008589208ac54c9c0c-0065e6069d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.935 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-12 2024-03-04T17:36:29.935 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:36:29.935 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:36:29.936 DEBUG:boto:Method: PUT 2024-03-04T17:36:29.936 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:36:29.936 DEBUG:boto:Data: 2024-03-04T17:36:29.936 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.936 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:29.936 DEBUG:boto:Port: 8001 2024-03-04T17:36:29.937 DEBUG:boto:Params: {} 2024-03-04T17:36:29.937 DEBUG:boto:Token: None 2024-03-04T17:36:29.937 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-12/ 2024-03-04T17:36:29.937 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hgia7Jnk+BxgvTMVlla2+UGbWgw= 2024-03-04T17:36:29.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hgia7Jnk+BxgvTMVlla2+UGbWgw=', 'Content-Length': '0'} 2024-03-04T17:36:29.997 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c554912450babb4d-0065e6069d-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:29.997 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:29.998 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:29.998 DEBUG:boto:Method: HEAD 2024-03-04T17:36:29.998 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:36:29.998 DEBUG:boto:Data: 2024-03-04T17:36:29.998 DEBUG:boto:Headers: {} 2024-03-04T17:36:29.998 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:29.998 DEBUG:boto:Port: 8000 2024-03-04T17:36:29.998 DEBUG:boto:Params: {} 2024-03-04T17:36:29.998 DEBUG:boto:Token: None 2024-03-04T17:36:29.999 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:29 GMT /ppfahn-10/ 2024-03-04T17:36:29.999 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NzldHQK2oL4OdbcClih9e3y+t0I= 2024-03-04T17:36:29.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NzldHQK2oL4OdbcClih9e3y+t0I=', 'Content-Length': '0'} 2024-03-04T17:36:30.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000066491ebcfbdf8c7-0065e6069d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.003 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:36:30.004 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:36:30.004 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.004 DEBUG:boto:Path: /ppfahn-10/myobj 2024-03-04T17:36:30.004 DEBUG:boto:Data: 2024-03-04T17:36:30.004 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.004 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.004 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.004 DEBUG:boto:Params: {} 2024-03-04T17:36:30.004 DEBUG:boto:Token: None 2024-03-04T17:36:30.005 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/myobj 2024-03-04T17:36:30.005 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m4qEBH46s6o39USiLdD4sLka7pM= 2024-03-04T17:36:30.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m4qEBH46s6o39USiLdD4sLka7pM='} 2024-03-04T17:36:30.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070991a9a64e07be6-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.030 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:30.030 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:30.030 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.030 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:36:30.030 DEBUG:boto:Data: 2024-03-04T17:36:30.030 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.030 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.031 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.031 DEBUG:boto:Params: {} 2024-03-04T17:36:30.031 DEBUG:boto:Token: None 2024-03-04T17:36:30.031 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/ 2024-03-04T17:36:30.031 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg= 2024-03-04T17:36:30.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg=', 'Content-Length': '0'} 2024-03-04T17:36:30.035 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c5c7e5b8767c4e0-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.036 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:36:30.036 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:36:30.036 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.036 DEBUG:boto:Path: /ppfahn-10/_myobj 2024-03-04T17:36:30.036 DEBUG:boto:Data: 2024-03-04T17:36:30.036 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.036 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.036 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.036 DEBUG:boto:Params: {} 2024-03-04T17:36:30.037 DEBUG:boto:Token: None 2024-03-04T17:36:30.037 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/_myobj 2024-03-04T17:36:30.037 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZXccwd4GO+/z9q/ZXlwBjEr1K1k= 2024-03-04T17:36:30.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZXccwd4GO+/z9q/ZXlwBjEr1K1k='} 2024-03-04T17:36:30.085 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3351f15bc46a77e-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.085 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:30.085 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:30.085 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.085 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:36:30.085 DEBUG:boto:Data: 2024-03-04T17:36:30.085 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.086 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.086 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.086 DEBUG:boto:Params: {} 2024-03-04T17:36:30.086 DEBUG:boto:Token: None 2024-03-04T17:36:30.086 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/ 2024-03-04T17:36:30.086 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg= 2024-03-04T17:36:30.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg=', 'Content-Length': '0'} 2024-03-04T17:36:30.091 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000075b7d76f1a67298-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.091 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:36:30.091 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:36:30.092 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.092 DEBUG:boto:Path: /ppfahn-10/%3A 2024-03-04T17:36:30.092 DEBUG:boto:Data: 2024-03-04T17:36:30.092 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.092 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.092 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.092 DEBUG:boto:Params: {} 2024-03-04T17:36:30.093 DEBUG:boto:Token: None 2024-03-04T17:36:30.093 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/%3A 2024-03-04T17:36:30.093 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2OYpHWIHrtXM8Di89/doHxGjmL8= 2024-03-04T17:36:30.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2OYpHWIHrtXM8Di89/doHxGjmL8='} 2024-03-04T17:36:30.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ac4a0a8fb0d9b04-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.113 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:30.113 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:30.114 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.114 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:36:30.114 DEBUG:boto:Data: 2024-03-04T17:36:30.114 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.114 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.114 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.114 DEBUG:boto:Params: {} 2024-03-04T17:36:30.114 DEBUG:boto:Token: None 2024-03-04T17:36:30.114 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/ 2024-03-04T17:36:30.115 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg= 2024-03-04T17:36:30.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg=', 'Content-Length': '0'} 2024-03-04T17:36:30.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b0bb1ce6d5fc73c-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.127 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:36:30.127 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:36:30.127 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.127 DEBUG:boto:Path: /ppfahn-10/%26 2024-03-04T17:36:30.127 DEBUG:boto:Data: 2024-03-04T17:36:30.127 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.127 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.127 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.128 DEBUG:boto:Params: {} 2024-03-04T17:36:30.128 DEBUG:boto:Token: None 2024-03-04T17:36:30.128 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/%26 2024-03-04T17:36:30.128 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iXa0BgmebzeR2rfGRsHdHWOMFUE= 2024-03-04T17:36:30.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iXa0BgmebzeR2rfGRsHdHWOMFUE='} 2024-03-04T17:36:30.172 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd82d5c9bb3c0090-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.172 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:30.172 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:30.173 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.173 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:36:30.173 DEBUG:boto:Data: 2024-03-04T17:36:30.173 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.173 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.173 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.173 DEBUG:boto:Params: {} 2024-03-04T17:36:30.173 DEBUG:boto:Token: None 2024-03-04T17:36:30.173 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/ 2024-03-04T17:36:30.173 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg= 2024-03-04T17:36:30.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg=', 'Content-Length': '0'} 2024-03-04T17:36:30.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a25ef65d878b18b-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.178 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:36:30.178 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:36:30.178 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.178 DEBUG:boto:Path: /ppfahn-10/. 2024-03-04T17:36:30.178 DEBUG:boto:Data: 2024-03-04T17:36:30.178 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.178 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.179 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.179 DEBUG:boto:Params: {} 2024-03-04T17:36:30.179 DEBUG:boto:Token: None 2024-03-04T17:36:30.179 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/. 2024-03-04T17:36:30.179 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:19yDa4zRHSNAolGV1VYpvXXy0cQ= 2024-03-04T17:36:30.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:19yDa4zRHSNAolGV1VYpvXXy0cQ='} 2024-03-04T17:36:30.197 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057e18634cedf4b0f-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.198 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:30.198 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:30.198 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.198 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:36:30.198 DEBUG:boto:Data: 2024-03-04T17:36:30.198 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.198 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.198 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.198 DEBUG:boto:Params: {} 2024-03-04T17:36:30.199 DEBUG:boto:Token: None 2024-03-04T17:36:30.199 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/ 2024-03-04T17:36:30.199 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg= 2024-03-04T17:36:30.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg=', 'Content-Length': '0'} 2024-03-04T17:36:30.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb9a87706dfd195a-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.203 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:36:30.203 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:36:30.203 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.204 DEBUG:boto:Path: /ppfahn-10/.. 2024-03-04T17:36:30.204 DEBUG:boto:Data: 2024-03-04T17:36:30.204 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.204 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.204 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.204 DEBUG:boto:Params: {} 2024-03-04T17:36:30.204 DEBUG:boto:Token: None 2024-03-04T17:36:30.205 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/.. 2024-03-04T17:36:30.205 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HGIpz7Erpcewy0uf1HNTAdGk60o= 2024-03-04T17:36:30.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HGIpz7Erpcewy0uf1HNTAdGk60o='} 2024-03-04T17:36:30.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002748476cd771d673-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.248 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:30.248 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:30.249 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.249 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:36:30.249 DEBUG:boto:Data: 2024-03-04T17:36:30.249 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.249 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.249 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.249 DEBUG:boto:Params: {} 2024-03-04T17:36:30.249 DEBUG:boto:Token: None 2024-03-04T17:36:30.249 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/ 2024-03-04T17:36:30.249 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg= 2024-03-04T17:36:30.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg=', 'Content-Length': '0'} 2024-03-04T17:36:30.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf57e8d94b936cd4-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.254 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:36:30.254 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:36:30.254 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.255 DEBUG:boto:Path: /ppfahn-10/... 2024-03-04T17:36:30.255 DEBUG:boto:Data: 2024-03-04T17:36:30.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.255 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.255 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.255 DEBUG:boto:Params: {} 2024-03-04T17:36:30.255 DEBUG:boto:Token: None 2024-03-04T17:36:30.255 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/... 2024-03-04T17:36:30.255 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f6EXl8UoQLq8DOIue3SlMJPt/1I= 2024-03-04T17:36:30.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f6EXl8UoQLq8DOIue3SlMJPt/1I='} 2024-03-04T17:36:30.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa10468a7ccfff2a-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.264 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:30.264 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:30.264 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.264 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:36:30.264 DEBUG:boto:Data: 2024-03-04T17:36:30.264 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.264 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.265 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.265 DEBUG:boto:Params: {} 2024-03-04T17:36:30.265 DEBUG:boto:Token: None 2024-03-04T17:36:30.265 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/ 2024-03-04T17:36:30.265 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg= 2024-03-04T17:36:30.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg=', 'Content-Length': '0'} 2024-03-04T17:36:30.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7e31e9960533546-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.269 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:36:30.269 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:36:30.269 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.269 DEBUG:boto:Path: /ppfahn-10/.o 2024-03-04T17:36:30.270 DEBUG:boto:Data: 2024-03-04T17:36:30.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.270 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.270 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.270 DEBUG:boto:Params: {} 2024-03-04T17:36:30.270 DEBUG:boto:Token: None 2024-03-04T17:36:30.270 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/.o 2024-03-04T17:36:30.270 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uZtLvwi1aNfP6XGQ0gvHXUg6RL8= 2024-03-04T17:36:30.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uZtLvwi1aNfP6XGQ0gvHXUg6RL8='} 2024-03-04T17:36:30.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005247e161111d2408-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.312 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:30.312 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:30.313 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.313 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:36:30.313 DEBUG:boto:Data: 2024-03-04T17:36:30.313 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.313 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.313 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.313 DEBUG:boto:Params: {} 2024-03-04T17:36:30.313 DEBUG:boto:Token: None 2024-03-04T17:36:30.313 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/ 2024-03-04T17:36:30.313 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg= 2024-03-04T17:36:30.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FxHcRHJL7J14v+4RhlhDSMssjeg=', 'Content-Length': '0'} 2024-03-04T17:36:30.317 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d528a40b69da6b4-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.318 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:36:30.318 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:36:30.318 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.318 DEBUG:boto:Path: /ppfahn-10/.o. 2024-03-04T17:36:30.318 DEBUG:boto:Data: 2024-03-04T17:36:30.318 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.318 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:30.318 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.318 DEBUG:boto:Params: {} 2024-03-04T17:36:30.319 DEBUG:boto:Token: None 2024-03-04T17:36:30.319 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-10/.o. 2024-03-04T17:36:30.319 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5rE+PC2sBUgTYYIBALA7epikoEE= 2024-03-04T17:36:30.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5rE+PC2sBUgTYYIBALA7epikoEE='} 2024-03-04T17:36:30.331 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001f1d209ce61d6e3-0065e6069e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.332 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:36:30.332 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:36:30.332 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.332 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:36:30.332 DEBUG:boto:Data: 2024-03-04T17:36:30.332 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.332 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.332 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.332 DEBUG:boto:Params: {} 2024-03-04T17:36:30.333 DEBUG:boto:Token: None 2024-03-04T17:36:30.333 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/ 2024-03-04T17:36:30.333 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o= 2024-03-04T17:36:30.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o=', 'Content-Length': '0'} 2024-03-04T17:36:30.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8d8742bb207ea1b-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.337 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:36:30.338 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:36:30.338 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.338 DEBUG:boto:Path: /ppfahn-11/myobj 2024-03-04T17:36:30.338 DEBUG:boto:Data: 2024-03-04T17:36:30.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.338 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.338 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.338 DEBUG:boto:Params: {} 2024-03-04T17:36:30.338 DEBUG:boto:Token: None 2024-03-04T17:36:30.339 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/myobj 2024-03-04T17:36:30.339 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JMGqiDZ7q2KAyRhaTxTJoSySGn4= 2024-03-04T17:36:30.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JMGqiDZ7q2KAyRhaTxTJoSySGn4='} 2024-03-04T17:36:30.361 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057b887d7c8dc2d03-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.361 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:36:30.361 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:36:30.361 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.362 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:36:30.362 DEBUG:boto:Data: 2024-03-04T17:36:30.362 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.362 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.362 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.362 DEBUG:boto:Params: {} 2024-03-04T17:36:30.362 DEBUG:boto:Token: None 2024-03-04T17:36:30.362 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/ 2024-03-04T17:36:30.362 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o= 2024-03-04T17:36:30.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o=', 'Content-Length': '0'} 2024-03-04T17:36:30.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8f66ccf90ed73fc-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.367 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:36:30.367 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:36:30.367 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.367 DEBUG:boto:Path: /ppfahn-11/_myobj 2024-03-04T17:36:30.368 DEBUG:boto:Data: 2024-03-04T17:36:30.368 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.368 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.368 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.368 DEBUG:boto:Params: {} 2024-03-04T17:36:30.368 DEBUG:boto:Token: None 2024-03-04T17:36:30.368 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/_myobj 2024-03-04T17:36:30.368 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:B4S9k0dd2qr4WW5nzvgdQq9jcAE= 2024-03-04T17:36:30.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:B4S9k0dd2qr4WW5nzvgdQq9jcAE='} 2024-03-04T17:36:30.412 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c3311a3a8cc1f1c-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.412 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:36:30.413 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:36:30.413 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.413 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:36:30.413 DEBUG:boto:Data: 2024-03-04T17:36:30.413 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.413 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.413 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.413 DEBUG:boto:Params: {} 2024-03-04T17:36:30.413 DEBUG:boto:Token: None 2024-03-04T17:36:30.414 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/ 2024-03-04T17:36:30.414 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o= 2024-03-04T17:36:30.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o=', 'Content-Length': '0'} 2024-03-04T17:36:30.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000717b48de63eea876-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.418 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:36:30.418 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:36:30.418 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.419 DEBUG:boto:Path: /ppfahn-11/%3A 2024-03-04T17:36:30.419 DEBUG:boto:Data: 2024-03-04T17:36:30.419 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.419 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.419 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.419 DEBUG:boto:Params: {} 2024-03-04T17:36:30.419 DEBUG:boto:Token: None 2024-03-04T17:36:30.419 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/%3A 2024-03-04T17:36:30.419 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UzewTdO4Nvo/PZOQLzpO6vRjB+I= 2024-03-04T17:36:30.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UzewTdO4Nvo/PZOQLzpO6vRjB+I='} 2024-03-04T17:36:30.440 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4b99af7b7f780d9-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.440 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:36:30.440 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:36:30.440 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.441 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:36:30.441 DEBUG:boto:Data: 2024-03-04T17:36:30.441 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.441 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.441 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.441 DEBUG:boto:Params: {} 2024-03-04T17:36:30.441 DEBUG:boto:Token: None 2024-03-04T17:36:30.441 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/ 2024-03-04T17:36:30.441 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o= 2024-03-04T17:36:30.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o=', 'Content-Length': '0'} 2024-03-04T17:36:30.445 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aab1f756e905c06d-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.445 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:36:30.446 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:36:30.446 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.446 DEBUG:boto:Path: /ppfahn-11/%26 2024-03-04T17:36:30.446 DEBUG:boto:Data: 2024-03-04T17:36:30.446 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.446 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.446 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.446 DEBUG:boto:Params: {} 2024-03-04T17:36:30.446 DEBUG:boto:Token: None 2024-03-04T17:36:30.446 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/%26 2024-03-04T17:36:30.446 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rgLPH50J+7xRdqJGdSzFlpIBZMs= 2024-03-04T17:36:30.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rgLPH50J+7xRdqJGdSzFlpIBZMs='} 2024-03-04T17:36:30.492 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddf4af0ede52bb7e-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.493 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:36:30.493 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:36:30.493 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.493 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:36:30.493 DEBUG:boto:Data: 2024-03-04T17:36:30.493 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.493 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.494 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.494 DEBUG:boto:Params: {} 2024-03-04T17:36:30.494 DEBUG:boto:Token: None 2024-03-04T17:36:30.494 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/ 2024-03-04T17:36:30.494 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o= 2024-03-04T17:36:30.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o=', 'Content-Length': '0'} 2024-03-04T17:36:30.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c72e9a9dec9ae878-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.499 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:36:30.499 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:36:30.499 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.499 DEBUG:boto:Path: /ppfahn-11/. 2024-03-04T17:36:30.499 DEBUG:boto:Data: 2024-03-04T17:36:30.499 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.499 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.500 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.500 DEBUG:boto:Params: {} 2024-03-04T17:36:30.500 DEBUG:boto:Token: None 2024-03-04T17:36:30.500 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/. 2024-03-04T17:36:30.500 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xQYj/GF01F2prs6uGG4DrtWMQdw= 2024-03-04T17:36:30.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xQYj/GF01F2prs6uGG4DrtWMQdw='} 2024-03-04T17:36:30.520 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1b9361266388075-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.520 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:36:30.520 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:36:30.520 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.520 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:36:30.520 DEBUG:boto:Data: 2024-03-04T17:36:30.520 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.520 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.520 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.521 DEBUG:boto:Params: {} 2024-03-04T17:36:30.521 DEBUG:boto:Token: None 2024-03-04T17:36:30.521 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/ 2024-03-04T17:36:30.521 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o= 2024-03-04T17:36:30.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o=', 'Content-Length': '0'} 2024-03-04T17:36:30.525 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ff204aff45bbce1-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.526 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:36:30.526 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:36:30.526 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.526 DEBUG:boto:Path: /ppfahn-11/.. 2024-03-04T17:36:30.526 DEBUG:boto:Data: 2024-03-04T17:36:30.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.526 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.526 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.526 DEBUG:boto:Params: {} 2024-03-04T17:36:30.526 DEBUG:boto:Token: None 2024-03-04T17:36:30.527 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/.. 2024-03-04T17:36:30.527 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jSebZ5zfP/phSIDicA2M2BJeFls= 2024-03-04T17:36:30.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jSebZ5zfP/phSIDicA2M2BJeFls='} 2024-03-04T17:36:30.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000520f015c660a1501-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.576 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:36:30.576 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:36:30.577 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.577 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:36:30.577 DEBUG:boto:Data: 2024-03-04T17:36:30.577 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.577 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.577 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.577 DEBUG:boto:Params: {} 2024-03-04T17:36:30.577 DEBUG:boto:Token: None 2024-03-04T17:36:30.577 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/ 2024-03-04T17:36:30.577 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o= 2024-03-04T17:36:30.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o=', 'Content-Length': '0'} 2024-03-04T17:36:30.581 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009afe0622ca9419a4-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.581 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:36:30.582 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:36:30.582 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.582 DEBUG:boto:Path: /ppfahn-11/... 2024-03-04T17:36:30.582 DEBUG:boto:Data: 2024-03-04T17:36:30.582 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.582 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.582 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.582 DEBUG:boto:Params: {} 2024-03-04T17:36:30.582 DEBUG:boto:Token: None 2024-03-04T17:36:30.582 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/... 2024-03-04T17:36:30.583 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9s51dUXzopqpVqfxRhLWmCuTlEI= 2024-03-04T17:36:30.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9s51dUXzopqpVqfxRhLWmCuTlEI='} 2024-03-04T17:36:30.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae59262e04515352-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.591 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:36:30.591 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:36:30.591 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.591 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:36:30.591 DEBUG:boto:Data: 2024-03-04T17:36:30.592 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.592 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.592 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.592 DEBUG:boto:Params: {} 2024-03-04T17:36:30.592 DEBUG:boto:Token: None 2024-03-04T17:36:30.592 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/ 2024-03-04T17:36:30.592 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o= 2024-03-04T17:36:30.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o=', 'Content-Length': '0'} 2024-03-04T17:36:30.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a5dcba79e5539da-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.597 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:36:30.597 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:36:30.597 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.597 DEBUG:boto:Path: /ppfahn-11/.o 2024-03-04T17:36:30.597 DEBUG:boto:Data: 2024-03-04T17:36:30.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.597 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.597 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.597 DEBUG:boto:Params: {} 2024-03-04T17:36:30.597 DEBUG:boto:Token: None 2024-03-04T17:36:30.598 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/.o 2024-03-04T17:36:30.598 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QWGn+6X/497aW9c/x8ih+lx7KMw= 2024-03-04T17:36:30.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QWGn+6X/497aW9c/x8ih+lx7KMw='} 2024-03-04T17:36:30.640 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000078600047ebe6b087-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.640 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:36:30.641 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:36:30.641 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.641 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:36:30.641 DEBUG:boto:Data: 2024-03-04T17:36:30.641 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.641 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.641 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.641 DEBUG:boto:Params: {} 2024-03-04T17:36:30.641 DEBUG:boto:Token: None 2024-03-04T17:36:30.642 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/ 2024-03-04T17:36:30.642 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o= 2024-03-04T17:36:30.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6fK/zJJUkpDJGc/5N2pfobrAd0o=', 'Content-Length': '0'} 2024-03-04T17:36:30.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5e019b91ff3d1cf-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.646 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:36:30.646 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:36:30.646 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.646 DEBUG:boto:Path: /ppfahn-11/.o. 2024-03-04T17:36:30.646 DEBUG:boto:Data: 2024-03-04T17:36:30.646 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.647 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:30.647 DEBUG:boto:Port: 8000 2024-03-04T17:36:30.647 DEBUG:boto:Params: {} 2024-03-04T17:36:30.647 DEBUG:boto:Token: None 2024-03-04T17:36:30.647 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-11/.o. 2024-03-04T17:36:30.647 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DK70P4TZTdbPQCnRzzs0YWy+qnw= 2024-03-04T17:36:30.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DK70P4TZTdbPQCnRzzs0YWy+qnw='} 2024-03-04T17:36:30.659 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c410da314c90ace-0065e6069e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.659 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:36:30.659 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:36:30.660 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.660 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:36:30.660 DEBUG:boto:Data: 2024-03-04T17:36:30.660 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.660 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.660 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.660 DEBUG:boto:Params: {} 2024-03-04T17:36:30.660 DEBUG:boto:Token: None 2024-03-04T17:36:30.660 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/ 2024-03-04T17:36:30.661 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw= 2024-03-04T17:36:30.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw=', 'Content-Length': '0'} 2024-03-04T17:36:30.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebcfa47b6b348237-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.665 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:36:30.665 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:36:30.665 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.665 DEBUG:boto:Path: /ppfahn-12/myobj 2024-03-04T17:36:30.665 DEBUG:boto:Data: 2024-03-04T17:36:30.665 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.665 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.665 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.666 DEBUG:boto:Params: {} 2024-03-04T17:36:30.666 DEBUG:boto:Token: None 2024-03-04T17:36:30.666 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/myobj 2024-03-04T17:36:30.666 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TVFVNQmoCN0GKqtUiC5Z+Bjg7d4= 2024-03-04T17:36:30.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TVFVNQmoCN0GKqtUiC5Z+Bjg7d4='} 2024-03-04T17:36:30.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c98d44c08d07c05f-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.690 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:36:30.690 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:36:30.690 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.690 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:36:30.690 DEBUG:boto:Data: 2024-03-04T17:36:30.690 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.690 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.690 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.691 DEBUG:boto:Params: {} 2024-03-04T17:36:30.691 DEBUG:boto:Token: None 2024-03-04T17:36:30.691 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/ 2024-03-04T17:36:30.691 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw= 2024-03-04T17:36:30.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw=', 'Content-Length': '0'} 2024-03-04T17:36:30.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1360c5141f0bb8e-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.696 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:36:30.696 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:36:30.696 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.696 DEBUG:boto:Path: /ppfahn-12/_myobj 2024-03-04T17:36:30.696 DEBUG:boto:Data: 2024-03-04T17:36:30.696 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.696 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.696 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.697 DEBUG:boto:Params: {} 2024-03-04T17:36:30.697 DEBUG:boto:Token: None 2024-03-04T17:36:30.697 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/_myobj 2024-03-04T17:36:30.697 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fiL3T2g4yHBHSBNgb5hmcDL/UZA= 2024-03-04T17:36:30.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fiL3T2g4yHBHSBNgb5hmcDL/UZA='} 2024-03-04T17:36:30.716 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000716e954ee060526e-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.716 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:36:30.716 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:36:30.717 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.717 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:36:30.717 DEBUG:boto:Data: 2024-03-04T17:36:30.717 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.719 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.719 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.719 DEBUG:boto:Params: {} 2024-03-04T17:36:30.720 DEBUG:boto:Token: None 2024-03-04T17:36:30.720 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/ 2024-03-04T17:36:30.720 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw= 2024-03-04T17:36:30.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw=', 'Content-Length': '0'} 2024-03-04T17:36:30.724 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a3b4bcd4cf02996-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.724 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:36:30.724 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:36:30.724 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.724 DEBUG:boto:Path: /ppfahn-12/%3A 2024-03-04T17:36:30.724 DEBUG:boto:Data: 2024-03-04T17:36:30.724 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.725 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.725 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.725 DEBUG:boto:Params: {} 2024-03-04T17:36:30.725 DEBUG:boto:Token: None 2024-03-04T17:36:30.725 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/%3A 2024-03-04T17:36:30.725 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+/5cQroB+LJbHdUF+An+SfhBueY= 2024-03-04T17:36:30.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+/5cQroB+LJbHdUF+An+SfhBueY='} 2024-03-04T17:36:30.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c5611993be4082e-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.772 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:36:30.772 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:36:30.773 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.773 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:36:30.773 DEBUG:boto:Data: 2024-03-04T17:36:30.773 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.773 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.773 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.773 DEBUG:boto:Params: {} 2024-03-04T17:36:30.773 DEBUG:boto:Token: None 2024-03-04T17:36:30.773 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/ 2024-03-04T17:36:30.773 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw= 2024-03-04T17:36:30.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw=', 'Content-Length': '0'} 2024-03-04T17:36:30.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000269621f5fac75dfc-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.778 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:36:30.778 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:36:30.778 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.778 DEBUG:boto:Path: /ppfahn-12/%26 2024-03-04T17:36:30.778 DEBUG:boto:Data: 2024-03-04T17:36:30.778 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.778 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.778 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.778 DEBUG:boto:Params: {} 2024-03-04T17:36:30.779 DEBUG:boto:Token: None 2024-03-04T17:36:30.779 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/%26 2024-03-04T17:36:30.779 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QrMUb2+PSEicwKAIMJ8sQASspyI= 2024-03-04T17:36:30.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QrMUb2+PSEicwKAIMJ8sQASspyI='} 2024-03-04T17:36:30.791 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a62949c1a8b00e0f-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.791 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:36:30.792 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:36:30.792 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.792 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:36:30.792 DEBUG:boto:Data: 2024-03-04T17:36:30.792 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.792 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.792 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.792 DEBUG:boto:Params: {} 2024-03-04T17:36:30.792 DEBUG:boto:Token: None 2024-03-04T17:36:30.792 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/ 2024-03-04T17:36:30.792 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw= 2024-03-04T17:36:30.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw=', 'Content-Length': '0'} 2024-03-04T17:36:30.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000459bfbc870efaf74-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.797 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:36:30.797 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:36:30.797 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.797 DEBUG:boto:Path: /ppfahn-12/. 2024-03-04T17:36:30.797 DEBUG:boto:Data: 2024-03-04T17:36:30.797 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.797 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.797 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.797 DEBUG:boto:Params: {} 2024-03-04T17:36:30.797 DEBUG:boto:Token: None 2024-03-04T17:36:30.798 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/. 2024-03-04T17:36:30.798 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G9OQzZVNhRK/cOm3aIVL3mPVci4= 2024-03-04T17:36:30.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G9OQzZVNhRK/cOm3aIVL3mPVci4='} 2024-03-04T17:36:30.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026a052dc34bf3282-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.840 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:36:30.840 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:36:30.841 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.841 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:36:30.841 DEBUG:boto:Data: 2024-03-04T17:36:30.841 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.841 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.841 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.841 DEBUG:boto:Params: {} 2024-03-04T17:36:30.841 DEBUG:boto:Token: None 2024-03-04T17:36:30.841 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/ 2024-03-04T17:36:30.841 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw= 2024-03-04T17:36:30.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw=', 'Content-Length': '0'} 2024-03-04T17:36:30.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a969bf5015939ce-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.846 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:36:30.846 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:36:30.846 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.846 DEBUG:boto:Path: /ppfahn-12/.. 2024-03-04T17:36:30.846 DEBUG:boto:Data: 2024-03-04T17:36:30.846 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.846 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.846 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.846 DEBUG:boto:Params: {} 2024-03-04T17:36:30.847 DEBUG:boto:Token: None 2024-03-04T17:36:30.847 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/.. 2024-03-04T17:36:30.847 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TKFgxSdIM+c7TeiqRk9KYpMEgCk= 2024-03-04T17:36:30.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TKFgxSdIM+c7TeiqRk9KYpMEgCk='} 2024-03-04T17:36:30.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034b78fd72bab5284-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.860 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:36:30.860 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:36:30.860 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.860 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:36:30.860 DEBUG:boto:Data: 2024-03-04T17:36:30.861 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.861 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.861 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.861 DEBUG:boto:Params: {} 2024-03-04T17:36:30.861 DEBUG:boto:Token: None 2024-03-04T17:36:30.861 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/ 2024-03-04T17:36:30.861 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw= 2024-03-04T17:36:30.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw=', 'Content-Length': '0'} 2024-03-04T17:36:30.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000407fc94e08f4634d-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.866 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:36:30.866 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:36:30.866 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.866 DEBUG:boto:Path: /ppfahn-12/... 2024-03-04T17:36:30.866 DEBUG:boto:Data: 2024-03-04T17:36:30.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.866 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.866 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.866 DEBUG:boto:Params: {} 2024-03-04T17:36:30.867 DEBUG:boto:Token: None 2024-03-04T17:36:30.867 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/... 2024-03-04T17:36:30.867 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jHySBppR9TGa3CvVyMKwhCsCpdg= 2024-03-04T17:36:30.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jHySBppR9TGa3CvVyMKwhCsCpdg='} 2024-03-04T17:36:30.916 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000156f6e85b7c4dc5f-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.917 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:36:30.917 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:36:30.917 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.917 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:36:30.917 DEBUG:boto:Data: 2024-03-04T17:36:30.917 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.917 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.917 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.917 DEBUG:boto:Params: {} 2024-03-04T17:36:30.918 DEBUG:boto:Token: None 2024-03-04T17:36:30.918 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/ 2024-03-04T17:36:30.918 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw= 2024-03-04T17:36:30.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw=', 'Content-Length': '0'} 2024-03-04T17:36:30.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f854661cf8aa2c3d-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.922 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:36:30.922 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:36:30.922 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.922 DEBUG:boto:Path: /ppfahn-12/.o 2024-03-04T17:36:30.922 DEBUG:boto:Data: 2024-03-04T17:36:30.922 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.922 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.922 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.923 DEBUG:boto:Params: {} 2024-03-04T17:36:30.923 DEBUG:boto:Token: None 2024-03-04T17:36:30.923 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/.o 2024-03-04T17:36:30.923 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yTZQPm/CivBdny4P6/JBZWiIOaE= 2024-03-04T17:36:30.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yTZQPm/CivBdny4P6/JBZWiIOaE='} 2024-03-04T17:36:30.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfbc74511c02ca0b-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.931 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:36:30.932 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:36:30.932 DEBUG:boto:Method: HEAD 2024-03-04T17:36:30.932 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:36:30.932 DEBUG:boto:Data: 2024-03-04T17:36:30.932 DEBUG:boto:Headers: {} 2024-03-04T17:36:30.932 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.932 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.932 DEBUG:boto:Params: {} 2024-03-04T17:36:30.932 DEBUG:boto:Token: None 2024-03-04T17:36:30.932 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/ 2024-03-04T17:36:30.933 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw= 2024-03-04T17:36:30.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CpnR+rsop8NuKhuX8OU3apw1mIw=', 'Content-Length': '0'} 2024-03-04T17:36:30.937 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000cc24baf9ce4fdf7-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.937 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:36:30.937 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:36:30.937 DEBUG:boto:Method: PUT 2024-03-04T17:36:30.937 DEBUG:boto:Path: /ppfahn-12/.o. 2024-03-04T17:36:30.937 DEBUG:boto:Data: 2024-03-04T17:36:30.937 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:36:30.937 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:36:30.937 DEBUG:boto:Port: 8001 2024-03-04T17:36:30.937 DEBUG:boto:Params: {} 2024-03-04T17:36:30.938 DEBUG:boto:Token: None 2024-03-04T17:36:30.938 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:36:30 GMT /ppfahn-12/.o. 2024-03-04T17:36:30.938 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/Co2xnZFR0qgqMOmqx/Z/6by64k= 2024-03-04T17:36:30.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:36:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/Co2xnZFR0qgqMOmqx/Z/6by64k='} 2024-03-04T17:36:30.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a4963fa07602077-0065e6069e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:30.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:30.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:30.985 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:31.025 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:31.026 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:31.080 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:36:31.104 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709573789.970213_38.1","last_update":"2024-03-04T17:36:29.970213Z"}] 2024-03-04T17:36:31.105 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573789.970213_38.1'} 2024-03-04T17:36:31.105 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:36:31.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:31.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:31.105 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:31.149 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:31.149 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:31.201+0000 7fcea8e63980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:31.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:31.205+0000 7fcea8e63980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:31.230 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573789.961499_36.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.961499Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:31Z"} 2024-03-04T17:36:31.230 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573789.961499_36.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.961499Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:31Z"} 2024-03-04T17:36:31.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709573789.961499_36.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.961499Z', 'realm_epoch': 2}}] 2024-03-04T17:36:31.230 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573789.970213_38.1'} 2024-03-04T17:36:31.231 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573789.961499_36.1'} 2024-03-04T17:36:31.231 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709573789.970213_38.1 target=1_1709573789.961499_36.1 2024-03-04T17:36:36.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:36.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:36.233 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:36.275 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:36.275 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:36.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:36.325+0000 7f7893238980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:36.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:36.325+0000 7f7893238980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:36.353 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573789.961499_36.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.961499Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:36Z"} 2024-03-04T17:36:36.353 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573789.961499_36.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.961499Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:36Z"} 2024-03-04T17:36:36.353 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709573789.961499_36.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.961499Z', 'realm_epoch': 2}}] 2024-03-04T17:36:36.354 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573789.970213_38.1'} 2024-03-04T17:36:36.354 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573789.961499_36.1'} 2024-03-04T17:36:36.354 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709573789.970213_38.1 target=1_1709573789.961499_36.1 2024-03-04T17:36:40.658 INFO:tasks.rgw.c2.client.0.smithi153.stdout:2024-03-04T17:36:40.653+0000 7fa53962a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:36:41.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:41.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:41.355 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:41.399 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:41.399 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:41.449+0000 7fa1acb13980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:41.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:41.449+0000 7fa1acb13980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:41.478 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573789.970213_38.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.970213Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:41Z"} 2024-03-04T17:36:41.478 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573789.970213_38.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.970213Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:41Z"} 2024-03-04T17:36:41.478 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709573789.970213_38.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.970213Z', 'realm_epoch': 2}}] 2024-03-04T17:36:41.478 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573789.970213_38.1'} 2024-03-04T17:36:41.478 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573789.970213_38.1'} 2024-03-04T17:36:41.479 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:36:41.479 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:36:41.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:41.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:41.479 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:41.523 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:41.523 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:41.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:41.574+0000 7f4775a65980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:41.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:41.574+0000 7f4775a65980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:41.645 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573789.961499_36.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.961499Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:41Z"} 2024-03-04T17:36:41.645 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573789.961499_36.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.961499Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:41Z"} 2024-03-04T17:36:41.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709573789.961499_36.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.961499Z', 'realm_epoch': 2}}] 2024-03-04T17:36:41.645 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573789.970213_38.1'} 2024-03-04T17:36:41.645 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573789.961499_36.1'} 2024-03-04T17:36:41.645 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1709573789.970213_38.1 target=1_1709573789.961499_36.1 2024-03-04T17:36:46.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:46.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:46.647 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:46.689 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:46.689 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:46.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:46.742+0000 7f4a66141980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:46.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:46.742+0000 7f4a66141980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:46.769 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573789.970213_38.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.970213Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:46Z"} 2024-03-04T17:36:46.770 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573789.970213_38.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.970213Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:46Z"} 2024-03-04T17:36:46.770 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709573789.970213_38.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.970213Z', 'realm_epoch': 2}}] 2024-03-04T17:36:46.770 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573789.970213_38.1'} 2024-03-04T17:36:46.770 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573789.970213_38.1'} 2024-03-04T17:36:46.770 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:36:46.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:36:46.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:36:46.771 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:36:46.822 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:46.822 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:46.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:46.881+0000 7f8dd6ce2980 1 bucket sync caught up with source: 2024-03-04T17:36:46.884 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000002.10.6, , 00000000001.17.6, , 00000000001.14.6, 00000000002.13.6, 00000000001.21.6, 00000000001.16.6, 00000000001.9.6, ] 2024-03-04T17:36:46.884 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000002.10.6, , 00000000001.17.6, , 00000000001.14.6, 00000000002.13.6, 00000000001.21.6, 00000000001.16.6, 00000000001.9.6, ] 2024-03-04T17:36:46.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:36:46.881+0000 7f8dd6ce2980 0 bucket checkpoint complete 2024-03-04T17:36:46.906 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-10 zones={test-zone2, test-zone1} 2024-03-04T17:36:46.906 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:46.906 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:46.906 DEBUG:boto:Method: HEAD 2024-03-04T17:36:46.906 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:36:46.906 DEBUG:boto:Data: 2024-03-04T17:36:46.906 DEBUG:boto:Headers: {} 2024-03-04T17:36:46.907 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:46.907 DEBUG:boto:Port: 8000 2024-03-04T17:36:46.907 DEBUG:boto:Params: {} 2024-03-04T17:36:46.907 DEBUG:boto:Token: None 2024-03-04T17:36:46.907 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:46 GMT /ppfahn-10/ 2024-03-04T17:36:46.907 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qW5tY97Mm/QTHNOXZU05LjWObZM= 2024-03-04T17:36:46.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qW5tY97Mm/QTHNOXZU05LjWObZM=', 'Content-Length': '0'} 2024-03-04T17:36:46.910 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe09270e1da58c67-0065e606ae-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:46.910 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:46.910 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:46.910 DEBUG:boto:Method: HEAD 2024-03-04T17:36:46.910 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:36:46.910 DEBUG:boto:Data: 2024-03-04T17:36:46.910 DEBUG:boto:Headers: {} 2024-03-04T17:36:46.910 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:46.910 DEBUG:boto:Port: 8000 2024-03-04T17:36:46.911 DEBUG:boto:Params: {} 2024-03-04T17:36:46.911 DEBUG:boto:Token: None 2024-03-04T17:36:46.911 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:46 GMT /ppfahn-10/ 2024-03-04T17:36:46.911 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qW5tY97Mm/QTHNOXZU05LjWObZM= 2024-03-04T17:36:46.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qW5tY97Mm/QTHNOXZU05LjWObZM=', 'Content-Length': '0'} 2024-03-04T17:36:46.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000767c623be63ba01-0065e606ae-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:46.915 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:36:46.915 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:46.915 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:46.915 DEBUG:boto:path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:36:46.915 DEBUG:boto:auth_path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:36:46.916 DEBUG:boto:Method: GET 2024-03-04T17:36:46.916 DEBUG:boto:Path: /ppfahn-10/?versions&max-keys=999 2024-03-04T17:36:46.916 DEBUG:boto:Data: 2024-03-04T17:36:46.916 DEBUG:boto:Headers: {} 2024-03-04T17:36:46.916 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:46.916 DEBUG:boto:Port: 8000 2024-03-04T17:36:46.916 DEBUG:boto:Params: {} 2024-03-04T17:36:46.916 DEBUG:boto:Token: None 2024-03-04T17:36:46.916 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:46 GMT /ppfahn-10/?versions 2024-03-04T17:36:46.916 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wlL2FCRPUPTQR5IrfHXXDC/vUhI= 2024-03-04T17:36:46.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wlL2FCRPUPTQR5IrfHXXDC/vUhI=', 'Content-Length': '0'} 2024-03-04T17:36:46.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000785f9034b26c9209-0065e606ae-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:46.919 DEBUG:boto:b'ppfahn-10999false&nulltrue2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:46.920 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T17:36:46.920 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T17:36:46.921 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T17:36:46.921 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T17:36:46.921 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T17:36:46.921 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T17:36:46.921 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T17:36:46.921 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T17:36:46.921 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:36:46.921 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:36:46.921 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:46.921 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:46.921 DEBUG:boto:path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:36:46.922 DEBUG:boto:auth_path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:36:46.922 DEBUG:boto:Method: GET 2024-03-04T17:36:46.922 DEBUG:boto:Path: /ppfahn-10/?versions&max-keys=999 2024-03-04T17:36:46.922 DEBUG:boto:Data: 2024-03-04T17:36:46.922 DEBUG:boto:Headers: {} 2024-03-04T17:36:46.922 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:46.922 DEBUG:boto:Port: 8000 2024-03-04T17:36:46.922 DEBUG:boto:Params: {} 2024-03-04T17:36:46.922 DEBUG:boto:Token: None 2024-03-04T17:36:46.922 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:46 GMT /ppfahn-10/?versions 2024-03-04T17:36:46.922 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wlL2FCRPUPTQR5IrfHXXDC/vUhI= 2024-03-04T17:36:46.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wlL2FCRPUPTQR5IrfHXXDC/vUhI=', 'Content-Length': '0'} 2024-03-04T17:36:46.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025a9193beacba945-0065e606ae-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:46.927 DEBUG:boto:b'ppfahn-10999false&nulltrue2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:46.928 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T17:36:46.928 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T17:36:46.928 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T17:36:46.928 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T17:36:46.929 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T17:36:46.929 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T17:36:46.929 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T17:36:46.929 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T17:36:46.929 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:36:46.929 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:46.929 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:46.929 DEBUG:boto:path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:36:46.929 DEBUG:boto:auth_path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:36:46.929 DEBUG:boto:Method: GET 2024-03-04T17:36:46.929 DEBUG:boto:Path: /ppfahn-10/?versions&max-keys=999 2024-03-04T17:36:46.930 DEBUG:boto:Data: 2024-03-04T17:36:46.930 DEBUG:boto:Headers: {} 2024-03-04T17:36:46.930 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:46.930 DEBUG:boto:Port: 8000 2024-03-04T17:36:46.930 DEBUG:boto:Params: {} 2024-03-04T17:36:46.930 DEBUG:boto:Token: None 2024-03-04T17:36:46.930 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:46 GMT /ppfahn-10/?versions 2024-03-04T17:36:46.930 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wlL2FCRPUPTQR5IrfHXXDC/vUhI= 2024-03-04T17:36:46.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wlL2FCRPUPTQR5IrfHXXDC/vUhI=', 'Content-Length': '0'} 2024-03-04T17:36:46.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c189fe6b8f17e495-0065e606ae-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:46.933 DEBUG:boto:b'ppfahn-10999false&nulltrue2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:46.933 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:46.933 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:46.933 DEBUG:boto:path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:36:46.933 DEBUG:boto:auth_path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:36:46.934 DEBUG:boto:Method: GET 2024-03-04T17:36:46.934 DEBUG:boto:Path: /ppfahn-10/?versions&max-keys=999 2024-03-04T17:36:46.934 DEBUG:boto:Data: 2024-03-04T17:36:46.934 DEBUG:boto:Headers: {} 2024-03-04T17:36:46.934 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:46.934 DEBUG:boto:Port: 8000 2024-03-04T17:36:46.934 DEBUG:boto:Params: {} 2024-03-04T17:36:46.934 DEBUG:boto:Token: None 2024-03-04T17:36:46.934 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:46 GMT /ppfahn-10/?versions 2024-03-04T17:36:46.934 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wlL2FCRPUPTQR5IrfHXXDC/vUhI= 2024-03-04T17:36:46.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wlL2FCRPUPTQR5IrfHXXDC/vUhI=', 'Content-Length': '0'} 2024-03-04T17:36:46.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e5609cecbe630ca-0065e606ae-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:46.939 DEBUG:boto:b'ppfahn-10999false&nulltrue2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:46.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:36:46.940 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:36:46.940 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:36:46.940 DEBUG:boto:path=/ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.940 DEBUG:boto:auth_path=/ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.940 DEBUG:boto:Method: GET 2024-03-04T17:36:46.940 DEBUG:boto:Path: /ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.940 DEBUG:boto:Data: 2024-03-04T17:36:46.940 DEBUG:boto:Headers: {} 2024-03-04T17:36:46.941 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:46.941 DEBUG:boto:Port: 8000 2024-03-04T17:36:46.941 DEBUG:boto:Params: {} 2024-03-04T17:36:46.941 DEBUG:boto:Token: None 2024-03-04T17:36:46.941 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:46 GMT /ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.941 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:reuuKWrrP14kwWh6n4fRE8yYsjg= 2024-03-04T17:36:46.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:reuuKWrrP14kwWh6n4fRE8yYsjg=', 'Content-Length': '0'} 2024-03-04T17:36:46.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f71f694701f79324-0065e606ae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:46.984 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:36:46.984 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:36:46.984 DEBUG:boto:path=/ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.984 DEBUG:boto:auth_path=/ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.985 DEBUG:boto:Method: GET 2024-03-04T17:36:46.985 DEBUG:boto:Path: /ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.985 DEBUG:boto:Data: 2024-03-04T17:36:46.985 DEBUG:boto:Headers: {} 2024-03-04T17:36:46.985 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:46.985 DEBUG:boto:Port: 8000 2024-03-04T17:36:46.985 DEBUG:boto:Params: {} 2024-03-04T17:36:46.985 DEBUG:boto:Token: None 2024-03-04T17:36:46.985 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:46 GMT /ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.986 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:reuuKWrrP14kwWh6n4fRE8yYsjg= 2024-03-04T17:36:46.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:reuuKWrrP14kwWh6n4fRE8yYsjg=', 'Content-Length': '0'} 2024-03-04T17:36:46.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062f0e6e95fcda814-0065e606ae-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:46.989 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:36:46.989 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:36:46.989 DEBUG:boto:path=/ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.990 DEBUG:boto:auth_path=/ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.990 DEBUG:boto:Method: HEAD 2024-03-04T17:36:46.990 DEBUG:boto:Path: /ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.990 DEBUG:boto:Data: 2024-03-04T17:36:46.990 DEBUG:boto:Headers: {} 2024-03-04T17:36:46.990 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:46.990 DEBUG:boto:Port: 8000 2024-03-04T17:36:46.990 DEBUG:boto:Params: {} 2024-03-04T17:36:46.990 DEBUG:boto:Token: None 2024-03-04T17:36:46.990 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:46 GMT /ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.991 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yp/IMHYHffrJaE0rn78tL4mS0QE= 2024-03-04T17:36:46.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yp/IMHYHffrJaE0rn78tL4mS0QE=', 'Content-Length': '0'} 2024-03-04T17:36:46.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000680e126ec781926a-0065e606ae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:46.994 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:36:46.994 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:36:46.994 DEBUG:boto:path=/ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.994 DEBUG:boto:auth_path=/ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.994 DEBUG:boto:Method: HEAD 2024-03-04T17:36:46.994 DEBUG:boto:Path: /ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.995 DEBUG:boto:Data: 2024-03-04T17:36:46.995 DEBUG:boto:Headers: {} 2024-03-04T17:36:46.995 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:46.995 DEBUG:boto:Port: 8000 2024-03-04T17:36:46.995 DEBUG:boto:Params: {} 2024-03-04T17:36:46.995 DEBUG:boto:Token: None 2024-03-04T17:36:46.995 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:46 GMT /ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.995 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yp/IMHYHffrJaE0rn78tL4mS0QE= 2024-03-04T17:36:46.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yp/IMHYHffrJaE0rn78tL4mS0QE=', 'Content-Length': '0'} 2024-03-04T17:36:46.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c79c917551b4480e-0065e606ae-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:46.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:36:46.999 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:36:46.999 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:36:46.999 DEBUG:boto:path=/ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.999 DEBUG:boto:auth_path=/ppfahn-10/%26?versionId=null 2024-03-04T17:36:46.999 DEBUG:boto:Method: GET 2024-03-04T17:36:46.999 DEBUG:boto:Path: /ppfahn-10/%26?versionId=null 2024-03-04T17:36:47.000 DEBUG:boto:Data: 2024-03-04T17:36:47.000 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.000 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.000 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.000 DEBUG:boto:Params: {} 2024-03-04T17:36:47.000 DEBUG:boto:Token: None 2024-03-04T17:36:47.000 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/%26?versionId=null 2024-03-04T17:36:47.000 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:46ydHURTLMViGvW3hufj9aQ1VNk= 2024-03-04T17:36:47.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:46ydHURTLMViGvW3hufj9aQ1VNk=', 'Content-Length': '0'} 2024-03-04T17:36:47.003 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043823d31a8f750a6-0065e606ae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.044 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:36:47.044 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:36:47.044 DEBUG:boto:path=/ppfahn-10/%26?versionId=null 2024-03-04T17:36:47.044 DEBUG:boto:auth_path=/ppfahn-10/%26?versionId=null 2024-03-04T17:36:47.044 DEBUG:boto:Method: GET 2024-03-04T17:36:47.045 DEBUG:boto:Path: /ppfahn-10/%26?versionId=null 2024-03-04T17:36:47.045 DEBUG:boto:Data: 2024-03-04T17:36:47.045 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.045 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.045 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.045 DEBUG:boto:Params: {} 2024-03-04T17:36:47.045 DEBUG:boto:Token: None 2024-03-04T17:36:47.045 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/%26?versionId=null 2024-03-04T17:36:47.045 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:46ydHURTLMViGvW3hufj9aQ1VNk= 2024-03-04T17:36:47.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:46ydHURTLMViGvW3hufj9aQ1VNk=', 'Content-Length': '0'} 2024-03-04T17:36:47.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005957973f25b89918-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.049 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.049 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.049 DEBUG:boto:path=/ppfahn-10/?prefix=%26 2024-03-04T17:36:47.049 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%26 2024-03-04T17:36:47.049 DEBUG:boto:Method: GET 2024-03-04T17:36:47.049 DEBUG:boto:Path: /ppfahn-10/?prefix=%26 2024-03-04T17:36:47.049 DEBUG:boto:Data: 2024-03-04T17:36:47.050 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.050 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.050 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.050 DEBUG:boto:Params: {} 2024-03-04T17:36:47.050 DEBUG:boto:Token: None 2024-03-04T17:36:47.050 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.050 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008192bb408a3e7298-0065e606af-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.096 DEBUG:boto:b'ppfahn-10&1000false&2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.096 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.097 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.097 DEBUG:boto:path=/ppfahn-10/?prefix=%26 2024-03-04T17:36:47.097 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%26 2024-03-04T17:36:47.097 DEBUG:boto:Method: GET 2024-03-04T17:36:47.097 DEBUG:boto:Path: /ppfahn-10/?prefix=%26 2024-03-04T17:36:47.097 DEBUG:boto:Data: 2024-03-04T17:36:47.097 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.097 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.097 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.097 DEBUG:boto:Params: {} 2024-03-04T17:36:47.097 DEBUG:boto:Token: None 2024-03-04T17:36:47.098 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.098 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058ea218782d76977-0065e606af-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.102 DEBUG:boto:b'ppfahn-10&1000false&2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.103 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:36:47.103 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:36:47.103 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.103 DEBUG:boto:Path: /ppfahn-10/%26 2024-03-04T17:36:47.103 DEBUG:boto:Data: 2024-03-04T17:36:47.103 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.103 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.103 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.103 DEBUG:boto:Params: {} 2024-03-04T17:36:47.103 DEBUG:boto:Token: None 2024-03-04T17:36:47.104 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/%26 2024-03-04T17:36:47.104 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3jsnmoh5+o4tgVL4c5k2VzE6+ow= 2024-03-04T17:36:47.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3jsnmoh5+o4tgVL4c5k2VzE6+ow=', 'Content-Length': '0'} 2024-03-04T17:36:47.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008cd2e8a0a205c91-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.107 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:36:47.107 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:36:47.107 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.107 DEBUG:boto:Path: /ppfahn-10/%26 2024-03-04T17:36:47.107 DEBUG:boto:Data: 2024-03-04T17:36:47.107 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.108 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.108 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.108 DEBUG:boto:Params: {} 2024-03-04T17:36:47.108 DEBUG:boto:Token: None 2024-03-04T17:36:47.108 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/%26 2024-03-04T17:36:47.108 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3jsnmoh5+o4tgVL4c5k2VzE6+ow= 2024-03-04T17:36:47.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3jsnmoh5+o4tgVL4c5k2VzE6+ow=', 'Content-Length': '0'} 2024-03-04T17:36:47.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075e550ec6cb42b09-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:36:47.111 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:36:47.112 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:36:47.112 DEBUG:boto:Method: GET 2024-03-04T17:36:47.112 DEBUG:boto:Path: /ppfahn-10/%26 2024-03-04T17:36:47.112 DEBUG:boto:Data: 2024-03-04T17:36:47.112 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.112 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.112 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.112 DEBUG:boto:Params: {} 2024-03-04T17:36:47.112 DEBUG:boto:Token: None 2024-03-04T17:36:47.112 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/%26 2024-03-04T17:36:47.113 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vwRLIsRruXYCprQMCtWcQQl8FyA= 2024-03-04T17:36:47.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vwRLIsRruXYCprQMCtWcQQl8FyA=', 'Content-Length': '0'} 2024-03-04T17:36:47.116 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000475d9d5c88972209-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.116 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:36:47.116 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:36:47.116 DEBUG:boto:Method: GET 2024-03-04T17:36:47.116 DEBUG:boto:Path: /ppfahn-10/%26 2024-03-04T17:36:47.116 DEBUG:boto:Data: 2024-03-04T17:36:47.116 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.116 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.116 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.117 DEBUG:boto:Params: {} 2024-03-04T17:36:47.119 DEBUG:boto:Token: None 2024-03-04T17:36:47.119 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/%26 2024-03-04T17:36:47.119 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vwRLIsRruXYCprQMCtWcQQl8FyA= 2024-03-04T17:36:47.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vwRLIsRruXYCprQMCtWcQQl8FyA=', 'Content-Length': '0'} 2024-03-04T17:36:47.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028563b73c959f68e-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.123 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.123 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.123 DEBUG:boto:path=/ppfahn-10/?prefix=%26 2024-03-04T17:36:47.123 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%26 2024-03-04T17:36:47.123 DEBUG:boto:Method: GET 2024-03-04T17:36:47.123 DEBUG:boto:Path: /ppfahn-10/?prefix=%26 2024-03-04T17:36:47.123 DEBUG:boto:Data: 2024-03-04T17:36:47.123 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.124 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.124 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.124 DEBUG:boto:Params: {} 2024-03-04T17:36:47.124 DEBUG:boto:Token: None 2024-03-04T17:36:47.124 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.124 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056ba7987fe1918de-0065e606af-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.128 DEBUG:boto:b'ppfahn-10&1000false&2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.129 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.129 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.129 DEBUG:boto:path=/ppfahn-10/?prefix=%26 2024-03-04T17:36:47.129 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%26 2024-03-04T17:36:47.129 DEBUG:boto:Method: GET 2024-03-04T17:36:47.129 DEBUG:boto:Path: /ppfahn-10/?prefix=%26 2024-03-04T17:36:47.129 DEBUG:boto:Data: 2024-03-04T17:36:47.129 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.129 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.130 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.130 DEBUG:boto:Params: {} 2024-03-04T17:36:47.130 DEBUG:boto:Token: None 2024-03-04T17:36:47.130 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.130 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3f99bc6c0e0a252-0065e606af-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.135 DEBUG:boto:b'ppfahn-10&1000false&2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:36:47.135 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:36:47.135 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:36:47.135 DEBUG:boto:path=/ppfahn-10/.?versionId=null 2024-03-04T17:36:47.135 DEBUG:boto:auth_path=/ppfahn-10/.?versionId=null 2024-03-04T17:36:47.135 DEBUG:boto:Method: GET 2024-03-04T17:36:47.135 DEBUG:boto:Path: /ppfahn-10/.?versionId=null 2024-03-04T17:36:47.136 DEBUG:boto:Data: 2024-03-04T17:36:47.136 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.136 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.136 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.136 DEBUG:boto:Params: {} 2024-03-04T17:36:47.136 DEBUG:boto:Token: None 2024-03-04T17:36:47.136 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.?versionId=null 2024-03-04T17:36:47.136 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l29VveMEJfpxC6itbxOTdPNzs7U= 2024-03-04T17:36:47.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l29VveMEJfpxC6itbxOTdPNzs7U=', 'Content-Length': '0'} 2024-03-04T17:36:47.139 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d69fd667d4ee9eb3-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.139 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:36:47.139 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:36:47.140 DEBUG:boto:path=/ppfahn-10/.?versionId=null 2024-03-04T17:36:47.140 DEBUG:boto:auth_path=/ppfahn-10/.?versionId=null 2024-03-04T17:36:47.140 DEBUG:boto:Method: GET 2024-03-04T17:36:47.140 DEBUG:boto:Path: /ppfahn-10/.?versionId=null 2024-03-04T17:36:47.140 DEBUG:boto:Data: 2024-03-04T17:36:47.140 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.140 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.140 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.140 DEBUG:boto:Params: {} 2024-03-04T17:36:47.140 DEBUG:boto:Token: None 2024-03-04T17:36:47.141 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.?versionId=null 2024-03-04T17:36:47.141 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l29VveMEJfpxC6itbxOTdPNzs7U= 2024-03-04T17:36:47.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l29VveMEJfpxC6itbxOTdPNzs7U=', 'Content-Length': '0'} 2024-03-04T17:36:47.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e26a60c7e249ca8b-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.184 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:36:47.184 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:36:47.184 DEBUG:boto:path=/ppfahn-10/.?versionId=null 2024-03-04T17:36:47.184 DEBUG:boto:auth_path=/ppfahn-10/.?versionId=null 2024-03-04T17:36:47.184 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.185 DEBUG:boto:Path: /ppfahn-10/.?versionId=null 2024-03-04T17:36:47.185 DEBUG:boto:Data: 2024-03-04T17:36:47.185 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.185 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.185 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.185 DEBUG:boto:Params: {} 2024-03-04T17:36:47.185 DEBUG:boto:Token: None 2024-03-04T17:36:47.185 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.?versionId=null 2024-03-04T17:36:47.185 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aBxMGRZ1FLGnVwwUMqudYPxCQ/Q= 2024-03-04T17:36:47.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aBxMGRZ1FLGnVwwUMqudYPxCQ/Q=', 'Content-Length': '0'} 2024-03-04T17:36:47.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4b3b17fafc99c23-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.189 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:36:47.189 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:36:47.189 DEBUG:boto:path=/ppfahn-10/.?versionId=null 2024-03-04T17:36:47.189 DEBUG:boto:auth_path=/ppfahn-10/.?versionId=null 2024-03-04T17:36:47.189 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.189 DEBUG:boto:Path: /ppfahn-10/.?versionId=null 2024-03-04T17:36:47.189 DEBUG:boto:Data: 2024-03-04T17:36:47.190 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.190 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.190 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.190 DEBUG:boto:Params: {} 2024-03-04T17:36:47.190 DEBUG:boto:Token: None 2024-03-04T17:36:47.190 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.?versionId=null 2024-03-04T17:36:47.190 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aBxMGRZ1FLGnVwwUMqudYPxCQ/Q= 2024-03-04T17:36:47.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aBxMGRZ1FLGnVwwUMqudYPxCQ/Q=', 'Content-Length': '0'} 2024-03-04T17:36:47.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055a9876f83509331-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:36:47.193 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:36:47.193 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:36:47.193 DEBUG:boto:path=/ppfahn-10/.?versionId=null 2024-03-04T17:36:47.194 DEBUG:boto:auth_path=/ppfahn-10/.?versionId=null 2024-03-04T17:36:47.194 DEBUG:boto:Method: GET 2024-03-04T17:36:47.194 DEBUG:boto:Path: /ppfahn-10/.?versionId=null 2024-03-04T17:36:47.194 DEBUG:boto:Data: 2024-03-04T17:36:47.194 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.194 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.194 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.194 DEBUG:boto:Params: {} 2024-03-04T17:36:47.194 DEBUG:boto:Token: None 2024-03-04T17:36:47.194 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.?versionId=null 2024-03-04T17:36:47.195 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l29VveMEJfpxC6itbxOTdPNzs7U= 2024-03-04T17:36:47.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l29VveMEJfpxC6itbxOTdPNzs7U=', 'Content-Length': '0'} 2024-03-04T17:36:47.198 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008612ad8a50361cf4-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.240 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:36:47.240 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:36:47.240 DEBUG:boto:path=/ppfahn-10/.?versionId=null 2024-03-04T17:36:47.240 DEBUG:boto:auth_path=/ppfahn-10/.?versionId=null 2024-03-04T17:36:47.240 DEBUG:boto:Method: GET 2024-03-04T17:36:47.241 DEBUG:boto:Path: /ppfahn-10/.?versionId=null 2024-03-04T17:36:47.241 DEBUG:boto:Data: 2024-03-04T17:36:47.241 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.241 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.241 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.241 DEBUG:boto:Params: {} 2024-03-04T17:36:47.241 DEBUG:boto:Token: None 2024-03-04T17:36:47.241 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.?versionId=null 2024-03-04T17:36:47.241 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l29VveMEJfpxC6itbxOTdPNzs7U= 2024-03-04T17:36:47.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l29VveMEJfpxC6itbxOTdPNzs7U=', 'Content-Length': '0'} 2024-03-04T17:36:47.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008aba1abf557e13b2-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.244 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.245 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.245 DEBUG:boto:path=/ppfahn-10/?prefix=. 2024-03-04T17:36:47.245 DEBUG:boto:auth_path=/ppfahn-10/?prefix=. 2024-03-04T17:36:47.245 DEBUG:boto:Method: GET 2024-03-04T17:36:47.245 DEBUG:boto:Path: /ppfahn-10/?prefix=. 2024-03-04T17:36:47.245 DEBUG:boto:Data: 2024-03-04T17:36:47.245 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.245 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.245 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.245 DEBUG:boto:Params: {} 2024-03-04T17:36:47.245 DEBUG:boto:Token: None 2024-03-04T17:36:47.246 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.246 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a76c5dbad94c4054-0065e606af-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.250 DEBUG:boto:b'ppfahn-10.1000false.2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.251 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.251 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.251 DEBUG:boto:path=/ppfahn-10/?prefix=. 2024-03-04T17:36:47.251 DEBUG:boto:auth_path=/ppfahn-10/?prefix=. 2024-03-04T17:36:47.251 DEBUG:boto:Method: GET 2024-03-04T17:36:47.251 DEBUG:boto:Path: /ppfahn-10/?prefix=. 2024-03-04T17:36:47.251 DEBUG:boto:Data: 2024-03-04T17:36:47.251 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.251 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.251 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.251 DEBUG:boto:Params: {} 2024-03-04T17:36:47.252 DEBUG:boto:Token: None 2024-03-04T17:36:47.252 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.252 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006209d19edd1c601e-0065e606af-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.256 DEBUG:boto:b'ppfahn-10.1000false.2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.257 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:36:47.257 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:36:47.257 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.257 DEBUG:boto:Path: /ppfahn-10/. 2024-03-04T17:36:47.257 DEBUG:boto:Data: 2024-03-04T17:36:47.258 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.258 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.258 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.258 DEBUG:boto:Params: {} 2024-03-04T17:36:47.258 DEBUG:boto:Token: None 2024-03-04T17:36:47.258 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/. 2024-03-04T17:36:47.258 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7/fOlP+DesLwB+isbz61QWiDBuA= 2024-03-04T17:36:47.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7/fOlP+DesLwB+isbz61QWiDBuA=', 'Content-Length': '0'} 2024-03-04T17:36:47.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000698df3b6a6c08ac4-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.261 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:36:47.261 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:36:47.261 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.261 DEBUG:boto:Path: /ppfahn-10/. 2024-03-04T17:36:47.261 DEBUG:boto:Data: 2024-03-04T17:36:47.262 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.262 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.262 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.262 DEBUG:boto:Params: {} 2024-03-04T17:36:47.262 DEBUG:boto:Token: None 2024-03-04T17:36:47.262 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/. 2024-03-04T17:36:47.262 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7/fOlP+DesLwB+isbz61QWiDBuA= 2024-03-04T17:36:47.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7/fOlP+DesLwB+isbz61QWiDBuA=', 'Content-Length': '0'} 2024-03-04T17:36:47.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000571259c28f3dd9a1-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:36:47.265 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:36:47.265 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:36:47.265 DEBUG:boto:Method: GET 2024-03-04T17:36:47.265 DEBUG:boto:Path: /ppfahn-10/. 2024-03-04T17:36:47.265 DEBUG:boto:Data: 2024-03-04T17:36:47.266 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.266 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.266 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.266 DEBUG:boto:Params: {} 2024-03-04T17:36:47.266 DEBUG:boto:Token: None 2024-03-04T17:36:47.266 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/. 2024-03-04T17:36:47.266 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6XD6ZThTUtl7EcQlw0TzGhD7E+8= 2024-03-04T17:36:47.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6XD6ZThTUtl7EcQlw0TzGhD7E+8=', 'Content-Length': '0'} 2024-03-04T17:36:47.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f092fc11417d9a76-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.312 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:36:47.312 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:36:47.313 DEBUG:boto:Method: GET 2024-03-04T17:36:47.313 DEBUG:boto:Path: /ppfahn-10/. 2024-03-04T17:36:47.313 DEBUG:boto:Data: 2024-03-04T17:36:47.313 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.313 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.313 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.313 DEBUG:boto:Params: {} 2024-03-04T17:36:47.313 DEBUG:boto:Token: None 2024-03-04T17:36:47.314 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/. 2024-03-04T17:36:47.314 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6XD6ZThTUtl7EcQlw0TzGhD7E+8= 2024-03-04T17:36:47.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6XD6ZThTUtl7EcQlw0TzGhD7E+8=', 'Content-Length': '0'} 2024-03-04T17:36:47.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a3bbc5926a7a186-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.317 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.317 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.317 DEBUG:boto:path=/ppfahn-10/?prefix=. 2024-03-04T17:36:47.317 DEBUG:boto:auth_path=/ppfahn-10/?prefix=. 2024-03-04T17:36:47.318 DEBUG:boto:Method: GET 2024-03-04T17:36:47.318 DEBUG:boto:Path: /ppfahn-10/?prefix=. 2024-03-04T17:36:47.318 DEBUG:boto:Data: 2024-03-04T17:36:47.318 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.318 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.318 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.319 DEBUG:boto:Params: {} 2024-03-04T17:36:47.319 DEBUG:boto:Token: None 2024-03-04T17:36:47.319 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.319 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ab7a8a01460eae4-0065e606af-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.324 DEBUG:boto:b'ppfahn-10.1000false.2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.324 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.324 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.325 DEBUG:boto:path=/ppfahn-10/?prefix=. 2024-03-04T17:36:47.325 DEBUG:boto:auth_path=/ppfahn-10/?prefix=. 2024-03-04T17:36:47.325 DEBUG:boto:Method: GET 2024-03-04T17:36:47.325 DEBUG:boto:Path: /ppfahn-10/?prefix=. 2024-03-04T17:36:47.325 DEBUG:boto:Data: 2024-03-04T17:36:47.325 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.325 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.325 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.325 DEBUG:boto:Params: {} 2024-03-04T17:36:47.325 DEBUG:boto:Token: None 2024-03-04T17:36:47.325 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.326 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeecb017ae89ade8-0065e606af-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.331 DEBUG:boto:b'ppfahn-10.1000false.2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:36:47.331 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:36:47.332 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:36:47.332 DEBUG:boto:path=/ppfahn-10/..?versionId=null 2024-03-04T17:36:47.332 DEBUG:boto:auth_path=/ppfahn-10/..?versionId=null 2024-03-04T17:36:47.332 DEBUG:boto:Method: GET 2024-03-04T17:36:47.332 DEBUG:boto:Path: /ppfahn-10/..?versionId=null 2024-03-04T17:36:47.332 DEBUG:boto:Data: 2024-03-04T17:36:47.332 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.332 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.332 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.332 DEBUG:boto:Params: {} 2024-03-04T17:36:47.333 DEBUG:boto:Token: None 2024-03-04T17:36:47.333 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/..?versionId=null 2024-03-04T17:36:47.333 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9sAmUx/TFxhmOV8LA6XEH7my8LE= 2024-03-04T17:36:47.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9sAmUx/TFxhmOV8LA6XEH7my8LE=', 'Content-Length': '0'} 2024-03-04T17:36:47.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e8229860ce14c54-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.336 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:36:47.336 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:36:47.336 DEBUG:boto:path=/ppfahn-10/..?versionId=null 2024-03-04T17:36:47.336 DEBUG:boto:auth_path=/ppfahn-10/..?versionId=null 2024-03-04T17:36:47.336 DEBUG:boto:Method: GET 2024-03-04T17:36:47.336 DEBUG:boto:Path: /ppfahn-10/..?versionId=null 2024-03-04T17:36:47.337 DEBUG:boto:Data: 2024-03-04T17:36:47.337 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.337 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.337 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.337 DEBUG:boto:Params: {} 2024-03-04T17:36:47.337 DEBUG:boto:Token: None 2024-03-04T17:36:47.337 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/..?versionId=null 2024-03-04T17:36:47.337 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9sAmUx/TFxhmOV8LA6XEH7my8LE= 2024-03-04T17:36:47.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9sAmUx/TFxhmOV8LA6XEH7my8LE=', 'Content-Length': '0'} 2024-03-04T17:36:47.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001157960dd51add7c-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.341 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:36:47.341 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:36:47.341 DEBUG:boto:path=/ppfahn-10/..?versionId=null 2024-03-04T17:36:47.341 DEBUG:boto:auth_path=/ppfahn-10/..?versionId=null 2024-03-04T17:36:47.341 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.341 DEBUG:boto:Path: /ppfahn-10/..?versionId=null 2024-03-04T17:36:47.341 DEBUG:boto:Data: 2024-03-04T17:36:47.341 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.341 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.341 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.342 DEBUG:boto:Params: {} 2024-03-04T17:36:47.342 DEBUG:boto:Token: None 2024-03-04T17:36:47.342 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/..?versionId=null 2024-03-04T17:36:47.342 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ClJAr9BM31nOGWyr/6bAlzgNTAs= 2024-03-04T17:36:47.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ClJAr9BM31nOGWyr/6bAlzgNTAs=', 'Content-Length': '0'} 2024-03-04T17:36:47.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f105c4579f2f029-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.345 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:36:47.345 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:36:47.345 DEBUG:boto:path=/ppfahn-10/..?versionId=null 2024-03-04T17:36:47.345 DEBUG:boto:auth_path=/ppfahn-10/..?versionId=null 2024-03-04T17:36:47.345 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.345 DEBUG:boto:Path: /ppfahn-10/..?versionId=null 2024-03-04T17:36:47.346 DEBUG:boto:Data: 2024-03-04T17:36:47.346 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.346 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.346 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.346 DEBUG:boto:Params: {} 2024-03-04T17:36:47.346 DEBUG:boto:Token: None 2024-03-04T17:36:47.346 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/..?versionId=null 2024-03-04T17:36:47.346 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ClJAr9BM31nOGWyr/6bAlzgNTAs= 2024-03-04T17:36:47.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ClJAr9BM31nOGWyr/6bAlzgNTAs=', 'Content-Length': '0'} 2024-03-04T17:36:47.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e494ac4e2742550-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:36:47.349 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:36:47.349 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:36:47.349 DEBUG:boto:path=/ppfahn-10/..?versionId=null 2024-03-04T17:36:47.349 DEBUG:boto:auth_path=/ppfahn-10/..?versionId=null 2024-03-04T17:36:47.350 DEBUG:boto:Method: GET 2024-03-04T17:36:47.350 DEBUG:boto:Path: /ppfahn-10/..?versionId=null 2024-03-04T17:36:47.350 DEBUG:boto:Data: 2024-03-04T17:36:47.350 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.350 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.350 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.350 DEBUG:boto:Params: {} 2024-03-04T17:36:47.350 DEBUG:boto:Token: None 2024-03-04T17:36:47.350 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/..?versionId=null 2024-03-04T17:36:47.350 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9sAmUx/TFxhmOV8LA6XEH7my8LE= 2024-03-04T17:36:47.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9sAmUx/TFxhmOV8LA6XEH7my8LE=', 'Content-Length': '0'} 2024-03-04T17:36:47.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d6321c3b8f0750b-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.396 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:36:47.396 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:36:47.396 DEBUG:boto:path=/ppfahn-10/..?versionId=null 2024-03-04T17:36:47.396 DEBUG:boto:auth_path=/ppfahn-10/..?versionId=null 2024-03-04T17:36:47.396 DEBUG:boto:Method: GET 2024-03-04T17:36:47.396 DEBUG:boto:Path: /ppfahn-10/..?versionId=null 2024-03-04T17:36:47.397 DEBUG:boto:Data: 2024-03-04T17:36:47.397 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.397 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.397 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.397 DEBUG:boto:Params: {} 2024-03-04T17:36:47.397 DEBUG:boto:Token: None 2024-03-04T17:36:47.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/..?versionId=null 2024-03-04T17:36:47.397 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9sAmUx/TFxhmOV8LA6XEH7my8LE= 2024-03-04T17:36:47.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9sAmUx/TFxhmOV8LA6XEH7my8LE=', 'Content-Length': '0'} 2024-03-04T17:36:47.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c0f7a42880ac3bf-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.401 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.401 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.401 DEBUG:boto:path=/ppfahn-10/?prefix=.. 2024-03-04T17:36:47.401 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.. 2024-03-04T17:36:47.401 DEBUG:boto:Method: GET 2024-03-04T17:36:47.401 DEBUG:boto:Path: /ppfahn-10/?prefix=.. 2024-03-04T17:36:47.401 DEBUG:boto:Data: 2024-03-04T17:36:47.401 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.401 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.401 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.402 DEBUG:boto:Params: {} 2024-03-04T17:36:47.402 DEBUG:boto:Token: None 2024-03-04T17:36:47.402 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.402 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dde2623f7bba58f0-0065e606af-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.406 DEBUG:boto:b'ppfahn-10..1000false..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.407 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.407 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.407 DEBUG:boto:path=/ppfahn-10/?prefix=.. 2024-03-04T17:36:47.407 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.. 2024-03-04T17:36:47.407 DEBUG:boto:Method: GET 2024-03-04T17:36:47.407 DEBUG:boto:Path: /ppfahn-10/?prefix=.. 2024-03-04T17:36:47.407 DEBUG:boto:Data: 2024-03-04T17:36:47.407 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.408 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.408 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.408 DEBUG:boto:Params: {} 2024-03-04T17:36:47.408 DEBUG:boto:Token: None 2024-03-04T17:36:47.408 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.408 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b93d44fd660674f9-0065e606af-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.413 DEBUG:boto:b'ppfahn-10..1000false..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.413 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:36:47.413 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:36:47.413 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.413 DEBUG:boto:Path: /ppfahn-10/.. 2024-03-04T17:36:47.413 DEBUG:boto:Data: 2024-03-04T17:36:47.414 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.414 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.414 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.414 DEBUG:boto:Params: {} 2024-03-04T17:36:47.414 DEBUG:boto:Token: None 2024-03-04T17:36:47.414 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.. 2024-03-04T17:36:47.414 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZvRN2ao46fy1fZc+l2KJJDl+IlM= 2024-03-04T17:36:47.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZvRN2ao46fy1fZc+l2KJJDl+IlM=', 'Content-Length': '0'} 2024-03-04T17:36:47.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7fa83cb3fdeb9c3-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.417 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:36:47.417 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:36:47.417 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.417 DEBUG:boto:Path: /ppfahn-10/.. 2024-03-04T17:36:47.417 DEBUG:boto:Data: 2024-03-04T17:36:47.417 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.418 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.418 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.418 DEBUG:boto:Params: {} 2024-03-04T17:36:47.418 DEBUG:boto:Token: None 2024-03-04T17:36:47.418 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.. 2024-03-04T17:36:47.418 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZvRN2ao46fy1fZc+l2KJJDl+IlM= 2024-03-04T17:36:47.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZvRN2ao46fy1fZc+l2KJJDl+IlM=', 'Content-Length': '0'} 2024-03-04T17:36:47.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053b4754a0a3021d8-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:36:47.422 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:36:47.422 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:36:47.422 DEBUG:boto:Method: GET 2024-03-04T17:36:47.422 DEBUG:boto:Path: /ppfahn-10/.. 2024-03-04T17:36:47.422 DEBUG:boto:Data: 2024-03-04T17:36:47.422 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.422 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.422 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.422 DEBUG:boto:Params: {} 2024-03-04T17:36:47.422 DEBUG:boto:Token: None 2024-03-04T17:36:47.423 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.. 2024-03-04T17:36:47.423 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:C32LKbUdeodSQaOUoQJTpMdIrEM= 2024-03-04T17:36:47.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:C32LKbUdeodSQaOUoQJTpMdIrEM=', 'Content-Length': '0'} 2024-03-04T17:36:47.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094dc8735e0f21bd8-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.468 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:36:47.468 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:36:47.468 DEBUG:boto:Method: GET 2024-03-04T17:36:47.468 DEBUG:boto:Path: /ppfahn-10/.. 2024-03-04T17:36:47.469 DEBUG:boto:Data: 2024-03-04T17:36:47.469 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.469 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.469 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.469 DEBUG:boto:Params: {} 2024-03-04T17:36:47.469 DEBUG:boto:Token: None 2024-03-04T17:36:47.469 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.. 2024-03-04T17:36:47.469 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:C32LKbUdeodSQaOUoQJTpMdIrEM= 2024-03-04T17:36:47.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:C32LKbUdeodSQaOUoQJTpMdIrEM=', 'Content-Length': '0'} 2024-03-04T17:36:47.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000296d82c7839491f5-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.473 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.473 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.473 DEBUG:boto:path=/ppfahn-10/?prefix=.. 2024-03-04T17:36:47.473 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.. 2024-03-04T17:36:47.473 DEBUG:boto:Method: GET 2024-03-04T17:36:47.473 DEBUG:boto:Path: /ppfahn-10/?prefix=.. 2024-03-04T17:36:47.473 DEBUG:boto:Data: 2024-03-04T17:36:47.473 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.473 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.474 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.474 DEBUG:boto:Params: {} 2024-03-04T17:36:47.474 DEBUG:boto:Token: None 2024-03-04T17:36:47.474 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.474 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7b5e8e330c485e4-0065e606af-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.479 DEBUG:boto:b'ppfahn-10..1000false..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.479 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.479 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.479 DEBUG:boto:path=/ppfahn-10/?prefix=.. 2024-03-04T17:36:47.479 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.. 2024-03-04T17:36:47.479 DEBUG:boto:Method: GET 2024-03-04T17:36:47.480 DEBUG:boto:Path: /ppfahn-10/?prefix=.. 2024-03-04T17:36:47.480 DEBUG:boto:Data: 2024-03-04T17:36:47.480 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.480 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.480 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.480 DEBUG:boto:Params: {} 2024-03-04T17:36:47.480 DEBUG:boto:Token: None 2024-03-04T17:36:47.480 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.480 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015f23dd3cda9b371-0065e606af-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.529 DEBUG:boto:b'ppfahn-10..1000false..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:36:47.530 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:36:47.530 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:36:47.530 DEBUG:boto:path=/ppfahn-10/...?versionId=null 2024-03-04T17:36:47.530 DEBUG:boto:auth_path=/ppfahn-10/...?versionId=null 2024-03-04T17:36:47.530 DEBUG:boto:Method: GET 2024-03-04T17:36:47.530 DEBUG:boto:Path: /ppfahn-10/...?versionId=null 2024-03-04T17:36:47.530 DEBUG:boto:Data: 2024-03-04T17:36:47.531 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.531 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.531 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.531 DEBUG:boto:Params: {} 2024-03-04T17:36:47.531 DEBUG:boto:Token: None 2024-03-04T17:36:47.531 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/...?versionId=null 2024-03-04T17:36:47.531 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vPyJkf5gJ2Fk1tAxKGBAyeQk4zE= 2024-03-04T17:36:47.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vPyJkf5gJ2Fk1tAxKGBAyeQk4zE=', 'Content-Length': '0'} 2024-03-04T17:36:47.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000124f7a617699991d-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.535 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:36:47.535 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:36:47.535 DEBUG:boto:path=/ppfahn-10/...?versionId=null 2024-03-04T17:36:47.535 DEBUG:boto:auth_path=/ppfahn-10/...?versionId=null 2024-03-04T17:36:47.535 DEBUG:boto:Method: GET 2024-03-04T17:36:47.536 DEBUG:boto:Path: /ppfahn-10/...?versionId=null 2024-03-04T17:36:47.536 DEBUG:boto:Data: 2024-03-04T17:36:47.536 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.536 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.536 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.536 DEBUG:boto:Params: {} 2024-03-04T17:36:47.536 DEBUG:boto:Token: None 2024-03-04T17:36:47.536 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/...?versionId=null 2024-03-04T17:36:47.536 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vPyJkf5gJ2Fk1tAxKGBAyeQk4zE= 2024-03-04T17:36:47.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vPyJkf5gJ2Fk1tAxKGBAyeQk4zE=', 'Content-Length': '0'} 2024-03-04T17:36:47.539 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edf59cbcf837cbda-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.539 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:36:47.540 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:36:47.540 DEBUG:boto:path=/ppfahn-10/...?versionId=null 2024-03-04T17:36:47.540 DEBUG:boto:auth_path=/ppfahn-10/...?versionId=null 2024-03-04T17:36:47.540 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.540 DEBUG:boto:Path: /ppfahn-10/...?versionId=null 2024-03-04T17:36:47.540 DEBUG:boto:Data: 2024-03-04T17:36:47.540 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.540 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.540 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.540 DEBUG:boto:Params: {} 2024-03-04T17:36:47.540 DEBUG:boto:Token: None 2024-03-04T17:36:47.541 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/...?versionId=null 2024-03-04T17:36:47.541 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ItA4/xWIW0FyHL2SzIJFCPXMk7I= 2024-03-04T17:36:47.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ItA4/xWIW0FyHL2SzIJFCPXMk7I=', 'Content-Length': '0'} 2024-03-04T17:36:47.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000494c43f436d2d24f-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.544 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:36:47.544 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:36:47.544 DEBUG:boto:path=/ppfahn-10/...?versionId=null 2024-03-04T17:36:47.544 DEBUG:boto:auth_path=/ppfahn-10/...?versionId=null 2024-03-04T17:36:47.544 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.545 DEBUG:boto:Path: /ppfahn-10/...?versionId=null 2024-03-04T17:36:47.545 DEBUG:boto:Data: 2024-03-04T17:36:47.545 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.545 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.545 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.545 DEBUG:boto:Params: {} 2024-03-04T17:36:47.545 DEBUG:boto:Token: None 2024-03-04T17:36:47.545 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/...?versionId=null 2024-03-04T17:36:47.545 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ItA4/xWIW0FyHL2SzIJFCPXMk7I= 2024-03-04T17:36:47.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ItA4/xWIW0FyHL2SzIJFCPXMk7I=', 'Content-Length': '0'} 2024-03-04T17:36:47.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae810c5b7aa32522-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:36:47.548 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:36:47.549 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:36:47.549 DEBUG:boto:path=/ppfahn-10/...?versionId=null 2024-03-04T17:36:47.549 DEBUG:boto:auth_path=/ppfahn-10/...?versionId=null 2024-03-04T17:36:47.549 DEBUG:boto:Method: GET 2024-03-04T17:36:47.549 DEBUG:boto:Path: /ppfahn-10/...?versionId=null 2024-03-04T17:36:47.549 DEBUG:boto:Data: 2024-03-04T17:36:47.549 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.549 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.549 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.549 DEBUG:boto:Params: {} 2024-03-04T17:36:47.549 DEBUG:boto:Token: None 2024-03-04T17:36:47.550 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/...?versionId=null 2024-03-04T17:36:47.550 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vPyJkf5gJ2Fk1tAxKGBAyeQk4zE= 2024-03-04T17:36:47.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vPyJkf5gJ2Fk1tAxKGBAyeQk4zE=', 'Content-Length': '0'} 2024-03-04T17:36:47.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e5d00473e07c07f-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.553 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:36:47.553 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:36:47.553 DEBUG:boto:path=/ppfahn-10/...?versionId=null 2024-03-04T17:36:47.553 DEBUG:boto:auth_path=/ppfahn-10/...?versionId=null 2024-03-04T17:36:47.553 DEBUG:boto:Method: GET 2024-03-04T17:36:47.553 DEBUG:boto:Path: /ppfahn-10/...?versionId=null 2024-03-04T17:36:47.554 DEBUG:boto:Data: 2024-03-04T17:36:47.554 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.554 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.554 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.554 DEBUG:boto:Params: {} 2024-03-04T17:36:47.554 DEBUG:boto:Token: None 2024-03-04T17:36:47.554 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/...?versionId=null 2024-03-04T17:36:47.554 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vPyJkf5gJ2Fk1tAxKGBAyeQk4zE= 2024-03-04T17:36:47.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vPyJkf5gJ2Fk1tAxKGBAyeQk4zE=', 'Content-Length': '0'} 2024-03-04T17:36:47.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea8082fb0c30103a-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.557 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.558 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.558 DEBUG:boto:path=/ppfahn-10/?prefix=... 2024-03-04T17:36:47.558 DEBUG:boto:auth_path=/ppfahn-10/?prefix=... 2024-03-04T17:36:47.558 DEBUG:boto:Method: GET 2024-03-04T17:36:47.558 DEBUG:boto:Path: /ppfahn-10/?prefix=... 2024-03-04T17:36:47.558 DEBUG:boto:Data: 2024-03-04T17:36:47.558 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.558 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.558 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.558 DEBUG:boto:Params: {} 2024-03-04T17:36:47.558 DEBUG:boto:Token: None 2024-03-04T17:36:47.559 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.559 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072f668e352e87a6d-0065e606af-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.563 DEBUG:boto:b'ppfahn-10...1000false...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.564 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.564 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.564 DEBUG:boto:path=/ppfahn-10/?prefix=... 2024-03-04T17:36:47.564 DEBUG:boto:auth_path=/ppfahn-10/?prefix=... 2024-03-04T17:36:47.564 DEBUG:boto:Method: GET 2024-03-04T17:36:47.564 DEBUG:boto:Path: /ppfahn-10/?prefix=... 2024-03-04T17:36:47.564 DEBUG:boto:Data: 2024-03-04T17:36:47.564 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.564 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.564 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.564 DEBUG:boto:Params: {} 2024-03-04T17:36:47.565 DEBUG:boto:Token: None 2024-03-04T17:36:47.565 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.565 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dff92d1d40209941-0065e606af-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.612 DEBUG:boto:b'ppfahn-10...1000false...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.612 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:36:47.613 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:36:47.613 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.613 DEBUG:boto:Path: /ppfahn-10/... 2024-03-04T17:36:47.613 DEBUG:boto:Data: 2024-03-04T17:36:47.613 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.613 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.613 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.613 DEBUG:boto:Params: {} 2024-03-04T17:36:47.613 DEBUG:boto:Token: None 2024-03-04T17:36:47.614 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/... 2024-03-04T17:36:47.614 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:viO0OJBzmQLcdwi6oSFi025sBcM= 2024-03-04T17:36:47.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:viO0OJBzmQLcdwi6oSFi025sBcM=', 'Content-Length': '0'} 2024-03-04T17:36:47.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000faab5708a31496dc-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.617 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:36:47.617 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:36:47.617 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.617 DEBUG:boto:Path: /ppfahn-10/... 2024-03-04T17:36:47.617 DEBUG:boto:Data: 2024-03-04T17:36:47.617 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.618 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.618 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.618 DEBUG:boto:Params: {} 2024-03-04T17:36:47.618 DEBUG:boto:Token: None 2024-03-04T17:36:47.618 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/... 2024-03-04T17:36:47.618 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:viO0OJBzmQLcdwi6oSFi025sBcM= 2024-03-04T17:36:47.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:viO0OJBzmQLcdwi6oSFi025sBcM=', 'Content-Length': '0'} 2024-03-04T17:36:47.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0f774b328430da2-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:36:47.621 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:36:47.621 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:36:47.621 DEBUG:boto:Method: GET 2024-03-04T17:36:47.621 DEBUG:boto:Path: /ppfahn-10/... 2024-03-04T17:36:47.622 DEBUG:boto:Data: 2024-03-04T17:36:47.622 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.622 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.622 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.622 DEBUG:boto:Params: {} 2024-03-04T17:36:47.622 DEBUG:boto:Token: None 2024-03-04T17:36:47.622 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/... 2024-03-04T17:36:47.622 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pAMLezN/RMFTr9Aw+cVtqwuw19U= 2024-03-04T17:36:47.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pAMLezN/RMFTr9Aw+cVtqwuw19U=', 'Content-Length': '0'} 2024-03-04T17:36:47.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e42c8a0bbf878601-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.668 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:36:47.668 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:36:47.668 DEBUG:boto:Method: GET 2024-03-04T17:36:47.669 DEBUG:boto:Path: /ppfahn-10/... 2024-03-04T17:36:47.669 DEBUG:boto:Data: 2024-03-04T17:36:47.669 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.669 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.669 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.669 DEBUG:boto:Params: {} 2024-03-04T17:36:47.669 DEBUG:boto:Token: None 2024-03-04T17:36:47.669 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/... 2024-03-04T17:36:47.669 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pAMLezN/RMFTr9Aw+cVtqwuw19U= 2024-03-04T17:36:47.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pAMLezN/RMFTr9Aw+cVtqwuw19U=', 'Content-Length': '0'} 2024-03-04T17:36:47.672 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009263ce9ee39778a9-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.673 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.673 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.673 DEBUG:boto:path=/ppfahn-10/?prefix=... 2024-03-04T17:36:47.673 DEBUG:boto:auth_path=/ppfahn-10/?prefix=... 2024-03-04T17:36:47.673 DEBUG:boto:Method: GET 2024-03-04T17:36:47.673 DEBUG:boto:Path: /ppfahn-10/?prefix=... 2024-03-04T17:36:47.673 DEBUG:boto:Data: 2024-03-04T17:36:47.674 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.674 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.674 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.674 DEBUG:boto:Params: {} 2024-03-04T17:36:47.674 DEBUG:boto:Token: None 2024-03-04T17:36:47.674 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.674 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f62eb7f511c0bda-0065e606af-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.679 DEBUG:boto:b'ppfahn-10...1000false...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.679 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.679 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.679 DEBUG:boto:path=/ppfahn-10/?prefix=... 2024-03-04T17:36:47.679 DEBUG:boto:auth_path=/ppfahn-10/?prefix=... 2024-03-04T17:36:47.680 DEBUG:boto:Method: GET 2024-03-04T17:36:47.680 DEBUG:boto:Path: /ppfahn-10/?prefix=... 2024-03-04T17:36:47.680 DEBUG:boto:Data: 2024-03-04T17:36:47.680 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.680 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.680 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.680 DEBUG:boto:Params: {} 2024-03-04T17:36:47.680 DEBUG:boto:Token: None 2024-03-04T17:36:47.680 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.680 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064d06d78cc45fdfb-0065e606af-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.685 DEBUG:boto:b'ppfahn-10...1000false...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:36:47.685 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:36:47.685 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:36:47.685 DEBUG:boto:path=/ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.686 DEBUG:boto:auth_path=/ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.686 DEBUG:boto:Method: GET 2024-03-04T17:36:47.686 DEBUG:boto:Path: /ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.686 DEBUG:boto:Data: 2024-03-04T17:36:47.686 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.686 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.686 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.686 DEBUG:boto:Params: {} 2024-03-04T17:36:47.686 DEBUG:boto:Token: None 2024-03-04T17:36:47.686 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.687 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gB4l/80RpSE0WyAeBdPKp/ZInH8= 2024-03-04T17:36:47.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gB4l/80RpSE0WyAeBdPKp/ZInH8=', 'Content-Length': '0'} 2024-03-04T17:36:47.690 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000690ad10b661b03d0-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.690 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:36:47.690 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:36:47.690 DEBUG:boto:path=/ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.690 DEBUG:boto:auth_path=/ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.690 DEBUG:boto:Method: GET 2024-03-04T17:36:47.690 DEBUG:boto:Path: /ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.690 DEBUG:boto:Data: 2024-03-04T17:36:47.690 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.691 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.691 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.691 DEBUG:boto:Params: {} 2024-03-04T17:36:47.691 DEBUG:boto:Token: None 2024-03-04T17:36:47.691 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.691 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gB4l/80RpSE0WyAeBdPKp/ZInH8= 2024-03-04T17:36:47.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gB4l/80RpSE0WyAeBdPKp/ZInH8=', 'Content-Length': '0'} 2024-03-04T17:36:47.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a07caa7e2f6e5b0-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.736 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:36:47.736 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:36:47.736 DEBUG:boto:path=/ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.736 DEBUG:boto:auth_path=/ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.737 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.737 DEBUG:boto:Path: /ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.737 DEBUG:boto:Data: 2024-03-04T17:36:47.737 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.737 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.737 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.737 DEBUG:boto:Params: {} 2024-03-04T17:36:47.737 DEBUG:boto:Token: None 2024-03-04T17:36:47.737 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.738 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/l8gJj6ETDfxSbv71aMrxjFG4rk= 2024-03-04T17:36:47.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/l8gJj6ETDfxSbv71aMrxjFG4rk=', 'Content-Length': '0'} 2024-03-04T17:36:47.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7d9a1ae82a7d1b6-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.741 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:36:47.741 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:36:47.741 DEBUG:boto:path=/ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.741 DEBUG:boto:auth_path=/ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.741 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.741 DEBUG:boto:Path: /ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.741 DEBUG:boto:Data: 2024-03-04T17:36:47.741 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.741 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.741 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.742 DEBUG:boto:Params: {} 2024-03-04T17:36:47.742 DEBUG:boto:Token: None 2024-03-04T17:36:47.742 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.742 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/l8gJj6ETDfxSbv71aMrxjFG4rk= 2024-03-04T17:36:47.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/l8gJj6ETDfxSbv71aMrxjFG4rk=', 'Content-Length': '0'} 2024-03-04T17:36:47.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000caa2c68bdeaa68cf-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:36:47.745 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:36:47.745 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:36:47.745 DEBUG:boto:path=/ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.745 DEBUG:boto:auth_path=/ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.745 DEBUG:boto:Method: GET 2024-03-04T17:36:47.745 DEBUG:boto:Path: /ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.745 DEBUG:boto:Data: 2024-03-04T17:36:47.746 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.746 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.746 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.746 DEBUG:boto:Params: {} 2024-03-04T17:36:47.746 DEBUG:boto:Token: None 2024-03-04T17:36:47.746 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.746 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gB4l/80RpSE0WyAeBdPKp/ZInH8= 2024-03-04T17:36:47.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gB4l/80RpSE0WyAeBdPKp/ZInH8=', 'Content-Length': '0'} 2024-03-04T17:36:47.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ef0103769b2d094-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.792 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:36:47.792 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:36:47.792 DEBUG:boto:path=/ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.792 DEBUG:boto:auth_path=/ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.793 DEBUG:boto:Method: GET 2024-03-04T17:36:47.793 DEBUG:boto:Path: /ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.793 DEBUG:boto:Data: 2024-03-04T17:36:47.793 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.793 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.793 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.793 DEBUG:boto:Params: {} 2024-03-04T17:36:47.793 DEBUG:boto:Token: None 2024-03-04T17:36:47.793 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o?versionId=null 2024-03-04T17:36:47.793 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gB4l/80RpSE0WyAeBdPKp/ZInH8= 2024-03-04T17:36:47.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gB4l/80RpSE0WyAeBdPKp/ZInH8=', 'Content-Length': '0'} 2024-03-04T17:36:47.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d1e9dc3159711b8-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.797 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.797 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.797 DEBUG:boto:path=/ppfahn-10/?prefix=.o 2024-03-04T17:36:47.797 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o 2024-03-04T17:36:47.797 DEBUG:boto:Method: GET 2024-03-04T17:36:47.797 DEBUG:boto:Path: /ppfahn-10/?prefix=.o 2024-03-04T17:36:47.797 DEBUG:boto:Data: 2024-03-04T17:36:47.797 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.797 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.798 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.798 DEBUG:boto:Params: {} 2024-03-04T17:36:47.798 DEBUG:boto:Token: None 2024-03-04T17:36:47.798 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.798 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012f6d14e4874014b-0065e606af-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.803 DEBUG:boto:b'ppfahn-10.o1000false.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.803 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.803 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.803 DEBUG:boto:path=/ppfahn-10/?prefix=.o 2024-03-04T17:36:47.803 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o 2024-03-04T17:36:47.804 DEBUG:boto:Method: GET 2024-03-04T17:36:47.804 DEBUG:boto:Path: /ppfahn-10/?prefix=.o 2024-03-04T17:36:47.804 DEBUG:boto:Data: 2024-03-04T17:36:47.804 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.804 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.804 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.804 DEBUG:boto:Params: {} 2024-03-04T17:36:47.804 DEBUG:boto:Token: None 2024-03-04T17:36:47.804 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.804 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000911a5355ea43793d-0065e606af-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.810 DEBUG:boto:b'ppfahn-10.o1000false.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.810 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:36:47.810 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:36:47.810 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.810 DEBUG:boto:Path: /ppfahn-10/.o 2024-03-04T17:36:47.810 DEBUG:boto:Data: 2024-03-04T17:36:47.811 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.811 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.811 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.811 DEBUG:boto:Params: {} 2024-03-04T17:36:47.811 DEBUG:boto:Token: None 2024-03-04T17:36:47.811 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o 2024-03-04T17:36:47.811 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oQFQZJoXPFgryjNDWtTbFbY7bqI= 2024-03-04T17:36:47.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oQFQZJoXPFgryjNDWtTbFbY7bqI=', 'Content-Length': '0'} 2024-03-04T17:36:47.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f0e2a043999dae6-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.815 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:36:47.815 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:36:47.815 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.815 DEBUG:boto:Path: /ppfahn-10/.o 2024-03-04T17:36:47.815 DEBUG:boto:Data: 2024-03-04T17:36:47.815 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.815 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.815 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.815 DEBUG:boto:Params: {} 2024-03-04T17:36:47.815 DEBUG:boto:Token: None 2024-03-04T17:36:47.816 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o 2024-03-04T17:36:47.816 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oQFQZJoXPFgryjNDWtTbFbY7bqI= 2024-03-04T17:36:47.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oQFQZJoXPFgryjNDWtTbFbY7bqI=', 'Content-Length': '0'} 2024-03-04T17:36:47.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000137f89e4666e1f33-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:36:47.819 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:36:47.819 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:36:47.819 DEBUG:boto:Method: GET 2024-03-04T17:36:47.819 DEBUG:boto:Path: /ppfahn-10/.o 2024-03-04T17:36:47.819 DEBUG:boto:Data: 2024-03-04T17:36:47.819 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.819 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.819 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.819 DEBUG:boto:Params: {} 2024-03-04T17:36:47.820 DEBUG:boto:Token: None 2024-03-04T17:36:47.820 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o 2024-03-04T17:36:47.820 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qtp3aVBRzHrGr2GDkdz7gllj5vc= 2024-03-04T17:36:47.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qtp3aVBRzHrGr2GDkdz7gllj5vc=', 'Content-Length': '0'} 2024-03-04T17:36:47.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8e116149f60f27e-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.864 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:36:47.864 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:36:47.865 DEBUG:boto:Method: GET 2024-03-04T17:36:47.865 DEBUG:boto:Path: /ppfahn-10/.o 2024-03-04T17:36:47.865 DEBUG:boto:Data: 2024-03-04T17:36:47.865 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.865 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.865 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.865 DEBUG:boto:Params: {} 2024-03-04T17:36:47.865 DEBUG:boto:Token: None 2024-03-04T17:36:47.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o 2024-03-04T17:36:47.865 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qtp3aVBRzHrGr2GDkdz7gllj5vc= 2024-03-04T17:36:47.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qtp3aVBRzHrGr2GDkdz7gllj5vc=', 'Content-Length': '0'} 2024-03-04T17:36:47.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d5ea9ccce4358cf-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.869 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.869 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.869 DEBUG:boto:path=/ppfahn-10/?prefix=.o 2024-03-04T17:36:47.869 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o 2024-03-04T17:36:47.869 DEBUG:boto:Method: GET 2024-03-04T17:36:47.869 DEBUG:boto:Path: /ppfahn-10/?prefix=.o 2024-03-04T17:36:47.869 DEBUG:boto:Data: 2024-03-04T17:36:47.869 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.869 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.869 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.870 DEBUG:boto:Params: {} 2024-03-04T17:36:47.870 DEBUG:boto:Token: None 2024-03-04T17:36:47.870 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.870 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb350ba8c24fa711-0065e606af-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.874 DEBUG:boto:b'ppfahn-10.o1000false.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.875 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.875 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.875 DEBUG:boto:path=/ppfahn-10/?prefix=.o 2024-03-04T17:36:47.875 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o 2024-03-04T17:36:47.875 DEBUG:boto:Method: GET 2024-03-04T17:36:47.875 DEBUG:boto:Path: /ppfahn-10/?prefix=.o 2024-03-04T17:36:47.875 DEBUG:boto:Data: 2024-03-04T17:36:47.875 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.876 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.876 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.876 DEBUG:boto:Params: {} 2024-03-04T17:36:47.876 DEBUG:boto:Token: None 2024-03-04T17:36:47.876 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.876 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d1a758353ca9eca-0065e606af-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.881 DEBUG:boto:b'ppfahn-10.o1000false.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:36:47.881 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:36:47.881 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:36:47.881 DEBUG:boto:path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.881 DEBUG:boto:auth_path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.881 DEBUG:boto:Method: GET 2024-03-04T17:36:47.882 DEBUG:boto:Path: /ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.882 DEBUG:boto:Data: 2024-03-04T17:36:47.882 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.882 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.882 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.882 DEBUG:boto:Params: {} 2024-03-04T17:36:47.882 DEBUG:boto:Token: None 2024-03-04T17:36:47.882 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.882 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mWDsVNvUe1hTBvwBE7vVjdLkcFo= 2024-03-04T17:36:47.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mWDsVNvUe1hTBvwBE7vVjdLkcFo=', 'Content-Length': '0'} 2024-03-04T17:36:47.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bc030951e6c52e5-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.886 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:36:47.886 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:36:47.886 DEBUG:boto:path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.886 DEBUG:boto:auth_path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.886 DEBUG:boto:Method: GET 2024-03-04T17:36:47.886 DEBUG:boto:Path: /ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.886 DEBUG:boto:Data: 2024-03-04T17:36:47.886 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.886 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.886 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.886 DEBUG:boto:Params: {} 2024-03-04T17:36:47.887 DEBUG:boto:Token: None 2024-03-04T17:36:47.887 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.887 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mWDsVNvUe1hTBvwBE7vVjdLkcFo= 2024-03-04T17:36:47.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mWDsVNvUe1hTBvwBE7vVjdLkcFo=', 'Content-Length': '0'} 2024-03-04T17:36:47.890 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fed5a86c7ef60c64-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.890 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:36:47.890 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:36:47.890 DEBUG:boto:path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.890 DEBUG:boto:auth_path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.890 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.890 DEBUG:boto:Path: /ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.891 DEBUG:boto:Data: 2024-03-04T17:36:47.891 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.891 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.891 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.891 DEBUG:boto:Params: {} 2024-03-04T17:36:47.891 DEBUG:boto:Token: None 2024-03-04T17:36:47.891 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.891 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8vmnhSB4DXmM1F5GYjKGG5/BsoE= 2024-03-04T17:36:47.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8vmnhSB4DXmM1F5GYjKGG5/BsoE=', 'Content-Length': '0'} 2024-03-04T17:36:47.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004afb40fb3ee28d0f-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.894 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:36:47.894 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:36:47.894 DEBUG:boto:path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.895 DEBUG:boto:auth_path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.895 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.895 DEBUG:boto:Path: /ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.895 DEBUG:boto:Data: 2024-03-04T17:36:47.895 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.895 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.895 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.895 DEBUG:boto:Params: {} 2024-03-04T17:36:47.895 DEBUG:boto:Token: None 2024-03-04T17:36:47.895 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.895 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8vmnhSB4DXmM1F5GYjKGG5/BsoE= 2024-03-04T17:36:47.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8vmnhSB4DXmM1F5GYjKGG5/BsoE=', 'Content-Length': '0'} 2024-03-04T17:36:47.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bebe362178b19b06-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:36:47.899 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:36:47.899 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:36:47.899 DEBUG:boto:path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.899 DEBUG:boto:auth_path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.899 DEBUG:boto:Method: GET 2024-03-04T17:36:47.899 DEBUG:boto:Path: /ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.899 DEBUG:boto:Data: 2024-03-04T17:36:47.899 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.899 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.899 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.899 DEBUG:boto:Params: {} 2024-03-04T17:36:47.900 DEBUG:boto:Token: None 2024-03-04T17:36:47.900 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.900 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mWDsVNvUe1hTBvwBE7vVjdLkcFo= 2024-03-04T17:36:47.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mWDsVNvUe1hTBvwBE7vVjdLkcFo=', 'Content-Length': '0'} 2024-03-04T17:36:47.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6aaf1a4e714a4f7-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.944 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:36:47.944 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:36:47.944 DEBUG:boto:path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.945 DEBUG:boto:auth_path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.945 DEBUG:boto:Method: GET 2024-03-04T17:36:47.945 DEBUG:boto:Path: /ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.945 DEBUG:boto:Data: 2024-03-04T17:36:47.945 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.945 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.945 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.945 DEBUG:boto:Params: {} 2024-03-04T17:36:47.945 DEBUG:boto:Token: None 2024-03-04T17:36:47.946 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o.?versionId=null 2024-03-04T17:36:47.946 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mWDsVNvUe1hTBvwBE7vVjdLkcFo= 2024-03-04T17:36:47.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mWDsVNvUe1hTBvwBE7vVjdLkcFo=', 'Content-Length': '0'} 2024-03-04T17:36:47.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b9661ee2d972174-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.949 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.949 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.949 DEBUG:boto:path=/ppfahn-10/?prefix=.o. 2024-03-04T17:36:47.949 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o. 2024-03-04T17:36:47.949 DEBUG:boto:Method: GET 2024-03-04T17:36:47.950 DEBUG:boto:Path: /ppfahn-10/?prefix=.o. 2024-03-04T17:36:47.950 DEBUG:boto:Data: 2024-03-04T17:36:47.950 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.950 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.950 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.950 DEBUG:boto:Params: {} 2024-03-04T17:36:47.951 DEBUG:boto:Token: None 2024-03-04T17:36:47.951 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.951 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003aa7d854d839cda-0065e606af-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.956 DEBUG:boto:b'ppfahn-10.o.1000false.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.957 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:47.957 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:47.957 DEBUG:boto:path=/ppfahn-10/?prefix=.o. 2024-03-04T17:36:47.958 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o. 2024-03-04T17:36:47.958 DEBUG:boto:Method: GET 2024-03-04T17:36:47.958 DEBUG:boto:Path: /ppfahn-10/?prefix=.o. 2024-03-04T17:36:47.958 DEBUG:boto:Data: 2024-03-04T17:36:47.958 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.958 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.959 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.959 DEBUG:boto:Params: {} 2024-03-04T17:36:47.959 DEBUG:boto:Token: None 2024-03-04T17:36:47.959 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/ 2024-03-04T17:36:47.960 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco= 2024-03-04T17:36:47.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glE9ggULVCPFpwiZrEwOj2pNoco=', 'Content-Length': '0'} 2024-03-04T17:36:47.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc8e18e937d68530-0065e606af-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.964 DEBUG:boto:b'ppfahn-10.o.1000false.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:47.965 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:36:47.965 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:36:47.965 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.965 DEBUG:boto:Path: /ppfahn-10/.o. 2024-03-04T17:36:47.966 DEBUG:boto:Data: 2024-03-04T17:36:47.966 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.966 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.966 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.966 DEBUG:boto:Params: {} 2024-03-04T17:36:47.967 DEBUG:boto:Token: None 2024-03-04T17:36:47.967 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o. 2024-03-04T17:36:47.967 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TChOOs5+cHR6v4OSrrv9d80ZaHM= 2024-03-04T17:36:47.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TChOOs5+cHR6v4OSrrv9d80ZaHM=', 'Content-Length': '0'} 2024-03-04T17:36:47.970 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0dfdc824e8f99e7-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.971 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:36:47.971 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:36:47.971 DEBUG:boto:Method: HEAD 2024-03-04T17:36:47.971 DEBUG:boto:Path: /ppfahn-10/.o. 2024-03-04T17:36:47.972 DEBUG:boto:Data: 2024-03-04T17:36:47.972 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.972 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.972 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.972 DEBUG:boto:Params: {} 2024-03-04T17:36:47.973 DEBUG:boto:Token: None 2024-03-04T17:36:47.973 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o. 2024-03-04T17:36:47.973 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TChOOs5+cHR6v4OSrrv9d80ZaHM= 2024-03-04T17:36:47.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TChOOs5+cHR6v4OSrrv9d80ZaHM=', 'Content-Length': '0'} 2024-03-04T17:36:47.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddb7d0a926ec8169-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:36:47.977 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:36:47.977 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:36:47.977 DEBUG:boto:Method: GET 2024-03-04T17:36:47.977 DEBUG:boto:Path: /ppfahn-10/.o. 2024-03-04T17:36:47.978 DEBUG:boto:Data: 2024-03-04T17:36:47.978 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.978 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:47.978 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.978 DEBUG:boto:Params: {} 2024-03-04T17:36:47.978 DEBUG:boto:Token: None 2024-03-04T17:36:47.979 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o. 2024-03-04T17:36:47.979 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BKD43hPxt+V6C6kRgAeOxYMHVQY= 2024-03-04T17:36:47.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BKD43hPxt+V6C6kRgAeOxYMHVQY=', 'Content-Length': '0'} 2024-03-04T17:36:47.982 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bab1fea90cea480-0065e606af-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:47.982 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:36:47.982 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:36:47.982 DEBUG:boto:Method: GET 2024-03-04T17:36:47.982 DEBUG:boto:Path: /ppfahn-10/.o. 2024-03-04T17:36:47.982 DEBUG:boto:Data: 2024-03-04T17:36:47.982 DEBUG:boto:Headers: {} 2024-03-04T17:36:47.983 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:47.983 DEBUG:boto:Port: 8000 2024-03-04T17:36:47.983 DEBUG:boto:Params: {} 2024-03-04T17:36:47.983 DEBUG:boto:Token: None 2024-03-04T17:36:47.983 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:47 GMT /ppfahn-10/.o. 2024-03-04T17:36:47.983 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BKD43hPxt+V6C6kRgAeOxYMHVQY= 2024-03-04T17:36:47.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BKD43hPxt+V6C6kRgAeOxYMHVQY=', 'Content-Length': '0'} 2024-03-04T17:36:47.986 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000347f6bef61117ae0-0065e606af-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.028 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.028 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.028 DEBUG:boto:path=/ppfahn-10/?prefix=.o. 2024-03-04T17:36:48.028 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o. 2024-03-04T17:36:48.029 DEBUG:boto:Method: GET 2024-03-04T17:36:48.029 DEBUG:boto:Path: /ppfahn-10/?prefix=.o. 2024-03-04T17:36:48.029 DEBUG:boto:Data: 2024-03-04T17:36:48.029 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.029 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.029 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.029 DEBUG:boto:Params: {} 2024-03-04T17:36:48.029 DEBUG:boto:Token: None 2024-03-04T17:36:48.029 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.029 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a0cecbaa4e2889d-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.034 DEBUG:boto:b'ppfahn-10.o.1000false.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.034 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.034 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.034 DEBUG:boto:path=/ppfahn-10/?prefix=.o. 2024-03-04T17:36:48.035 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o. 2024-03-04T17:36:48.035 DEBUG:boto:Method: GET 2024-03-04T17:36:48.035 DEBUG:boto:Path: /ppfahn-10/?prefix=.o. 2024-03-04T17:36:48.035 DEBUG:boto:Data: 2024-03-04T17:36:48.035 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.035 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.036 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.036 DEBUG:boto:Params: {} 2024-03-04T17:36:48.036 DEBUG:boto:Token: None 2024-03-04T17:36:48.036 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.036 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bff3015cd5f3cdf6-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.041 DEBUG:boto:b'ppfahn-10.o.1000false.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:36:48.042 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:36:48.042 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:36:48.042 DEBUG:boto:path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.043 DEBUG:boto:auth_path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.043 DEBUG:boto:Method: GET 2024-03-04T17:36:48.043 DEBUG:boto:Path: /ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.043 DEBUG:boto:Data: 2024-03-04T17:36:48.043 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.043 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.043 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.044 DEBUG:boto:Params: {} 2024-03-04T17:36:48.044 DEBUG:boto:Token: None 2024-03-04T17:36:48.044 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.044 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3wxsLcdwyhDMGCT9oyinLG5CwG8= 2024-03-04T17:36:48.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3wxsLcdwyhDMGCT9oyinLG5CwG8=', 'Content-Length': '0'} 2024-03-04T17:36:48.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b48856c0f3d09d28-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.088 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:36:48.088 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:36:48.088 DEBUG:boto:path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.088 DEBUG:boto:auth_path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.089 DEBUG:boto:Method: GET 2024-03-04T17:36:48.089 DEBUG:boto:Path: /ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.089 DEBUG:boto:Data: 2024-03-04T17:36:48.089 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.089 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.089 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.089 DEBUG:boto:Params: {} 2024-03-04T17:36:48.089 DEBUG:boto:Token: None 2024-03-04T17:36:48.089 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.090 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3wxsLcdwyhDMGCT9oyinLG5CwG8= 2024-03-04T17:36:48.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3wxsLcdwyhDMGCT9oyinLG5CwG8=', 'Content-Length': '0'} 2024-03-04T17:36:48.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d2687ed24b232b4-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.093 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:36:48.093 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:36:48.093 DEBUG:boto:path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.093 DEBUG:boto:auth_path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.094 DEBUG:boto:Method: HEAD 2024-03-04T17:36:48.094 DEBUG:boto:Path: /ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.094 DEBUG:boto:Data: 2024-03-04T17:36:48.094 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.094 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.094 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.094 DEBUG:boto:Params: {} 2024-03-04T17:36:48.094 DEBUG:boto:Token: None 2024-03-04T17:36:48.094 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.094 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rffav3rPjb2Qbrn67EOm8LmDhk4= 2024-03-04T17:36:48.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rffav3rPjb2Qbrn67EOm8LmDhk4=', 'Content-Length': '0'} 2024-03-04T17:36:48.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d39f38d9a647ddd-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.098 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:36:48.098 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:36:48.098 DEBUG:boto:path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.098 DEBUG:boto:auth_path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.098 DEBUG:boto:Method: HEAD 2024-03-04T17:36:48.098 DEBUG:boto:Path: /ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.098 DEBUG:boto:Data: 2024-03-04T17:36:48.098 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.098 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.098 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.098 DEBUG:boto:Params: {} 2024-03-04T17:36:48.099 DEBUG:boto:Token: None 2024-03-04T17:36:48.099 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.099 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rffav3rPjb2Qbrn67EOm8LmDhk4= 2024-03-04T17:36:48.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rffav3rPjb2Qbrn67EOm8LmDhk4=', 'Content-Length': '0'} 2024-03-04T17:36:48.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cfcd31bc11cb97a-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:36:48.102 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:36:48.102 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:36:48.102 DEBUG:boto:path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.102 DEBUG:boto:auth_path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.102 DEBUG:boto:Method: GET 2024-03-04T17:36:48.103 DEBUG:boto:Path: /ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.103 DEBUG:boto:Data: 2024-03-04T17:36:48.103 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.103 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.103 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.103 DEBUG:boto:Params: {} 2024-03-04T17:36:48.103 DEBUG:boto:Token: None 2024-03-04T17:36:48.103 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.103 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3wxsLcdwyhDMGCT9oyinLG5CwG8= 2024-03-04T17:36:48.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3wxsLcdwyhDMGCT9oyinLG5CwG8=', 'Content-Length': '0'} 2024-03-04T17:36:48.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b78de666ac0739c-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.106 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:36:48.106 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:36:48.107 DEBUG:boto:path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.107 DEBUG:boto:auth_path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.107 DEBUG:boto:Method: GET 2024-03-04T17:36:48.107 DEBUG:boto:Path: /ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.107 DEBUG:boto:Data: 2024-03-04T17:36:48.107 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.107 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.107 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.107 DEBUG:boto:Params: {} 2024-03-04T17:36:48.107 DEBUG:boto:Token: None 2024-03-04T17:36:48.107 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/%3A?versionId=null 2024-03-04T17:36:48.108 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3wxsLcdwyhDMGCT9oyinLG5CwG8= 2024-03-04T17:36:48.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3wxsLcdwyhDMGCT9oyinLG5CwG8=', 'Content-Length': '0'} 2024-03-04T17:36:48.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f45d4bf4e0da209c-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.152 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.152 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.152 DEBUG:boto:path=/ppfahn-10/?prefix=%3A 2024-03-04T17:36:48.153 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%3A 2024-03-04T17:36:48.153 DEBUG:boto:Method: GET 2024-03-04T17:36:48.153 DEBUG:boto:Path: /ppfahn-10/?prefix=%3A 2024-03-04T17:36:48.153 DEBUG:boto:Data: 2024-03-04T17:36:48.153 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.153 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.153 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.153 DEBUG:boto:Params: {} 2024-03-04T17:36:48.153 DEBUG:boto:Token: None 2024-03-04T17:36:48.153 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.154 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e04bdbe1a77b502-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.158 DEBUG:boto:b'ppfahn-10:1000false:2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.159 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.159 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.159 DEBUG:boto:path=/ppfahn-10/?prefix=%3A 2024-03-04T17:36:48.159 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%3A 2024-03-04T17:36:48.159 DEBUG:boto:Method: GET 2024-03-04T17:36:48.159 DEBUG:boto:Path: /ppfahn-10/?prefix=%3A 2024-03-04T17:36:48.159 DEBUG:boto:Data: 2024-03-04T17:36:48.160 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.160 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.160 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.160 DEBUG:boto:Params: {} 2024-03-04T17:36:48.160 DEBUG:boto:Token: None 2024-03-04T17:36:48.160 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.160 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c6ff7fda44518a7-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.165 DEBUG:boto:b'ppfahn-10:1000false:2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.165 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:36:48.165 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:36:48.166 DEBUG:boto:Method: HEAD 2024-03-04T17:36:48.166 DEBUG:boto:Path: /ppfahn-10/%3A 2024-03-04T17:36:48.166 DEBUG:boto:Data: 2024-03-04T17:36:48.166 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.166 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.166 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.166 DEBUG:boto:Params: {} 2024-03-04T17:36:48.166 DEBUG:boto:Token: None 2024-03-04T17:36:48.166 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/%3A 2024-03-04T17:36:48.167 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:93HZpJAmXy+x54R+Cf0KG4aGWJk= 2024-03-04T17:36:48.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:93HZpJAmXy+x54R+Cf0KG4aGWJk=', 'Content-Length': '0'} 2024-03-04T17:36:48.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a09e9a8096eb1c96-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.170 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:36:48.170 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:36:48.170 DEBUG:boto:Method: HEAD 2024-03-04T17:36:48.170 DEBUG:boto:Path: /ppfahn-10/%3A 2024-03-04T17:36:48.170 DEBUG:boto:Data: 2024-03-04T17:36:48.170 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.170 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.170 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.170 DEBUG:boto:Params: {} 2024-03-04T17:36:48.170 DEBUG:boto:Token: None 2024-03-04T17:36:48.171 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/%3A 2024-03-04T17:36:48.171 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:93HZpJAmXy+x54R+Cf0KG4aGWJk= 2024-03-04T17:36:48.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:93HZpJAmXy+x54R+Cf0KG4aGWJk=', 'Content-Length': '0'} 2024-03-04T17:36:48.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022553ec50bc39a75-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:36:48.174 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:36:48.174 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:36:48.174 DEBUG:boto:Method: GET 2024-03-04T17:36:48.174 DEBUG:boto:Path: /ppfahn-10/%3A 2024-03-04T17:36:48.174 DEBUG:boto:Data: 2024-03-04T17:36:48.174 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.174 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.175 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.175 DEBUG:boto:Params: {} 2024-03-04T17:36:48.175 DEBUG:boto:Token: None 2024-03-04T17:36:48.175 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/%3A 2024-03-04T17:36:48.175 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RcMkBNX+lVUS9thF3FD3d8r1OMY= 2024-03-04T17:36:48.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RcMkBNX+lVUS9thF3FD3d8r1OMY=', 'Content-Length': '0'} 2024-03-04T17:36:48.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e556e03ebfe08af1-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.220 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:36:48.220 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:36:48.220 DEBUG:boto:Method: GET 2024-03-04T17:36:48.220 DEBUG:boto:Path: /ppfahn-10/%3A 2024-03-04T17:36:48.221 DEBUG:boto:Data: 2024-03-04T17:36:48.221 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.221 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.221 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.221 DEBUG:boto:Params: {} 2024-03-04T17:36:48.221 DEBUG:boto:Token: None 2024-03-04T17:36:48.221 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/%3A 2024-03-04T17:36:48.221 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RcMkBNX+lVUS9thF3FD3d8r1OMY= 2024-03-04T17:36:48.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RcMkBNX+lVUS9thF3FD3d8r1OMY=', 'Content-Length': '0'} 2024-03-04T17:36:48.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5401ef054611c30-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.225 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.225 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.226 DEBUG:boto:path=/ppfahn-10/?prefix=%3A 2024-03-04T17:36:48.226 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%3A 2024-03-04T17:36:48.226 DEBUG:boto:Method: GET 2024-03-04T17:36:48.226 DEBUG:boto:Path: /ppfahn-10/?prefix=%3A 2024-03-04T17:36:48.226 DEBUG:boto:Data: 2024-03-04T17:36:48.226 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.226 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.226 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.227 DEBUG:boto:Params: {} 2024-03-04T17:36:48.227 DEBUG:boto:Token: None 2024-03-04T17:36:48.227 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.227 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee2b8bf2a1375208-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.232 DEBUG:boto:b'ppfahn-10:1000false:2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.232 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.233 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.233 DEBUG:boto:path=/ppfahn-10/?prefix=%3A 2024-03-04T17:36:48.233 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%3A 2024-03-04T17:36:48.233 DEBUG:boto:Method: GET 2024-03-04T17:36:48.233 DEBUG:boto:Path: /ppfahn-10/?prefix=%3A 2024-03-04T17:36:48.233 DEBUG:boto:Data: 2024-03-04T17:36:48.233 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.233 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.233 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.233 DEBUG:boto:Params: {} 2024-03-04T17:36:48.233 DEBUG:boto:Token: None 2024-03-04T17:36:48.234 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.234 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf1c3e2531f9f88f-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.280 DEBUG:boto:b'ppfahn-10:1000false:2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:36:48.281 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:36:48.281 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:36:48.281 DEBUG:boto:path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.281 DEBUG:boto:auth_path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.281 DEBUG:boto:Method: GET 2024-03-04T17:36:48.281 DEBUG:boto:Path: /ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.281 DEBUG:boto:Data: 2024-03-04T17:36:48.281 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.282 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.282 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.282 DEBUG:boto:Params: {} 2024-03-04T17:36:48.282 DEBUG:boto:Token: None 2024-03-04T17:36:48.282 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.282 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uAhE6L4CeJPsNyvWsukWy09p2QI= 2024-03-04T17:36:48.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uAhE6L4CeJPsNyvWsukWy09p2QI=', 'Content-Length': '0'} 2024-03-04T17:36:48.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f6d8e4ba57d6e73-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.286 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:36:48.286 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:36:48.286 DEBUG:boto:path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.286 DEBUG:boto:auth_path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.286 DEBUG:boto:Method: GET 2024-03-04T17:36:48.286 DEBUG:boto:Path: /ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.286 DEBUG:boto:Data: 2024-03-04T17:36:48.286 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.287 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.287 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.287 DEBUG:boto:Params: {} 2024-03-04T17:36:48.287 DEBUG:boto:Token: None 2024-03-04T17:36:48.287 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.287 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uAhE6L4CeJPsNyvWsukWy09p2QI= 2024-03-04T17:36:48.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uAhE6L4CeJPsNyvWsukWy09p2QI=', 'Content-Length': '0'} 2024-03-04T17:36:48.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000698b652857106a10-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.291 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:36:48.291 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:36:48.291 DEBUG:boto:path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.291 DEBUG:boto:auth_path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.291 DEBUG:boto:Method: HEAD 2024-03-04T17:36:48.291 DEBUG:boto:Path: /ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.291 DEBUG:boto:Data: 2024-03-04T17:36:48.292 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.292 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.292 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.292 DEBUG:boto:Params: {} 2024-03-04T17:36:48.292 DEBUG:boto:Token: None 2024-03-04T17:36:48.292 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.292 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8L/7GkYReSs+YtlbA/P5DdBi27A= 2024-03-04T17:36:48.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8L/7GkYReSs+YtlbA/P5DdBi27A=', 'Content-Length': '0'} 2024-03-04T17:36:48.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dba4d98858fb2cbe-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.296 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:36:48.296 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:36:48.296 DEBUG:boto:path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.297 DEBUG:boto:auth_path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.297 DEBUG:boto:Method: HEAD 2024-03-04T17:36:48.297 DEBUG:boto:Path: /ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.297 DEBUG:boto:Data: 2024-03-04T17:36:48.297 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.297 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.297 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.297 DEBUG:boto:Params: {} 2024-03-04T17:36:48.297 DEBUG:boto:Token: None 2024-03-04T17:36:48.297 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.298 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8L/7GkYReSs+YtlbA/P5DdBi27A= 2024-03-04T17:36:48.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8L/7GkYReSs+YtlbA/P5DdBi27A=', 'Content-Length': '0'} 2024-03-04T17:36:48.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000368127725885b9cd-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:36:48.301 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:36:48.302 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:36:48.302 DEBUG:boto:path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.302 DEBUG:boto:auth_path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.302 DEBUG:boto:Method: GET 2024-03-04T17:36:48.302 DEBUG:boto:Path: /ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.302 DEBUG:boto:Data: 2024-03-04T17:36:48.302 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.302 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.302 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.302 DEBUG:boto:Params: {} 2024-03-04T17:36:48.302 DEBUG:boto:Token: None 2024-03-04T17:36:48.303 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.303 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uAhE6L4CeJPsNyvWsukWy09p2QI= 2024-03-04T17:36:48.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uAhE6L4CeJPsNyvWsukWy09p2QI=', 'Content-Length': '0'} 2024-03-04T17:36:48.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7f80ded7cd84f3a-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.306 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:36:48.306 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:36:48.306 DEBUG:boto:path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.306 DEBUG:boto:auth_path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.306 DEBUG:boto:Method: GET 2024-03-04T17:36:48.306 DEBUG:boto:Path: /ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.307 DEBUG:boto:Data: 2024-03-04T17:36:48.307 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.307 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.307 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.307 DEBUG:boto:Params: {} 2024-03-04T17:36:48.307 DEBUG:boto:Token: None 2024-03-04T17:36:48.307 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/_myobj?versionId=null 2024-03-04T17:36:48.307 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uAhE6L4CeJPsNyvWsukWy09p2QI= 2024-03-04T17:36:48.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uAhE6L4CeJPsNyvWsukWy09p2QI=', 'Content-Length': '0'} 2024-03-04T17:36:48.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ed34ac9085b2934-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.352 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.352 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.353 DEBUG:boto:path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:36:48.353 DEBUG:boto:auth_path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:36:48.353 DEBUG:boto:Method: GET 2024-03-04T17:36:48.353 DEBUG:boto:Path: /ppfahn-10/?prefix=_myobj 2024-03-04T17:36:48.353 DEBUG:boto:Data: 2024-03-04T17:36:48.353 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.353 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.353 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.353 DEBUG:boto:Params: {} 2024-03-04T17:36:48.353 DEBUG:boto:Token: None 2024-03-04T17:36:48.354 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.354 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e82e6a3f5dccb380-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.358 DEBUG:boto:b'ppfahn-10_myobj1000false_myobj2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.358 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.359 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.359 DEBUG:boto:path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:36:48.359 DEBUG:boto:auth_path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:36:48.359 DEBUG:boto:Method: GET 2024-03-04T17:36:48.359 DEBUG:boto:Path: /ppfahn-10/?prefix=_myobj 2024-03-04T17:36:48.359 DEBUG:boto:Data: 2024-03-04T17:36:48.359 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.359 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.359 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.359 DEBUG:boto:Params: {} 2024-03-04T17:36:48.359 DEBUG:boto:Token: None 2024-03-04T17:36:48.360 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.360 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c00367753ef01c43-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.365 DEBUG:boto:b'ppfahn-10_myobj1000false_myobj2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.365 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:36:48.365 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:36:48.365 DEBUG:boto:Method: HEAD 2024-03-04T17:36:48.365 DEBUG:boto:Path: /ppfahn-10/_myobj 2024-03-04T17:36:48.366 DEBUG:boto:Data: 2024-03-04T17:36:48.366 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.366 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.366 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.366 DEBUG:boto:Params: {} 2024-03-04T17:36:48.366 DEBUG:boto:Token: None 2024-03-04T17:36:48.366 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/_myobj 2024-03-04T17:36:48.366 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vwo7I62V4qvQqN92LGrAGSvSdHw= 2024-03-04T17:36:48.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vwo7I62V4qvQqN92LGrAGSvSdHw=', 'Content-Length': '0'} 2024-03-04T17:36:48.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c211810e8e1d113e-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.369 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:36:48.369 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:36:48.369 DEBUG:boto:Method: HEAD 2024-03-04T17:36:48.369 DEBUG:boto:Path: /ppfahn-10/_myobj 2024-03-04T17:36:48.370 DEBUG:boto:Data: 2024-03-04T17:36:48.370 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.370 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.370 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.370 DEBUG:boto:Params: {} 2024-03-04T17:36:48.370 DEBUG:boto:Token: None 2024-03-04T17:36:48.370 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/_myobj 2024-03-04T17:36:48.370 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vwo7I62V4qvQqN92LGrAGSvSdHw= 2024-03-04T17:36:48.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vwo7I62V4qvQqN92LGrAGSvSdHw=', 'Content-Length': '0'} 2024-03-04T17:36:48.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bf358c0fac4c5e2-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:36:48.374 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:36:48.374 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:36:48.374 DEBUG:boto:Method: GET 2024-03-04T17:36:48.374 DEBUG:boto:Path: /ppfahn-10/_myobj 2024-03-04T17:36:48.374 DEBUG:boto:Data: 2024-03-04T17:36:48.374 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.374 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.375 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.375 DEBUG:boto:Params: {} 2024-03-04T17:36:48.375 DEBUG:boto:Token: None 2024-03-04T17:36:48.375 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/_myobj 2024-03-04T17:36:48.375 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xSIZHkJGCiFesMf5leB7jekbyIM= 2024-03-04T17:36:48.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xSIZHkJGCiFesMf5leB7jekbyIM=', 'Content-Length': '0'} 2024-03-04T17:36:48.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ae552601f718ea2-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.420 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:36:48.420 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:36:48.420 DEBUG:boto:Method: GET 2024-03-04T17:36:48.421 DEBUG:boto:Path: /ppfahn-10/_myobj 2024-03-04T17:36:48.421 DEBUG:boto:Data: 2024-03-04T17:36:48.421 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.421 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.421 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.421 DEBUG:boto:Params: {} 2024-03-04T17:36:48.421 DEBUG:boto:Token: None 2024-03-04T17:36:48.421 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/_myobj 2024-03-04T17:36:48.422 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xSIZHkJGCiFesMf5leB7jekbyIM= 2024-03-04T17:36:48.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xSIZHkJGCiFesMf5leB7jekbyIM=', 'Content-Length': '0'} 2024-03-04T17:36:48.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090514c12d1701611-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.425 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.425 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.425 DEBUG:boto:path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:36:48.425 DEBUG:boto:auth_path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:36:48.425 DEBUG:boto:Method: GET 2024-03-04T17:36:48.426 DEBUG:boto:Path: /ppfahn-10/?prefix=_myobj 2024-03-04T17:36:48.426 DEBUG:boto:Data: 2024-03-04T17:36:48.426 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.426 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.426 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.426 DEBUG:boto:Params: {} 2024-03-04T17:36:48.426 DEBUG:boto:Token: None 2024-03-04T17:36:48.426 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.426 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd8cae5f67e8a96f-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.430 DEBUG:boto:b'ppfahn-10_myobj1000false_myobj2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.431 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.431 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.431 DEBUG:boto:path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:36:48.431 DEBUG:boto:auth_path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:36:48.431 DEBUG:boto:Method: GET 2024-03-04T17:36:48.432 DEBUG:boto:Path: /ppfahn-10/?prefix=_myobj 2024-03-04T17:36:48.432 DEBUG:boto:Data: 2024-03-04T17:36:48.432 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.432 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.432 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.432 DEBUG:boto:Params: {} 2024-03-04T17:36:48.432 DEBUG:boto:Token: None 2024-03-04T17:36:48.432 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.432 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059a8eb42db38acee-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.480 DEBUG:boto:b'ppfahn-10_myobj1000false_myobj2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:36:48.481 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:36:48.481 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:36:48.481 DEBUG:boto:path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.481 DEBUG:boto:auth_path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.481 DEBUG:boto:Method: GET 2024-03-04T17:36:48.481 DEBUG:boto:Path: /ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.481 DEBUG:boto:Data: 2024-03-04T17:36:48.481 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.481 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.481 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.482 DEBUG:boto:Params: {} 2024-03-04T17:36:48.482 DEBUG:boto:Token: None 2024-03-04T17:36:48.482 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.482 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uOIO3oxGCnUCipyfGwkVLp0AXSI= 2024-03-04T17:36:48.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uOIO3oxGCnUCipyfGwkVLp0AXSI=', 'Content-Length': '0'} 2024-03-04T17:36:48.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0f6728e23de7957-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.486 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:36:48.486 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:36:48.486 DEBUG:boto:path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.486 DEBUG:boto:auth_path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.486 DEBUG:boto:Method: GET 2024-03-04T17:36:48.486 DEBUG:boto:Path: /ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.486 DEBUG:boto:Data: 2024-03-04T17:36:48.486 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.486 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.486 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.486 DEBUG:boto:Params: {} 2024-03-04T17:36:48.487 DEBUG:boto:Token: None 2024-03-04T17:36:48.487 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.487 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uOIO3oxGCnUCipyfGwkVLp0AXSI= 2024-03-04T17:36:48.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uOIO3oxGCnUCipyfGwkVLp0AXSI=', 'Content-Length': '0'} 2024-03-04T17:36:48.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000623a9f2fe468ff3a-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.490 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:36:48.490 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:36:48.490 DEBUG:boto:path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.491 DEBUG:boto:auth_path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.491 DEBUG:boto:Method: HEAD 2024-03-04T17:36:48.491 DEBUG:boto:Path: /ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.491 DEBUG:boto:Data: 2024-03-04T17:36:48.491 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.491 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.491 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.491 DEBUG:boto:Params: {} 2024-03-04T17:36:48.491 DEBUG:boto:Token: None 2024-03-04T17:36:48.491 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.492 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K5bijxI4Rce1fyuY6xuXSLaYh1M= 2024-03-04T17:36:48.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K5bijxI4Rce1fyuY6xuXSLaYh1M=', 'Content-Length': '0'} 2024-03-04T17:36:48.494 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073f33d79fcc6f271-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.495 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:36:48.495 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:36:48.495 DEBUG:boto:path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.495 DEBUG:boto:auth_path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.495 DEBUG:boto:Method: HEAD 2024-03-04T17:36:48.495 DEBUG:boto:Path: /ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.495 DEBUG:boto:Data: 2024-03-04T17:36:48.495 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.495 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.495 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.495 DEBUG:boto:Params: {} 2024-03-04T17:36:48.496 DEBUG:boto:Token: None 2024-03-04T17:36:48.496 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.496 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K5bijxI4Rce1fyuY6xuXSLaYh1M= 2024-03-04T17:36:48.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K5bijxI4Rce1fyuY6xuXSLaYh1M=', 'Content-Length': '0'} 2024-03-04T17:36:48.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044c8c7fd20619348-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:36:48.499 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:36:48.499 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:36:48.499 DEBUG:boto:path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.499 DEBUG:boto:auth_path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.499 DEBUG:boto:Method: GET 2024-03-04T17:36:48.499 DEBUG:boto:Path: /ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.500 DEBUG:boto:Data: 2024-03-04T17:36:48.500 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.500 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.500 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.500 DEBUG:boto:Params: {} 2024-03-04T17:36:48.500 DEBUG:boto:Token: None 2024-03-04T17:36:48.500 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.500 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uOIO3oxGCnUCipyfGwkVLp0AXSI= 2024-03-04T17:36:48.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uOIO3oxGCnUCipyfGwkVLp0AXSI=', 'Content-Length': '0'} 2024-03-04T17:36:48.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef317e73f807b104-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.503 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:36:48.503 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:36:48.504 DEBUG:boto:path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.504 DEBUG:boto:auth_path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.504 DEBUG:boto:Method: GET 2024-03-04T17:36:48.504 DEBUG:boto:Path: /ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.504 DEBUG:boto:Data: 2024-03-04T17:36:48.504 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.504 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.504 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.504 DEBUG:boto:Params: {} 2024-03-04T17:36:48.504 DEBUG:boto:Token: None 2024-03-04T17:36:48.504 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/myobj?versionId=null 2024-03-04T17:36:48.505 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uOIO3oxGCnUCipyfGwkVLp0AXSI= 2024-03-04T17:36:48.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uOIO3oxGCnUCipyfGwkVLp0AXSI=', 'Content-Length': '0'} 2024-03-04T17:36:48.508 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acdc5a863a077b3b-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.508 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.508 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.508 DEBUG:boto:path=/ppfahn-10/?prefix=myobj 2024-03-04T17:36:48.508 DEBUG:boto:auth_path=/ppfahn-10/?prefix=myobj 2024-03-04T17:36:48.508 DEBUG:boto:Method: GET 2024-03-04T17:36:48.508 DEBUG:boto:Path: /ppfahn-10/?prefix=myobj 2024-03-04T17:36:48.508 DEBUG:boto:Data: 2024-03-04T17:36:48.509 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.509 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.509 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.509 DEBUG:boto:Params: {} 2024-03-04T17:36:48.509 DEBUG:boto:Token: None 2024-03-04T17:36:48.509 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.509 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000342d3b8f3a8e5ecc-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.514 DEBUG:boto:b'ppfahn-10myobj1000falsemyobj2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.514 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.514 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.514 DEBUG:boto:path=/ppfahn-10/?prefix=myobj 2024-03-04T17:36:48.515 DEBUG:boto:auth_path=/ppfahn-10/?prefix=myobj 2024-03-04T17:36:48.515 DEBUG:boto:Method: GET 2024-03-04T17:36:48.515 DEBUG:boto:Path: /ppfahn-10/?prefix=myobj 2024-03-04T17:36:48.515 DEBUG:boto:Data: 2024-03-04T17:36:48.515 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.515 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.515 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.515 DEBUG:boto:Params: {} 2024-03-04T17:36:48.515 DEBUG:boto:Token: None 2024-03-04T17:36:48.515 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.516 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c197bd409bc9bef-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.520 DEBUG:boto:b'ppfahn-10myobj1000falsemyobj2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.520 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:36:48.520 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:36:48.521 DEBUG:boto:Method: HEAD 2024-03-04T17:36:48.521 DEBUG:boto:Path: /ppfahn-10/myobj 2024-03-04T17:36:48.521 DEBUG:boto:Data: 2024-03-04T17:36:48.521 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.521 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.521 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.521 DEBUG:boto:Params: {} 2024-03-04T17:36:48.521 DEBUG:boto:Token: None 2024-03-04T17:36:48.521 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/myobj 2024-03-04T17:36:48.521 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jZQ/dL/UbK2wzdeMSseYrYMTe90= 2024-03-04T17:36:48.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jZQ/dL/UbK2wzdeMSseYrYMTe90=', 'Content-Length': '0'} 2024-03-04T17:36:48.524 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029af25302ca8fded-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.524 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:36:48.525 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:36:48.525 DEBUG:boto:Method: HEAD 2024-03-04T17:36:48.525 DEBUG:boto:Path: /ppfahn-10/myobj 2024-03-04T17:36:48.525 DEBUG:boto:Data: 2024-03-04T17:36:48.525 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.525 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.525 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.525 DEBUG:boto:Params: {} 2024-03-04T17:36:48.525 DEBUG:boto:Token: None 2024-03-04T17:36:48.525 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/myobj 2024-03-04T17:36:48.526 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jZQ/dL/UbK2wzdeMSseYrYMTe90= 2024-03-04T17:36:48.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jZQ/dL/UbK2wzdeMSseYrYMTe90=', 'Content-Length': '0'} 2024-03-04T17:36:48.528 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f51ff90637b01cf3-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:36:48.528 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:36:48.528 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:36:48.529 DEBUG:boto:Method: GET 2024-03-04T17:36:48.529 DEBUG:boto:Path: /ppfahn-10/myobj 2024-03-04T17:36:48.529 DEBUG:boto:Data: 2024-03-04T17:36:48.529 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.529 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.529 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.529 DEBUG:boto:Params: {} 2024-03-04T17:36:48.529 DEBUG:boto:Token: None 2024-03-04T17:36:48.529 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/myobj 2024-03-04T17:36:48.529 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:y94YNkGg2+jtQhQv2AtrL3CFBcY= 2024-03-04T17:36:48.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:y94YNkGg2+jtQhQv2AtrL3CFBcY=', 'Content-Length': '0'} 2024-03-04T17:36:48.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005726886268ef33f6-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.580 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:36:48.580 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:36:48.580 DEBUG:boto:Method: GET 2024-03-04T17:36:48.581 DEBUG:boto:Path: /ppfahn-10/myobj 2024-03-04T17:36:48.581 DEBUG:boto:Data: 2024-03-04T17:36:48.581 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.581 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.581 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.581 DEBUG:boto:Params: {} 2024-03-04T17:36:48.581 DEBUG:boto:Token: None 2024-03-04T17:36:48.581 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/myobj 2024-03-04T17:36:48.581 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:y94YNkGg2+jtQhQv2AtrL3CFBcY= 2024-03-04T17:36:48.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:y94YNkGg2+jtQhQv2AtrL3CFBcY=', 'Content-Length': '0'} 2024-03-04T17:36:48.584 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033b2c2f46b703080-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.585 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.585 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.585 DEBUG:boto:path=/ppfahn-10/?prefix=myobj 2024-03-04T17:36:48.585 DEBUG:boto:auth_path=/ppfahn-10/?prefix=myobj 2024-03-04T17:36:48.585 DEBUG:boto:Method: GET 2024-03-04T17:36:48.585 DEBUG:boto:Path: /ppfahn-10/?prefix=myobj 2024-03-04T17:36:48.585 DEBUG:boto:Data: 2024-03-04T17:36:48.585 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.585 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:36:48.585 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.586 DEBUG:boto:Params: {} 2024-03-04T17:36:48.586 DEBUG:boto:Token: None 2024-03-04T17:36:48.586 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.586 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f81cbf87ee55c5f2-0065e606b0-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.591 DEBUG:boto:b'ppfahn-10myobj1000falsemyobj2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.591 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:36:48.591 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:36:48.592 DEBUG:boto:path=/ppfahn-10/?prefix=myobj 2024-03-04T17:36:48.592 DEBUG:boto:auth_path=/ppfahn-10/?prefix=myobj 2024-03-04T17:36:48.592 DEBUG:boto:Method: GET 2024-03-04T17:36:48.592 DEBUG:boto:Path: /ppfahn-10/?prefix=myobj 2024-03-04T17:36:48.592 DEBUG:boto:Data: 2024-03-04T17:36:48.592 DEBUG:boto:Headers: {} 2024-03-04T17:36:48.592 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:36:48.592 DEBUG:boto:Port: 8000 2024-03-04T17:36:48.592 DEBUG:boto:Params: {} 2024-03-04T17:36:48.592 DEBUG:boto:Token: None 2024-03-04T17:36:48.593 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:36:48 GMT /ppfahn-10/ 2024-03-04T17:36:48.593 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA= 2024-03-04T17:36:48.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:36:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f7Qd5BVkRKj/h1pIuqQNbZZ4wGA=', 'Content-Length': '0'} 2024-03-04T17:36:48.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff2d722784914f72-0065e606b0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:36:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:36:48.597 DEBUG:boto:b'ppfahn-10myobj1000falsemyobj2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:36:48.598 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-10 zones={test-zone2, test-zone1} 2024-03-04T17:36:48.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:36:48.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:36:48.598 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:36:48.648 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:48.648 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:48.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:48.702+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:36:53.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:53.702+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:36:58.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:36:58.706+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:37:03.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:37:03.706+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:37:08.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:37:08.706+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:37:13.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:37:13.706+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:37:18.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:37:18.710+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:37:23.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:37:23.710+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:37:28.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:37:28.710+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:37:33.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:37:33.709+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:37:38.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:37:38.713+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:37:43.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:37:43.713+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:37:48.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:37:48.713+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:37:53.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:37:53.717+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:37:58.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:37:58.717+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:38:03.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:38:03.721+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:38:08.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:38:08.721+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:38:13.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:38:13.724+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:38:18.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:38:18.724+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:38:23.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:38:23.724+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:38:28.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:38:28.728+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:38:33.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:38:33.728+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:38:38.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:38:38.728+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:38:43.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:38:43.732+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:38:48.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:38:48.732+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:38:53.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:38:53.732+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:38:58.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:38:58.732+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:39:03.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:03.736+0000 7f62d5d92980 1 waiting to reach incremental sync.. 2024-03-04T17:39:08.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:08.740+0000 7f62d5d92980 1 bucket sync caught up with source: 2024-03-04T17:39:08.740 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000002.10.6, , 00000000001.17.6, , 00000000001.14.6, 00000000002.13.6, 00000000001.21.6, 00000000001.16.6, 00000000001.9.6, ] 2024-03-04T17:39:08.740 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.10.6, , 00000000001.17.6, , 00000000001.14.6, 00000000002.13.6, 00000000001.21.6, 00000000001.16.6, 00000000001.9.6, ] 2024-03-04T17:39:08.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:08.740+0000 7f62d5d92980 0 bucket checkpoint complete 2024-03-04T17:39:08.784 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-10 zones={test-zone3, test-zone1} 2024-03-04T17:39:08.784 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:08.784 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:08.784 DEBUG:boto:Method: HEAD 2024-03-04T17:39:08.785 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:39:08.785 DEBUG:boto:Data: 2024-03-04T17:39:08.785 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.785 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:08.785 DEBUG:boto:Port: 8001 2024-03-04T17:39:08.785 DEBUG:boto:Params: {} 2024-03-04T17:39:08.785 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T17:39:08.786 DEBUG:boto:Token: None 2024-03-04T17:39:08.786 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/ 2024-03-04T17:39:08.786 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:djRZYz+p2XnhCStHdUqfPxixLDY= 2024-03-04T17:39:08.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:djRZYz+p2XnhCStHdUqfPxixLDY=', 'Content-Length': '0'} 2024-03-04T17:39:08.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000674a67336e52e194-0065e6073c-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.791 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:08.791 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:08.792 DEBUG:boto:Method: HEAD 2024-03-04T17:39:08.792 DEBUG:boto:Path: /ppfahn-10/ 2024-03-04T17:39:08.792 DEBUG:boto:Data: 2024-03-04T17:39:08.792 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.792 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:08.792 DEBUG:boto:Port: 8000 2024-03-04T17:39:08.792 DEBUG:boto:Params: {} 2024-03-04T17:39:08.792 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:39:08.792 DEBUG:boto:Token: None 2024-03-04T17:39:08.793 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/ 2024-03-04T17:39:08.793 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:djRZYz+p2XnhCStHdUqfPxixLDY= 2024-03-04T17:39:08.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:djRZYz+p2XnhCStHdUqfPxixLDY=', 'Content-Length': '0'} 2024-03-04T17:39:08.797 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004aba77d3853930e2-0065e6073c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.798 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:39:08.798 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:08.798 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:08.798 DEBUG:boto:path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:39:08.798 DEBUG:boto:auth_path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:39:08.798 DEBUG:boto:Method: GET 2024-03-04T17:39:08.798 DEBUG:boto:Path: /ppfahn-10/?versions&max-keys=999 2024-03-04T17:39:08.798 DEBUG:boto:Data: 2024-03-04T17:39:08.798 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.799 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:08.799 DEBUG:boto:Port: 8001 2024-03-04T17:39:08.799 DEBUG:boto:Params: {} 2024-03-04T17:39:08.799 DEBUG:boto:Token: None 2024-03-04T17:39:08.799 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/?versions 2024-03-04T17:39:08.799 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HL1RL35id4Z8LlCSpAtmbhi0Y9k= 2024-03-04T17:39:08.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HL1RL35id4Z8LlCSpAtmbhi0Y9k=', 'Content-Length': '0'} 2024-03-04T17:39:08.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000151fcead61f5e49-0065e6073c-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.803 DEBUG:boto:b'ppfahn-10999false&nulltrue2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:08.803 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T17:39:08.804 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T17:39:08.804 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T17:39:08.804 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T17:39:08.804 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T17:39:08.804 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T17:39:08.804 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T17:39:08.804 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T17:39:08.804 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:08.804 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:39:08.804 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:08.804 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:08.805 DEBUG:boto:path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:39:08.805 DEBUG:boto:auth_path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:39:08.805 DEBUG:boto:Method: GET 2024-03-04T17:39:08.805 DEBUG:boto:Path: /ppfahn-10/?versions&max-keys=999 2024-03-04T17:39:08.805 DEBUG:boto:Data: 2024-03-04T17:39:08.805 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.805 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:08.805 DEBUG:boto:Port: 8000 2024-03-04T17:39:08.805 DEBUG:boto:Params: {} 2024-03-04T17:39:08.805 DEBUG:boto:Token: None 2024-03-04T17:39:08.805 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/?versions 2024-03-04T17:39:08.806 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HL1RL35id4Z8LlCSpAtmbhi0Y9k= 2024-03-04T17:39:08.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HL1RL35id4Z8LlCSpAtmbhi0Y9k=', 'Content-Length': '0'} 2024-03-04T17:39:08.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d9f1800f946d9ea-0065e6073c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.809 DEBUG:boto:b'ppfahn-10999false&nulltrue2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:08.810 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T17:39:08.810 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T17:39:08.810 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T17:39:08.810 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T17:39:08.810 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T17:39:08.810 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T17:39:08.810 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T17:39:08.810 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T17:39:08.810 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:08.811 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:08.811 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:08.811 DEBUG:boto:path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:39:08.811 DEBUG:boto:auth_path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:39:08.811 DEBUG:boto:Method: GET 2024-03-04T17:39:08.811 DEBUG:boto:Path: /ppfahn-10/?versions&max-keys=999 2024-03-04T17:39:08.811 DEBUG:boto:Data: 2024-03-04T17:39:08.811 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.811 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:08.811 DEBUG:boto:Port: 8001 2024-03-04T17:39:08.811 DEBUG:boto:Params: {} 2024-03-04T17:39:08.811 DEBUG:boto:Token: None 2024-03-04T17:39:08.812 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/?versions 2024-03-04T17:39:08.812 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HL1RL35id4Z8LlCSpAtmbhi0Y9k= 2024-03-04T17:39:08.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HL1RL35id4Z8LlCSpAtmbhi0Y9k=', 'Content-Length': '0'} 2024-03-04T17:39:08.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000facf877b32552f9a-0065e6073c-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.815 DEBUG:boto:b'ppfahn-10999false&nulltrue2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:08.816 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:08.816 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:08.816 DEBUG:boto:path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:39:08.816 DEBUG:boto:auth_path=/ppfahn-10/?versions&max-keys=999 2024-03-04T17:39:08.816 DEBUG:boto:Method: GET 2024-03-04T17:39:08.816 DEBUG:boto:Path: /ppfahn-10/?versions&max-keys=999 2024-03-04T17:39:08.817 DEBUG:boto:Data: 2024-03-04T17:39:08.817 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.817 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:08.817 DEBUG:boto:Port: 8000 2024-03-04T17:39:08.817 DEBUG:boto:Params: {} 2024-03-04T17:39:08.817 DEBUG:boto:Token: None 2024-03-04T17:39:08.817 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/?versions 2024-03-04T17:39:08.817 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HL1RL35id4Z8LlCSpAtmbhi0Y9k= 2024-03-04T17:39:08.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HL1RL35id4Z8LlCSpAtmbhi0Y9k=', 'Content-Length': '0'} 2024-03-04T17:39:08.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac3cb71b5b42be4f-0065e6073c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.821 DEBUG:boto:b'ppfahn-10999false&nulltrue2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:08.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:08.822 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:39:08.822 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:39:08.822 DEBUG:boto:path=/ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.822 DEBUG:boto:auth_path=/ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.822 DEBUG:boto:Method: GET 2024-03-04T17:39:08.822 DEBUG:boto:Path: /ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.822 DEBUG:boto:Data: 2024-03-04T17:39:08.822 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.822 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:08.823 DEBUG:boto:Port: 8001 2024-03-04T17:39:08.823 DEBUG:boto:Params: {} 2024-03-04T17:39:08.823 DEBUG:boto:Token: None 2024-03-04T17:39:08.823 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.823 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SuSrmfMmxqOndOYvTve00RnZPUE= 2024-03-04T17:39:08.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SuSrmfMmxqOndOYvTve00RnZPUE=', 'Content-Length': '0'} 2024-03-04T17:39:08.826 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000565d3cdc2d19cc6c-0065e6073c-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.826 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:39:08.826 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:39:08.826 DEBUG:boto:path=/ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.826 DEBUG:boto:auth_path=/ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.826 DEBUG:boto:Method: GET 2024-03-04T17:39:08.826 DEBUG:boto:Path: /ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.827 DEBUG:boto:Data: 2024-03-04T17:39:08.827 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.827 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:08.827 DEBUG:boto:Port: 8000 2024-03-04T17:39:08.827 DEBUG:boto:Params: {} 2024-03-04T17:39:08.827 DEBUG:boto:Token: None 2024-03-04T17:39:08.827 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.827 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SuSrmfMmxqOndOYvTve00RnZPUE= 2024-03-04T17:39:08.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SuSrmfMmxqOndOYvTve00RnZPUE=', 'Content-Length': '0'} 2024-03-04T17:39:08.830 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000034c6e8f10ab675e-0065e6073c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.872 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:39:08.872 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:39:08.872 DEBUG:boto:path=/ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.872 DEBUG:boto:auth_path=/ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.872 DEBUG:boto:Method: HEAD 2024-03-04T17:39:08.872 DEBUG:boto:Path: /ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.873 DEBUG:boto:Data: 2024-03-04T17:39:08.873 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.873 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:08.873 DEBUG:boto:Port: 8001 2024-03-04T17:39:08.873 DEBUG:boto:Params: {} 2024-03-04T17:39:08.873 DEBUG:boto:Token: None 2024-03-04T17:39:08.873 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.873 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZShZHXXpLZ8JWcVe2mleuVk1OSg= 2024-03-04T17:39:08.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZShZHXXpLZ8JWcVe2mleuVk1OSg=', 'Content-Length': '0'} 2024-03-04T17:39:08.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000430353e992e80954-0065e6073c-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.876 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:39:08.876 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:39:08.877 DEBUG:boto:path=/ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.877 DEBUG:boto:auth_path=/ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.877 DEBUG:boto:Method: HEAD 2024-03-04T17:39:08.877 DEBUG:boto:Path: /ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.877 DEBUG:boto:Data: 2024-03-04T17:39:08.877 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.877 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:08.877 DEBUG:boto:Port: 8000 2024-03-04T17:39:08.877 DEBUG:boto:Params: {} 2024-03-04T17:39:08.877 DEBUG:boto:Token: None 2024-03-04T17:39:08.877 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.878 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZShZHXXpLZ8JWcVe2mleuVk1OSg= 2024-03-04T17:39:08.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZShZHXXpLZ8JWcVe2mleuVk1OSg=', 'Content-Length': '0'} 2024-03-04T17:39:08.880 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000114a103865b5ed37-0065e6073c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:08.881 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:39:08.881 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:39:08.881 DEBUG:boto:path=/ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.881 DEBUG:boto:auth_path=/ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.881 DEBUG:boto:Method: GET 2024-03-04T17:39:08.881 DEBUG:boto:Path: /ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.881 DEBUG:boto:Data: 2024-03-04T17:39:08.881 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.882 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:08.882 DEBUG:boto:Port: 8001 2024-03-04T17:39:08.882 DEBUG:boto:Params: {} 2024-03-04T17:39:08.882 DEBUG:boto:Token: None 2024-03-04T17:39:08.882 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.882 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SuSrmfMmxqOndOYvTve00RnZPUE= 2024-03-04T17:39:08.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SuSrmfMmxqOndOYvTve00RnZPUE=', 'Content-Length': '0'} 2024-03-04T17:39:08.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aedeb89b84dbb002-0065e6073c-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.928 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:39:08.928 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:39:08.928 DEBUG:boto:path=/ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.928 DEBUG:boto:auth_path=/ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.928 DEBUG:boto:Method: GET 2024-03-04T17:39:08.928 DEBUG:boto:Path: /ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.929 DEBUG:boto:Data: 2024-03-04T17:39:08.929 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.929 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:08.929 DEBUG:boto:Port: 8000 2024-03-04T17:39:08.929 DEBUG:boto:Params: {} 2024-03-04T17:39:08.929 DEBUG:boto:Token: None 2024-03-04T17:39:08.929 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/%26?versionId=null 2024-03-04T17:39:08.929 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SuSrmfMmxqOndOYvTve00RnZPUE= 2024-03-04T17:39:08.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SuSrmfMmxqOndOYvTve00RnZPUE=', 'Content-Length': '0'} 2024-03-04T17:39:08.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc9c0214da2d2e51-0065e6073c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.932 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:08.933 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:08.933 DEBUG:boto:path=/ppfahn-10/?prefix=%26 2024-03-04T17:39:08.933 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%26 2024-03-04T17:39:08.933 DEBUG:boto:Method: GET 2024-03-04T17:39:08.933 DEBUG:boto:Path: /ppfahn-10/?prefix=%26 2024-03-04T17:39:08.933 DEBUG:boto:Data: 2024-03-04T17:39:08.933 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.933 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:08.933 DEBUG:boto:Port: 8001 2024-03-04T17:39:08.933 DEBUG:boto:Params: {} 2024-03-04T17:39:08.933 DEBUG:boto:Token: None 2024-03-04T17:39:08.933 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/ 2024-03-04T17:39:08.934 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xAW8fDuTdOeGiUm3/HiqzgJ5xPY= 2024-03-04T17:39:08.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xAW8fDuTdOeGiUm3/HiqzgJ5xPY=', 'Content-Length': '0'} 2024-03-04T17:39:08.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a17130cb96b331c-0065e6073c-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.938 DEBUG:boto:b'ppfahn-10&1000false&2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:08.938 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:08.938 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:08.938 DEBUG:boto:path=/ppfahn-10/?prefix=%26 2024-03-04T17:39:08.938 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%26 2024-03-04T17:39:08.939 DEBUG:boto:Method: GET 2024-03-04T17:39:08.939 DEBUG:boto:Path: /ppfahn-10/?prefix=%26 2024-03-04T17:39:08.939 DEBUG:boto:Data: 2024-03-04T17:39:08.939 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.939 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:08.939 DEBUG:boto:Port: 8000 2024-03-04T17:39:08.939 DEBUG:boto:Params: {} 2024-03-04T17:39:08.939 DEBUG:boto:Token: None 2024-03-04T17:39:08.939 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/ 2024-03-04T17:39:08.939 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xAW8fDuTdOeGiUm3/HiqzgJ5xPY= 2024-03-04T17:39:08.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xAW8fDuTdOeGiUm3/HiqzgJ5xPY=', 'Content-Length': '0'} 2024-03-04T17:39:08.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7c43a7068cc370-0065e6073c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.944 DEBUG:boto:b'ppfahn-10&1000false&2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:08.944 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:39:08.944 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:39:08.944 DEBUG:boto:Method: HEAD 2024-03-04T17:39:08.945 DEBUG:boto:Path: /ppfahn-10/%26 2024-03-04T17:39:08.945 DEBUG:boto:Data: 2024-03-04T17:39:08.945 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.945 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:08.945 DEBUG:boto:Port: 8001 2024-03-04T17:39:08.945 DEBUG:boto:Params: {} 2024-03-04T17:39:08.945 DEBUG:boto:Token: None 2024-03-04T17:39:08.945 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/%26 2024-03-04T17:39:08.945 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nniIPpROMGtL3adCwbUZ4g+FvXY= 2024-03-04T17:39:08.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nniIPpROMGtL3adCwbUZ4g+FvXY=', 'Content-Length': '0'} 2024-03-04T17:39:08.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef746ce5cb64f43d-0065e6073c-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.948 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:39:08.948 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:39:08.948 DEBUG:boto:Method: HEAD 2024-03-04T17:39:08.948 DEBUG:boto:Path: /ppfahn-10/%26 2024-03-04T17:39:08.949 DEBUG:boto:Data: 2024-03-04T17:39:08.949 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.949 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:08.949 DEBUG:boto:Port: 8000 2024-03-04T17:39:08.949 DEBUG:boto:Params: {} 2024-03-04T17:39:08.949 DEBUG:boto:Token: None 2024-03-04T17:39:08.949 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/%26 2024-03-04T17:39:08.949 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nniIPpROMGtL3adCwbUZ4g+FvXY= 2024-03-04T17:39:08.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nniIPpROMGtL3adCwbUZ4g+FvXY=', 'Content-Length': '0'} 2024-03-04T17:39:08.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b2bcf6765cc99ff-0065e6073c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:08.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:08.952 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:39:08.952 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:39:08.953 DEBUG:boto:Method: GET 2024-03-04T17:39:08.953 DEBUG:boto:Path: /ppfahn-10/%26 2024-03-04T17:39:08.953 DEBUG:boto:Data: 2024-03-04T17:39:08.953 DEBUG:boto:Headers: {} 2024-03-04T17:39:08.953 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:08.953 DEBUG:boto:Port: 8001 2024-03-04T17:39:08.953 DEBUG:boto:Params: {} 2024-03-04T17:39:08.953 DEBUG:boto:Token: None 2024-03-04T17:39:08.953 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:08 GMT /ppfahn-10/%26 2024-03-04T17:39:08.953 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/7EQhRkI4ybXIcxtr0dJPCRmt3M= 2024-03-04T17:39:08.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:08 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/7EQhRkI4ybXIcxtr0dJPCRmt3M=', 'Content-Length': '0'} 2024-03-04T17:39:08.956 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b35b1420f3a379b3-0065e6073c-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.000 DEBUG:boto:path=/ppfahn-10/%26 2024-03-04T17:39:09.000 DEBUG:boto:auth_path=/ppfahn-10/%26 2024-03-04T17:39:09.000 DEBUG:boto:Method: GET 2024-03-04T17:39:09.000 DEBUG:boto:Path: /ppfahn-10/%26 2024-03-04T17:39:09.000 DEBUG:boto:Data: 2024-03-04T17:39:09.000 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.001 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.001 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.001 DEBUG:boto:Params: {} 2024-03-04T17:39:09.001 DEBUG:boto:Token: None 2024-03-04T17:39:09.001 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/%26 2024-03-04T17:39:09.001 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:h2MNxDszuaXvJdafqt2IPaMsFl0= 2024-03-04T17:39:09.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:h2MNxDszuaXvJdafqt2IPaMsFl0=', 'Content-Length': '0'} 2024-03-04T17:39:09.004 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b24d76b8ae277ee9-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.004 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.004 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.004 DEBUG:boto:path=/ppfahn-10/?prefix=%26 2024-03-04T17:39:09.004 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%26 2024-03-04T17:39:09.005 DEBUG:boto:Method: GET 2024-03-04T17:39:09.005 DEBUG:boto:Path: /ppfahn-10/?prefix=%26 2024-03-04T17:39:09.005 DEBUG:boto:Data: 2024-03-04T17:39:09.005 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.005 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.005 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.005 DEBUG:boto:Params: {} 2024-03-04T17:39:09.005 DEBUG:boto:Token: None 2024-03-04T17:39:09.005 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.005 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b9bf8c04b62d44d-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.010 DEBUG:boto:b'ppfahn-10&1000false&2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.010 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.010 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.010 DEBUG:boto:path=/ppfahn-10/?prefix=%26 2024-03-04T17:39:09.010 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%26 2024-03-04T17:39:09.010 DEBUG:boto:Method: GET 2024-03-04T17:39:09.010 DEBUG:boto:Path: /ppfahn-10/?prefix=%26 2024-03-04T17:39:09.010 DEBUG:boto:Data: 2024-03-04T17:39:09.010 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.011 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.011 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.011 DEBUG:boto:Params: {} 2024-03-04T17:39:09.011 DEBUG:boto:Token: None 2024-03-04T17:39:09.011 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.011 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008c757e90b9bef01-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.015 DEBUG:boto:b'ppfahn-10&1000false&2024-03-04T17:36:30.131Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:09.016 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:39:09.016 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:39:09.016 DEBUG:boto:path=/ppfahn-10/.?versionId=null 2024-03-04T17:39:09.016 DEBUG:boto:auth_path=/ppfahn-10/.?versionId=null 2024-03-04T17:39:09.016 DEBUG:boto:Method: GET 2024-03-04T17:39:09.016 DEBUG:boto:Path: /ppfahn-10/.?versionId=null 2024-03-04T17:39:09.017 DEBUG:boto:Data: 2024-03-04T17:39:09.017 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.017 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.017 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.017 DEBUG:boto:Params: {} 2024-03-04T17:39:09.017 DEBUG:boto:Token: None 2024-03-04T17:39:09.017 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.?versionId=null 2024-03-04T17:39:09.017 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0q/GA1bS6+xcX1pZoG+Xl/GlQM4= 2024-03-04T17:39:09.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0q/GA1bS6+xcX1pZoG+Xl/GlQM4=', 'Content-Length': '0'} 2024-03-04T17:39:09.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cf9b8f0c9d43f0a-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.060 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:39:09.060 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:39:09.060 DEBUG:boto:path=/ppfahn-10/.?versionId=null 2024-03-04T17:39:09.060 DEBUG:boto:auth_path=/ppfahn-10/.?versionId=null 2024-03-04T17:39:09.060 DEBUG:boto:Method: GET 2024-03-04T17:39:09.060 DEBUG:boto:Path: /ppfahn-10/.?versionId=null 2024-03-04T17:39:09.061 DEBUG:boto:Data: 2024-03-04T17:39:09.061 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.061 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.061 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.061 DEBUG:boto:Params: {} 2024-03-04T17:39:09.061 DEBUG:boto:Token: None 2024-03-04T17:39:09.061 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.?versionId=null 2024-03-04T17:39:09.061 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0q/GA1bS6+xcX1pZoG+Xl/GlQM4= 2024-03-04T17:39:09.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0q/GA1bS6+xcX1pZoG+Xl/GlQM4=', 'Content-Length': '0'} 2024-03-04T17:39:09.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bc36bfd06c00c42-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.064 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:39:09.064 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:39:09.065 DEBUG:boto:path=/ppfahn-10/.?versionId=null 2024-03-04T17:39:09.065 DEBUG:boto:auth_path=/ppfahn-10/.?versionId=null 2024-03-04T17:39:09.065 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.065 DEBUG:boto:Path: /ppfahn-10/.?versionId=null 2024-03-04T17:39:09.065 DEBUG:boto:Data: 2024-03-04T17:39:09.065 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.065 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.065 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.065 DEBUG:boto:Params: {} 2024-03-04T17:39:09.065 DEBUG:boto:Token: None 2024-03-04T17:39:09.065 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.?versionId=null 2024-03-04T17:39:09.066 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:P+cdNwLR7VyZVKcra4UOoJeyByA= 2024-03-04T17:39:09.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:P+cdNwLR7VyZVKcra4UOoJeyByA=', 'Content-Length': '0'} 2024-03-04T17:39:09.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce034bda3cc7ded1-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.069 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:39:09.069 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:39:09.069 DEBUG:boto:path=/ppfahn-10/.?versionId=null 2024-03-04T17:39:09.069 DEBUG:boto:auth_path=/ppfahn-10/.?versionId=null 2024-03-04T17:39:09.069 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.069 DEBUG:boto:Path: /ppfahn-10/.?versionId=null 2024-03-04T17:39:09.069 DEBUG:boto:Data: 2024-03-04T17:39:09.069 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.069 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.069 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.069 DEBUG:boto:Params: {} 2024-03-04T17:39:09.069 DEBUG:boto:Token: None 2024-03-04T17:39:09.070 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.?versionId=null 2024-03-04T17:39:09.070 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:P+cdNwLR7VyZVKcra4UOoJeyByA= 2024-03-04T17:39:09.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:P+cdNwLR7VyZVKcra4UOoJeyByA=', 'Content-Length': '0'} 2024-03-04T17:39:09.072 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001da5865f459c162a-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:09.073 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:39:09.073 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:39:09.073 DEBUG:boto:path=/ppfahn-10/.?versionId=null 2024-03-04T17:39:09.073 DEBUG:boto:auth_path=/ppfahn-10/.?versionId=null 2024-03-04T17:39:09.073 DEBUG:boto:Method: GET 2024-03-04T17:39:09.073 DEBUG:boto:Path: /ppfahn-10/.?versionId=null 2024-03-04T17:39:09.073 DEBUG:boto:Data: 2024-03-04T17:39:09.073 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.073 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.073 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.073 DEBUG:boto:Params: {} 2024-03-04T17:39:09.074 DEBUG:boto:Token: None 2024-03-04T17:39:09.074 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.?versionId=null 2024-03-04T17:39:09.074 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0q/GA1bS6+xcX1pZoG+Xl/GlQM4= 2024-03-04T17:39:09.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0q/GA1bS6+xcX1pZoG+Xl/GlQM4=', 'Content-Length': '0'} 2024-03-04T17:39:09.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ab183a38daf21fa-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.120 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:39:09.120 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:39:09.120 DEBUG:boto:path=/ppfahn-10/.?versionId=null 2024-03-04T17:39:09.120 DEBUG:boto:auth_path=/ppfahn-10/.?versionId=null 2024-03-04T17:39:09.120 DEBUG:boto:Method: GET 2024-03-04T17:39:09.121 DEBUG:boto:Path: /ppfahn-10/.?versionId=null 2024-03-04T17:39:09.121 DEBUG:boto:Data: 2024-03-04T17:39:09.121 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.121 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.121 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.121 DEBUG:boto:Params: {} 2024-03-04T17:39:09.121 DEBUG:boto:Token: None 2024-03-04T17:39:09.121 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.?versionId=null 2024-03-04T17:39:09.121 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0q/GA1bS6+xcX1pZoG+Xl/GlQM4= 2024-03-04T17:39:09.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0q/GA1bS6+xcX1pZoG+Xl/GlQM4=', 'Content-Length': '0'} 2024-03-04T17:39:09.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f6c45a65cf2b449-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.124 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.125 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.125 DEBUG:boto:path=/ppfahn-10/?prefix=. 2024-03-04T17:39:09.125 DEBUG:boto:auth_path=/ppfahn-10/?prefix=. 2024-03-04T17:39:09.125 DEBUG:boto:Method: GET 2024-03-04T17:39:09.125 DEBUG:boto:Path: /ppfahn-10/?prefix=. 2024-03-04T17:39:09.125 DEBUG:boto:Data: 2024-03-04T17:39:09.125 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.125 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.125 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.125 DEBUG:boto:Params: {} 2024-03-04T17:39:09.125 DEBUG:boto:Token: None 2024-03-04T17:39:09.126 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.126 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b5b74f0b5498b90-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.130 DEBUG:boto:b'ppfahn-10.1000false.2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.130 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.131 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.131 DEBUG:boto:path=/ppfahn-10/?prefix=. 2024-03-04T17:39:09.131 DEBUG:boto:auth_path=/ppfahn-10/?prefix=. 2024-03-04T17:39:09.131 DEBUG:boto:Method: GET 2024-03-04T17:39:09.131 DEBUG:boto:Path: /ppfahn-10/?prefix=. 2024-03-04T17:39:09.131 DEBUG:boto:Data: 2024-03-04T17:39:09.131 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.131 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.131 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.131 DEBUG:boto:Params: {} 2024-03-04T17:39:09.131 DEBUG:boto:Token: None 2024-03-04T17:39:09.131 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.132 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9c765319c6e5025-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.136 DEBUG:boto:b'ppfahn-10.1000false.2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.136 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:39:09.137 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:39:09.137 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.137 DEBUG:boto:Path: /ppfahn-10/. 2024-03-04T17:39:09.137 DEBUG:boto:Data: 2024-03-04T17:39:09.137 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.137 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.137 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.137 DEBUG:boto:Params: {} 2024-03-04T17:39:09.137 DEBUG:boto:Token: None 2024-03-04T17:39:09.137 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/. 2024-03-04T17:39:09.138 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Kc/OPE0EetGGug7TvQqlQUus2l8= 2024-03-04T17:39:09.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Kc/OPE0EetGGug7TvQqlQUus2l8=', 'Content-Length': '0'} 2024-03-04T17:39:09.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043547c8132234bf2-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.141 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:39:09.141 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:39:09.141 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.141 DEBUG:boto:Path: /ppfahn-10/. 2024-03-04T17:39:09.141 DEBUG:boto:Data: 2024-03-04T17:39:09.141 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.141 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.141 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.141 DEBUG:boto:Params: {} 2024-03-04T17:39:09.141 DEBUG:boto:Token: None 2024-03-04T17:39:09.141 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/. 2024-03-04T17:39:09.142 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Kc/OPE0EetGGug7TvQqlQUus2l8= 2024-03-04T17:39:09.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Kc/OPE0EetGGug7TvQqlQUus2l8=', 'Content-Length': '0'} 2024-03-04T17:39:09.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa95fe0e3060aedd-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:09.145 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:39:09.145 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:39:09.145 DEBUG:boto:Method: GET 2024-03-04T17:39:09.145 DEBUG:boto:Path: /ppfahn-10/. 2024-03-04T17:39:09.145 DEBUG:boto:Data: 2024-03-04T17:39:09.145 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.145 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.145 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.145 DEBUG:boto:Params: {} 2024-03-04T17:39:09.145 DEBUG:boto:Token: None 2024-03-04T17:39:09.146 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/. 2024-03-04T17:39:09.146 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wf/XOce8z6eKUzbWMk0RFeBGbts= 2024-03-04T17:39:09.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wf/XOce8z6eKUzbWMk0RFeBGbts=', 'Content-Length': '0'} 2024-03-04T17:39:09.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062c921067a0b8cf3-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.192 DEBUG:boto:path=/ppfahn-10/. 2024-03-04T17:39:09.192 DEBUG:boto:auth_path=/ppfahn-10/. 2024-03-04T17:39:09.192 DEBUG:boto:Method: GET 2024-03-04T17:39:09.192 DEBUG:boto:Path: /ppfahn-10/. 2024-03-04T17:39:09.193 DEBUG:boto:Data: 2024-03-04T17:39:09.193 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.193 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.193 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.193 DEBUG:boto:Params: {} 2024-03-04T17:39:09.193 DEBUG:boto:Token: None 2024-03-04T17:39:09.193 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/. 2024-03-04T17:39:09.193 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wf/XOce8z6eKUzbWMk0RFeBGbts= 2024-03-04T17:39:09.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wf/XOce8z6eKUzbWMk0RFeBGbts=', 'Content-Length': '0'} 2024-03-04T17:39:09.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd6fa4a2838821af-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.196 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.196 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.196 DEBUG:boto:path=/ppfahn-10/?prefix=. 2024-03-04T17:39:09.197 DEBUG:boto:auth_path=/ppfahn-10/?prefix=. 2024-03-04T17:39:09.197 DEBUG:boto:Method: GET 2024-03-04T17:39:09.197 DEBUG:boto:Path: /ppfahn-10/?prefix=. 2024-03-04T17:39:09.197 DEBUG:boto:Data: 2024-03-04T17:39:09.197 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.197 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.197 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.197 DEBUG:boto:Params: {} 2024-03-04T17:39:09.197 DEBUG:boto:Token: None 2024-03-04T17:39:09.197 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.197 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cd384a972086ae5-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.202 DEBUG:boto:b'ppfahn-10.1000false.2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.203 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.203 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.203 DEBUG:boto:path=/ppfahn-10/?prefix=. 2024-03-04T17:39:09.203 DEBUG:boto:auth_path=/ppfahn-10/?prefix=. 2024-03-04T17:39:09.203 DEBUG:boto:Method: GET 2024-03-04T17:39:09.203 DEBUG:boto:Path: /ppfahn-10/?prefix=. 2024-03-04T17:39:09.203 DEBUG:boto:Data: 2024-03-04T17:39:09.203 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.203 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.203 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.203 DEBUG:boto:Params: {} 2024-03-04T17:39:09.204 DEBUG:boto:Token: None 2024-03-04T17:39:09.204 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.204 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a34ffedb24b2e84-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.208 DEBUG:boto:b'ppfahn-10.1000false.2024-03-04T17:36:30.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:09.209 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:39:09.209 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:39:09.209 DEBUG:boto:path=/ppfahn-10/..?versionId=null 2024-03-04T17:39:09.209 DEBUG:boto:auth_path=/ppfahn-10/..?versionId=null 2024-03-04T17:39:09.209 DEBUG:boto:Method: GET 2024-03-04T17:39:09.209 DEBUG:boto:Path: /ppfahn-10/..?versionId=null 2024-03-04T17:39:09.209 DEBUG:boto:Data: 2024-03-04T17:39:09.209 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.209 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.209 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.210 DEBUG:boto:Params: {} 2024-03-04T17:39:09.210 DEBUG:boto:Token: None 2024-03-04T17:39:09.210 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/..?versionId=null 2024-03-04T17:39:09.210 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mdP30OfjE2LbSaMDJxYcHh1GSAg= 2024-03-04T17:39:09.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mdP30OfjE2LbSaMDJxYcHh1GSAg=', 'Content-Length': '0'} 2024-03-04T17:39:09.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f676af1ef9ef4ae7-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.213 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:39:09.213 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:39:09.213 DEBUG:boto:path=/ppfahn-10/..?versionId=null 2024-03-04T17:39:09.213 DEBUG:boto:auth_path=/ppfahn-10/..?versionId=null 2024-03-04T17:39:09.213 DEBUG:boto:Method: GET 2024-03-04T17:39:09.213 DEBUG:boto:Path: /ppfahn-10/..?versionId=null 2024-03-04T17:39:09.213 DEBUG:boto:Data: 2024-03-04T17:39:09.214 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.214 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.214 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.214 DEBUG:boto:Params: {} 2024-03-04T17:39:09.214 DEBUG:boto:Token: None 2024-03-04T17:39:09.214 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/..?versionId=null 2024-03-04T17:39:09.214 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mdP30OfjE2LbSaMDJxYcHh1GSAg= 2024-03-04T17:39:09.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mdP30OfjE2LbSaMDJxYcHh1GSAg=', 'Content-Length': '0'} 2024-03-04T17:39:09.217 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f2406850fc0a518-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.217 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:39:09.217 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:39:09.217 DEBUG:boto:path=/ppfahn-10/..?versionId=null 2024-03-04T17:39:09.217 DEBUG:boto:auth_path=/ppfahn-10/..?versionId=null 2024-03-04T17:39:09.218 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.218 DEBUG:boto:Path: /ppfahn-10/..?versionId=null 2024-03-04T17:39:09.218 DEBUG:boto:Data: 2024-03-04T17:39:09.218 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.218 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.218 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.218 DEBUG:boto:Params: {} 2024-03-04T17:39:09.218 DEBUG:boto:Token: None 2024-03-04T17:39:09.218 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/..?versionId=null 2024-03-04T17:39:09.218 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fmisN8n9RCMvT5/C4Zfd7ePzbVI= 2024-03-04T17:39:09.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fmisN8n9RCMvT5/C4Zfd7ePzbVI=', 'Content-Length': '0'} 2024-03-04T17:39:09.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a311151ee0dbd48-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.221 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:39:09.221 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:39:09.221 DEBUG:boto:path=/ppfahn-10/..?versionId=null 2024-03-04T17:39:09.221 DEBUG:boto:auth_path=/ppfahn-10/..?versionId=null 2024-03-04T17:39:09.222 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.222 DEBUG:boto:Path: /ppfahn-10/..?versionId=null 2024-03-04T17:39:09.222 DEBUG:boto:Data: 2024-03-04T17:39:09.222 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.222 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.222 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.222 DEBUG:boto:Params: {} 2024-03-04T17:39:09.222 DEBUG:boto:Token: None 2024-03-04T17:39:09.222 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/..?versionId=null 2024-03-04T17:39:09.222 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fmisN8n9RCMvT5/C4Zfd7ePzbVI= 2024-03-04T17:39:09.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fmisN8n9RCMvT5/C4Zfd7ePzbVI=', 'Content-Length': '0'} 2024-03-04T17:39:09.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f56c3bff542447b-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:09.225 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:39:09.225 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:39:09.225 DEBUG:boto:path=/ppfahn-10/..?versionId=null 2024-03-04T17:39:09.225 DEBUG:boto:auth_path=/ppfahn-10/..?versionId=null 2024-03-04T17:39:09.226 DEBUG:boto:Method: GET 2024-03-04T17:39:09.226 DEBUG:boto:Path: /ppfahn-10/..?versionId=null 2024-03-04T17:39:09.226 DEBUG:boto:Data: 2024-03-04T17:39:09.226 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.226 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.226 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.226 DEBUG:boto:Params: {} 2024-03-04T17:39:09.226 DEBUG:boto:Token: None 2024-03-04T17:39:09.226 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/..?versionId=null 2024-03-04T17:39:09.226 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mdP30OfjE2LbSaMDJxYcHh1GSAg= 2024-03-04T17:39:09.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mdP30OfjE2LbSaMDJxYcHh1GSAg=', 'Content-Length': '0'} 2024-03-04T17:39:09.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df8d1479d045cf0f-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.276 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:39:09.276 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:39:09.276 DEBUG:boto:path=/ppfahn-10/..?versionId=null 2024-03-04T17:39:09.276 DEBUG:boto:auth_path=/ppfahn-10/..?versionId=null 2024-03-04T17:39:09.276 DEBUG:boto:Method: GET 2024-03-04T17:39:09.276 DEBUG:boto:Path: /ppfahn-10/..?versionId=null 2024-03-04T17:39:09.277 DEBUG:boto:Data: 2024-03-04T17:39:09.277 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.277 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.277 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.277 DEBUG:boto:Params: {} 2024-03-04T17:39:09.277 DEBUG:boto:Token: None 2024-03-04T17:39:09.277 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/..?versionId=null 2024-03-04T17:39:09.277 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mdP30OfjE2LbSaMDJxYcHh1GSAg= 2024-03-04T17:39:09.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mdP30OfjE2LbSaMDJxYcHh1GSAg=', 'Content-Length': '0'} 2024-03-04T17:39:09.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b93911b6be928f14-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.280 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.280 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.281 DEBUG:boto:path=/ppfahn-10/?prefix=.. 2024-03-04T17:39:09.281 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.. 2024-03-04T17:39:09.281 DEBUG:boto:Method: GET 2024-03-04T17:39:09.281 DEBUG:boto:Path: /ppfahn-10/?prefix=.. 2024-03-04T17:39:09.281 DEBUG:boto:Data: 2024-03-04T17:39:09.281 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.281 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.281 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.281 DEBUG:boto:Params: {} 2024-03-04T17:39:09.281 DEBUG:boto:Token: None 2024-03-04T17:39:09.281 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.282 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ce8bdbe94610c85-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.286 DEBUG:boto:b'ppfahn-10..1000false..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.286 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.286 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.286 DEBUG:boto:path=/ppfahn-10/?prefix=.. 2024-03-04T17:39:09.286 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.. 2024-03-04T17:39:09.286 DEBUG:boto:Method: GET 2024-03-04T17:39:09.287 DEBUG:boto:Path: /ppfahn-10/?prefix=.. 2024-03-04T17:39:09.287 DEBUG:boto:Data: 2024-03-04T17:39:09.287 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.287 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.287 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.287 DEBUG:boto:Params: {} 2024-03-04T17:39:09.287 DEBUG:boto:Token: None 2024-03-04T17:39:09.287 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.287 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b59f26d4f0e27cc-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.293 DEBUG:boto:b'ppfahn-10..1000false..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.294 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:39:09.294 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:39:09.294 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.294 DEBUG:boto:Path: /ppfahn-10/.. 2024-03-04T17:39:09.294 DEBUG:boto:Data: 2024-03-04T17:39:09.294 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.294 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.294 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.294 DEBUG:boto:Params: {} 2024-03-04T17:39:09.294 DEBUG:boto:Token: None 2024-03-04T17:39:09.294 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.. 2024-03-04T17:39:09.295 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4ULGGKDMu6LcsTwAXIs+iRXstdE= 2024-03-04T17:39:09.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4ULGGKDMu6LcsTwAXIs+iRXstdE=', 'Content-Length': '0'} 2024-03-04T17:39:09.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011620f4eb42debab-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.297 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:39:09.297 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:39:09.298 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.298 DEBUG:boto:Path: /ppfahn-10/.. 2024-03-04T17:39:09.298 DEBUG:boto:Data: 2024-03-04T17:39:09.298 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.298 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.298 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.298 DEBUG:boto:Params: {} 2024-03-04T17:39:09.298 DEBUG:boto:Token: None 2024-03-04T17:39:09.298 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.. 2024-03-04T17:39:09.298 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4ULGGKDMu6LcsTwAXIs+iRXstdE= 2024-03-04T17:39:09.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4ULGGKDMu6LcsTwAXIs+iRXstdE=', 'Content-Length': '0'} 2024-03-04T17:39:09.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e84aefb2fe55a326-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:09.302 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:39:09.302 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:39:09.302 DEBUG:boto:Method: GET 2024-03-04T17:39:09.302 DEBUG:boto:Path: /ppfahn-10/.. 2024-03-04T17:39:09.302 DEBUG:boto:Data: 2024-03-04T17:39:09.302 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.302 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.302 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.302 DEBUG:boto:Params: {} 2024-03-04T17:39:09.303 DEBUG:boto:Token: None 2024-03-04T17:39:09.303 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.. 2024-03-04T17:39:09.303 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1V5cry8gL6EV9th1fxiEy23wX0s= 2024-03-04T17:39:09.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1V5cry8gL6EV9th1fxiEy23wX0s=', 'Content-Length': '0'} 2024-03-04T17:39:09.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4ecc627f5da8655-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.348 DEBUG:boto:path=/ppfahn-10/.. 2024-03-04T17:39:09.348 DEBUG:boto:auth_path=/ppfahn-10/.. 2024-03-04T17:39:09.348 DEBUG:boto:Method: GET 2024-03-04T17:39:09.348 DEBUG:boto:Path: /ppfahn-10/.. 2024-03-04T17:39:09.348 DEBUG:boto:Data: 2024-03-04T17:39:09.348 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.349 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.349 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.349 DEBUG:boto:Params: {} 2024-03-04T17:39:09.349 DEBUG:boto:Token: None 2024-03-04T17:39:09.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.. 2024-03-04T17:39:09.349 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1V5cry8gL6EV9th1fxiEy23wX0s= 2024-03-04T17:39:09.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1V5cry8gL6EV9th1fxiEy23wX0s=', 'Content-Length': '0'} 2024-03-04T17:39:09.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7b6daff0cee83ee-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.352 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.352 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.353 DEBUG:boto:path=/ppfahn-10/?prefix=.. 2024-03-04T17:39:09.353 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.. 2024-03-04T17:39:09.353 DEBUG:boto:Method: GET 2024-03-04T17:39:09.353 DEBUG:boto:Path: /ppfahn-10/?prefix=.. 2024-03-04T17:39:09.353 DEBUG:boto:Data: 2024-03-04T17:39:09.353 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.353 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.353 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.353 DEBUG:boto:Params: {} 2024-03-04T17:39:09.353 DEBUG:boto:Token: None 2024-03-04T17:39:09.353 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.354 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a89d5629409626e-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.358 DEBUG:boto:b'ppfahn-10..1000false..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.358 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.358 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.358 DEBUG:boto:path=/ppfahn-10/?prefix=.. 2024-03-04T17:39:09.358 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.. 2024-03-04T17:39:09.358 DEBUG:boto:Method: GET 2024-03-04T17:39:09.358 DEBUG:boto:Path: /ppfahn-10/?prefix=.. 2024-03-04T17:39:09.359 DEBUG:boto:Data: 2024-03-04T17:39:09.359 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.359 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.359 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.359 DEBUG:boto:Params: {} 2024-03-04T17:39:09.359 DEBUG:boto:Token: None 2024-03-04T17:39:09.359 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.359 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a4eaf00de4d7c84-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.404 DEBUG:boto:b'ppfahn-10..1000false..2024-03-04T17:36:30.207Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:09.405 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:39:09.405 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:39:09.405 DEBUG:boto:path=/ppfahn-10/...?versionId=null 2024-03-04T17:39:09.405 DEBUG:boto:auth_path=/ppfahn-10/...?versionId=null 2024-03-04T17:39:09.405 DEBUG:boto:Method: GET 2024-03-04T17:39:09.405 DEBUG:boto:Path: /ppfahn-10/...?versionId=null 2024-03-04T17:39:09.405 DEBUG:boto:Data: 2024-03-04T17:39:09.405 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.405 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.405 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.405 DEBUG:boto:Params: {} 2024-03-04T17:39:09.406 DEBUG:boto:Token: None 2024-03-04T17:39:09.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/...?versionId=null 2024-03-04T17:39:09.406 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:q2MOAn+5tkozx/+64zykWkBJBdM= 2024-03-04T17:39:09.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:q2MOAn+5tkozx/+64zykWkBJBdM=', 'Content-Length': '0'} 2024-03-04T17:39:09.409 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004589e013a7860e0f-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.409 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:39:09.409 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:39:09.409 DEBUG:boto:path=/ppfahn-10/...?versionId=null 2024-03-04T17:39:09.409 DEBUG:boto:auth_path=/ppfahn-10/...?versionId=null 2024-03-04T17:39:09.409 DEBUG:boto:Method: GET 2024-03-04T17:39:09.409 DEBUG:boto:Path: /ppfahn-10/...?versionId=null 2024-03-04T17:39:09.410 DEBUG:boto:Data: 2024-03-04T17:39:09.410 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.410 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.410 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.410 DEBUG:boto:Params: {} 2024-03-04T17:39:09.410 DEBUG:boto:Token: None 2024-03-04T17:39:09.410 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/...?versionId=null 2024-03-04T17:39:09.410 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:q2MOAn+5tkozx/+64zykWkBJBdM= 2024-03-04T17:39:09.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:q2MOAn+5tkozx/+64zykWkBJBdM=', 'Content-Length': '0'} 2024-03-04T17:39:09.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034c7e9f70e83bfd5-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.413 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:39:09.413 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:39:09.413 DEBUG:boto:path=/ppfahn-10/...?versionId=null 2024-03-04T17:39:09.413 DEBUG:boto:auth_path=/ppfahn-10/...?versionId=null 2024-03-04T17:39:09.414 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.414 DEBUG:boto:Path: /ppfahn-10/...?versionId=null 2024-03-04T17:39:09.414 DEBUG:boto:Data: 2024-03-04T17:39:09.414 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.414 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.414 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.414 DEBUG:boto:Params: {} 2024-03-04T17:39:09.414 DEBUG:boto:Token: None 2024-03-04T17:39:09.414 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/...?versionId=null 2024-03-04T17:39:09.414 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tsNYmW1tXhSgy3yP8yrwPaHRrww= 2024-03-04T17:39:09.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tsNYmW1tXhSgy3yP8yrwPaHRrww=', 'Content-Length': '0'} 2024-03-04T17:39:09.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000215a41ac53b2a539-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.417 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:39:09.417 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:39:09.417 DEBUG:boto:path=/ppfahn-10/...?versionId=null 2024-03-04T17:39:09.418 DEBUG:boto:auth_path=/ppfahn-10/...?versionId=null 2024-03-04T17:39:09.418 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.418 DEBUG:boto:Path: /ppfahn-10/...?versionId=null 2024-03-04T17:39:09.418 DEBUG:boto:Data: 2024-03-04T17:39:09.418 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.418 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.418 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.418 DEBUG:boto:Params: {} 2024-03-04T17:39:09.418 DEBUG:boto:Token: None 2024-03-04T17:39:09.418 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/...?versionId=null 2024-03-04T17:39:09.419 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tsNYmW1tXhSgy3yP8yrwPaHRrww= 2024-03-04T17:39:09.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tsNYmW1tXhSgy3yP8yrwPaHRrww=', 'Content-Length': '0'} 2024-03-04T17:39:09.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038c34aad57ab1c2c-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:09.422 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:39:09.422 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:39:09.422 DEBUG:boto:path=/ppfahn-10/...?versionId=null 2024-03-04T17:39:09.422 DEBUG:boto:auth_path=/ppfahn-10/...?versionId=null 2024-03-04T17:39:09.422 DEBUG:boto:Method: GET 2024-03-04T17:39:09.422 DEBUG:boto:Path: /ppfahn-10/...?versionId=null 2024-03-04T17:39:09.422 DEBUG:boto:Data: 2024-03-04T17:39:09.422 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.422 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.423 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.423 DEBUG:boto:Params: {} 2024-03-04T17:39:09.423 DEBUG:boto:Token: None 2024-03-04T17:39:09.423 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/...?versionId=null 2024-03-04T17:39:09.423 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:q2MOAn+5tkozx/+64zykWkBJBdM= 2024-03-04T17:39:09.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:q2MOAn+5tkozx/+64zykWkBJBdM=', 'Content-Length': '0'} 2024-03-04T17:39:09.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4864a91c6fdbf19-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.472 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:39:09.472 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:39:09.472 DEBUG:boto:path=/ppfahn-10/...?versionId=null 2024-03-04T17:39:09.472 DEBUG:boto:auth_path=/ppfahn-10/...?versionId=null 2024-03-04T17:39:09.473 DEBUG:boto:Method: GET 2024-03-04T17:39:09.473 DEBUG:boto:Path: /ppfahn-10/...?versionId=null 2024-03-04T17:39:09.473 DEBUG:boto:Data: 2024-03-04T17:39:09.473 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.473 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.473 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.473 DEBUG:boto:Params: {} 2024-03-04T17:39:09.473 DEBUG:boto:Token: None 2024-03-04T17:39:09.473 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/...?versionId=null 2024-03-04T17:39:09.473 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:q2MOAn+5tkozx/+64zykWkBJBdM= 2024-03-04T17:39:09.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:q2MOAn+5tkozx/+64zykWkBJBdM=', 'Content-Length': '0'} 2024-03-04T17:39:09.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d430dab85617835-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.477 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.477 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.477 DEBUG:boto:path=/ppfahn-10/?prefix=... 2024-03-04T17:39:09.477 DEBUG:boto:auth_path=/ppfahn-10/?prefix=... 2024-03-04T17:39:09.477 DEBUG:boto:Method: GET 2024-03-04T17:39:09.477 DEBUG:boto:Path: /ppfahn-10/?prefix=... 2024-03-04T17:39:09.477 DEBUG:boto:Data: 2024-03-04T17:39:09.477 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.477 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.477 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.477 DEBUG:boto:Params: {} 2024-03-04T17:39:09.477 DEBUG:boto:Token: None 2024-03-04T17:39:09.478 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.478 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000695745a71be04b91-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.482 DEBUG:boto:b'ppfahn-10...1000false...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.482 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.483 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.483 DEBUG:boto:path=/ppfahn-10/?prefix=... 2024-03-04T17:39:09.483 DEBUG:boto:auth_path=/ppfahn-10/?prefix=... 2024-03-04T17:39:09.483 DEBUG:boto:Method: GET 2024-03-04T17:39:09.483 DEBUG:boto:Path: /ppfahn-10/?prefix=... 2024-03-04T17:39:09.483 DEBUG:boto:Data: 2024-03-04T17:39:09.483 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.483 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.483 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.483 DEBUG:boto:Params: {} 2024-03-04T17:39:09.483 DEBUG:boto:Token: None 2024-03-04T17:39:09.484 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.484 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ddfe182131e0c0d-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.536 DEBUG:boto:b'ppfahn-10...1000false...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.536 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:39:09.536 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:39:09.536 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.537 DEBUG:boto:Path: /ppfahn-10/... 2024-03-04T17:39:09.537 DEBUG:boto:Data: 2024-03-04T17:39:09.537 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.537 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.537 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.537 DEBUG:boto:Params: {} 2024-03-04T17:39:09.537 DEBUG:boto:Token: None 2024-03-04T17:39:09.537 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/... 2024-03-04T17:39:09.537 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sEnXaUkEwFA3sFzxMx3D76RF+Ko= 2024-03-04T17:39:09.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sEnXaUkEwFA3sFzxMx3D76RF+Ko=', 'Content-Length': '0'} 2024-03-04T17:39:09.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004018ce8b38de1d85-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.541 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:39:09.541 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:39:09.541 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.541 DEBUG:boto:Path: /ppfahn-10/... 2024-03-04T17:39:09.541 DEBUG:boto:Data: 2024-03-04T17:39:09.541 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.541 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.541 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.541 DEBUG:boto:Params: {} 2024-03-04T17:39:09.542 DEBUG:boto:Token: None 2024-03-04T17:39:09.542 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/... 2024-03-04T17:39:09.542 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sEnXaUkEwFA3sFzxMx3D76RF+Ko= 2024-03-04T17:39:09.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sEnXaUkEwFA3sFzxMx3D76RF+Ko=', 'Content-Length': '0'} 2024-03-04T17:39:09.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b54684907c3b0a66-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:09.545 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:39:09.545 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:39:09.545 DEBUG:boto:Method: GET 2024-03-04T17:39:09.545 DEBUG:boto:Path: /ppfahn-10/... 2024-03-04T17:39:09.545 DEBUG:boto:Data: 2024-03-04T17:39:09.545 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.546 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.546 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.546 DEBUG:boto:Params: {} 2024-03-04T17:39:09.546 DEBUG:boto:Token: None 2024-03-04T17:39:09.546 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/... 2024-03-04T17:39:09.546 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uJ31j28UwpTVLMU6d5H7/m94e9I= 2024-03-04T17:39:09.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uJ31j28UwpTVLMU6d5H7/m94e9I=', 'Content-Length': '0'} 2024-03-04T17:39:09.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006072602fe849c766-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.592 DEBUG:boto:path=/ppfahn-10/... 2024-03-04T17:39:09.592 DEBUG:boto:auth_path=/ppfahn-10/... 2024-03-04T17:39:09.592 DEBUG:boto:Method: GET 2024-03-04T17:39:09.592 DEBUG:boto:Path: /ppfahn-10/... 2024-03-04T17:39:09.592 DEBUG:boto:Data: 2024-03-04T17:39:09.592 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.593 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.593 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.593 DEBUG:boto:Params: {} 2024-03-04T17:39:09.593 DEBUG:boto:Token: None 2024-03-04T17:39:09.593 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/... 2024-03-04T17:39:09.593 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uJ31j28UwpTVLMU6d5H7/m94e9I= 2024-03-04T17:39:09.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uJ31j28UwpTVLMU6d5H7/m94e9I=', 'Content-Length': '0'} 2024-03-04T17:39:09.596 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f9ee9d05596a009-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.596 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.596 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.596 DEBUG:boto:path=/ppfahn-10/?prefix=... 2024-03-04T17:39:09.596 DEBUG:boto:auth_path=/ppfahn-10/?prefix=... 2024-03-04T17:39:09.596 DEBUG:boto:Method: GET 2024-03-04T17:39:09.596 DEBUG:boto:Path: /ppfahn-10/?prefix=... 2024-03-04T17:39:09.596 DEBUG:boto:Data: 2024-03-04T17:39:09.597 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.597 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.597 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.597 DEBUG:boto:Params: {} 2024-03-04T17:39:09.597 DEBUG:boto:Token: None 2024-03-04T17:39:09.597 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.597 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3a772a07e12ff8b-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.644 DEBUG:boto:b'ppfahn-10...1000false...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.645 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.645 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.645 DEBUG:boto:path=/ppfahn-10/?prefix=... 2024-03-04T17:39:09.645 DEBUG:boto:auth_path=/ppfahn-10/?prefix=... 2024-03-04T17:39:09.645 DEBUG:boto:Method: GET 2024-03-04T17:39:09.645 DEBUG:boto:Path: /ppfahn-10/?prefix=... 2024-03-04T17:39:09.645 DEBUG:boto:Data: 2024-03-04T17:39:09.645 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.645 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.645 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.645 DEBUG:boto:Params: {} 2024-03-04T17:39:09.646 DEBUG:boto:Token: None 2024-03-04T17:39:09.646 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.646 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c33f1498b7c95657-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.650 DEBUG:boto:b'ppfahn-10...1000false...2024-03-04T17:36:30.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:09.651 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:39:09.651 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:39:09.651 DEBUG:boto:path=/ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.651 DEBUG:boto:auth_path=/ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.651 DEBUG:boto:Method: GET 2024-03-04T17:39:09.651 DEBUG:boto:Path: /ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.651 DEBUG:boto:Data: 2024-03-04T17:39:09.651 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.651 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.651 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.652 DEBUG:boto:Params: {} 2024-03-04T17:39:09.652 DEBUG:boto:Token: None 2024-03-04T17:39:09.652 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.652 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LINfooXE0T0t9/ezboDdMCMAxok= 2024-03-04T17:39:09.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LINfooXE0T0t9/ezboDdMCMAxok=', 'Content-Length': '0'} 2024-03-04T17:39:09.655 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d33c413ae4eb273-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.696 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:39:09.696 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:39:09.696 DEBUG:boto:path=/ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.696 DEBUG:boto:auth_path=/ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.696 DEBUG:boto:Method: GET 2024-03-04T17:39:09.696 DEBUG:boto:Path: /ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.697 DEBUG:boto:Data: 2024-03-04T17:39:09.697 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.697 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.697 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.697 DEBUG:boto:Params: {} 2024-03-04T17:39:09.697 DEBUG:boto:Token: None 2024-03-04T17:39:09.697 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.697 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LINfooXE0T0t9/ezboDdMCMAxok= 2024-03-04T17:39:09.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LINfooXE0T0t9/ezboDdMCMAxok=', 'Content-Length': '0'} 2024-03-04T17:39:09.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c5662348539f173-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.700 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:39:09.700 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:39:09.700 DEBUG:boto:path=/ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.701 DEBUG:boto:auth_path=/ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.701 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.701 DEBUG:boto:Path: /ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.701 DEBUG:boto:Data: 2024-03-04T17:39:09.701 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.701 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.701 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.701 DEBUG:boto:Params: {} 2024-03-04T17:39:09.701 DEBUG:boto:Token: None 2024-03-04T17:39:09.701 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.701 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fQab7HlVzS4wRGztIWvyE7Q3CmU= 2024-03-04T17:39:09.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fQab7HlVzS4wRGztIWvyE7Q3CmU=', 'Content-Length': '0'} 2024-03-04T17:39:09.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edb2f2b0c7ee3e78-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.705 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:39:09.705 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:39:09.705 DEBUG:boto:path=/ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.705 DEBUG:boto:auth_path=/ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.705 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.705 DEBUG:boto:Path: /ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.705 DEBUG:boto:Data: 2024-03-04T17:39:09.705 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.705 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.705 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.705 DEBUG:boto:Params: {} 2024-03-04T17:39:09.706 DEBUG:boto:Token: None 2024-03-04T17:39:09.706 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.706 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fQab7HlVzS4wRGztIWvyE7Q3CmU= 2024-03-04T17:39:09.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fQab7HlVzS4wRGztIWvyE7Q3CmU=', 'Content-Length': '0'} 2024-03-04T17:39:09.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020eeddf540a12ce5-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:09.709 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:39:09.709 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:39:09.709 DEBUG:boto:path=/ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.709 DEBUG:boto:auth_path=/ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.709 DEBUG:boto:Method: GET 2024-03-04T17:39:09.709 DEBUG:boto:Path: /ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.709 DEBUG:boto:Data: 2024-03-04T17:39:09.710 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.710 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.710 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.710 DEBUG:boto:Params: {} 2024-03-04T17:39:09.710 DEBUG:boto:Token: None 2024-03-04T17:39:09.710 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.710 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LINfooXE0T0t9/ezboDdMCMAxok= 2024-03-04T17:39:09.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LINfooXE0T0t9/ezboDdMCMAxok=', 'Content-Length': '0'} 2024-03-04T17:39:09.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008eb2de0ec75b0d6-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.756 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:39:09.756 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:39:09.756 DEBUG:boto:path=/ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.756 DEBUG:boto:auth_path=/ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.756 DEBUG:boto:Method: GET 2024-03-04T17:39:09.756 DEBUG:boto:Path: /ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.757 DEBUG:boto:Data: 2024-03-04T17:39:09.757 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.757 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.757 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.757 DEBUG:boto:Params: {} 2024-03-04T17:39:09.757 DEBUG:boto:Token: None 2024-03-04T17:39:09.757 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o?versionId=null 2024-03-04T17:39:09.757 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LINfooXE0T0t9/ezboDdMCMAxok= 2024-03-04T17:39:09.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LINfooXE0T0t9/ezboDdMCMAxok=', 'Content-Length': '0'} 2024-03-04T17:39:09.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eee6ca67284cfdf5-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.761 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.761 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.761 DEBUG:boto:path=/ppfahn-10/?prefix=.o 2024-03-04T17:39:09.761 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o 2024-03-04T17:39:09.761 DEBUG:boto:Method: GET 2024-03-04T17:39:09.761 DEBUG:boto:Path: /ppfahn-10/?prefix=.o 2024-03-04T17:39:09.761 DEBUG:boto:Data: 2024-03-04T17:39:09.761 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.761 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.761 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.761 DEBUG:boto:Params: {} 2024-03-04T17:39:09.762 DEBUG:boto:Token: None 2024-03-04T17:39:09.762 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.762 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eab338c38df9649-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.766 DEBUG:boto:b'ppfahn-10.o1000false.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.766 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.766 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.766 DEBUG:boto:path=/ppfahn-10/?prefix=.o 2024-03-04T17:39:09.766 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o 2024-03-04T17:39:09.767 DEBUG:boto:Method: GET 2024-03-04T17:39:09.767 DEBUG:boto:Path: /ppfahn-10/?prefix=.o 2024-03-04T17:39:09.767 DEBUG:boto:Data: 2024-03-04T17:39:09.767 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.767 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.767 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.767 DEBUG:boto:Params: {} 2024-03-04T17:39:09.767 DEBUG:boto:Token: None 2024-03-04T17:39:09.767 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.767 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f4f41da3b9d516e-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.812 DEBUG:boto:b'ppfahn-10.o1000false.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.813 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:39:09.813 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:39:09.813 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.813 DEBUG:boto:Path: /ppfahn-10/.o 2024-03-04T17:39:09.813 DEBUG:boto:Data: 2024-03-04T17:39:09.813 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.813 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.813 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.813 DEBUG:boto:Params: {} 2024-03-04T17:39:09.813 DEBUG:boto:Token: None 2024-03-04T17:39:09.814 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o 2024-03-04T17:39:09.814 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rDr9sG190sQzAlcH0XIwpl4EPVQ= 2024-03-04T17:39:09.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rDr9sG190sQzAlcH0XIwpl4EPVQ=', 'Content-Length': '0'} 2024-03-04T17:39:09.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000520eff27e4e89454-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.817 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:39:09.817 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:39:09.817 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.817 DEBUG:boto:Path: /ppfahn-10/.o 2024-03-04T17:39:09.817 DEBUG:boto:Data: 2024-03-04T17:39:09.817 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.817 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.818 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.818 DEBUG:boto:Params: {} 2024-03-04T17:39:09.818 DEBUG:boto:Token: None 2024-03-04T17:39:09.818 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o 2024-03-04T17:39:09.818 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rDr9sG190sQzAlcH0XIwpl4EPVQ= 2024-03-04T17:39:09.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rDr9sG190sQzAlcH0XIwpl4EPVQ=', 'Content-Length': '0'} 2024-03-04T17:39:09.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041b30ccb9d4f1f4f-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:09.821 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:39:09.821 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:39:09.821 DEBUG:boto:Method: GET 2024-03-04T17:39:09.821 DEBUG:boto:Path: /ppfahn-10/.o 2024-03-04T17:39:09.821 DEBUG:boto:Data: 2024-03-04T17:39:09.821 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.821 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.822 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.822 DEBUG:boto:Params: {} 2024-03-04T17:39:09.822 DEBUG:boto:Token: None 2024-03-04T17:39:09.822 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o 2024-03-04T17:39:09.822 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v+fI6CzwW/BMLD34mF3tCwGQJsw= 2024-03-04T17:39:09.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v+fI6CzwW/BMLD34mF3tCwGQJsw=', 'Content-Length': '0'} 2024-03-04T17:39:09.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c799e886f47da940-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.872 DEBUG:boto:path=/ppfahn-10/.o 2024-03-04T17:39:09.872 DEBUG:boto:auth_path=/ppfahn-10/.o 2024-03-04T17:39:09.872 DEBUG:boto:Method: GET 2024-03-04T17:39:09.873 DEBUG:boto:Path: /ppfahn-10/.o 2024-03-04T17:39:09.873 DEBUG:boto:Data: 2024-03-04T17:39:09.873 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.873 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.873 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.873 DEBUG:boto:Params: {} 2024-03-04T17:39:09.873 DEBUG:boto:Token: None 2024-03-04T17:39:09.873 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o 2024-03-04T17:39:09.873 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v+fI6CzwW/BMLD34mF3tCwGQJsw= 2024-03-04T17:39:09.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v+fI6CzwW/BMLD34mF3tCwGQJsw=', 'Content-Length': '0'} 2024-03-04T17:39:09.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000979b08bca84ba4af-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.877 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.877 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.877 DEBUG:boto:path=/ppfahn-10/?prefix=.o 2024-03-04T17:39:09.877 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o 2024-03-04T17:39:09.877 DEBUG:boto:Method: GET 2024-03-04T17:39:09.877 DEBUG:boto:Path: /ppfahn-10/?prefix=.o 2024-03-04T17:39:09.877 DEBUG:boto:Data: 2024-03-04T17:39:09.877 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.877 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.877 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.877 DEBUG:boto:Params: {} 2024-03-04T17:39:09.878 DEBUG:boto:Token: None 2024-03-04T17:39:09.878 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.878 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f345f1611cdcaba-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.882 DEBUG:boto:b'ppfahn-10.o1000false.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.882 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:09.882 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:09.882 DEBUG:boto:path=/ppfahn-10/?prefix=.o 2024-03-04T17:39:09.882 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o 2024-03-04T17:39:09.883 DEBUG:boto:Method: GET 2024-03-04T17:39:09.883 DEBUG:boto:Path: /ppfahn-10/?prefix=.o 2024-03-04T17:39:09.883 DEBUG:boto:Data: 2024-03-04T17:39:09.883 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.883 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.883 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.883 DEBUG:boto:Params: {} 2024-03-04T17:39:09.883 DEBUG:boto:Token: None 2024-03-04T17:39:09.883 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/ 2024-03-04T17:39:09.883 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50= 2024-03-04T17:39:09.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OL3JCNyZNY6/M7PZg6lBcRNEx50=', 'Content-Length': '0'} 2024-03-04T17:39:09.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bb12abb0a1665e2-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.888 DEBUG:boto:b'ppfahn-10.o1000false.o2024-03-04T17:36:30.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:09.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:09.889 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:39:09.889 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:39:09.889 DEBUG:boto:path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.889 DEBUG:boto:auth_path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.889 DEBUG:boto:Method: GET 2024-03-04T17:39:09.889 DEBUG:boto:Path: /ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.889 DEBUG:boto:Data: 2024-03-04T17:39:09.889 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.889 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.889 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.890 DEBUG:boto:Params: {} 2024-03-04T17:39:09.890 DEBUG:boto:Token: None 2024-03-04T17:39:09.890 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.890 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M3CvQL7eh2lQ9cUqq1RbUgyx6Vw= 2024-03-04T17:39:09.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M3CvQL7eh2lQ9cUqq1RbUgyx6Vw=', 'Content-Length': '0'} 2024-03-04T17:39:09.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003d69116e844b996-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.893 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:39:09.893 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:39:09.894 DEBUG:boto:path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.894 DEBUG:boto:auth_path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.894 DEBUG:boto:Method: GET 2024-03-04T17:39:09.894 DEBUG:boto:Path: /ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.894 DEBUG:boto:Data: 2024-03-04T17:39:09.894 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.894 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.894 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.894 DEBUG:boto:Params: {} 2024-03-04T17:39:09.894 DEBUG:boto:Token: None 2024-03-04T17:39:09.894 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.895 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M3CvQL7eh2lQ9cUqq1RbUgyx6Vw= 2024-03-04T17:39:09.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M3CvQL7eh2lQ9cUqq1RbUgyx6Vw=', 'Content-Length': '0'} 2024-03-04T17:39:09.897 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1591c4cfd40ec12-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.940 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:39:09.940 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:39:09.940 DEBUG:boto:path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.940 DEBUG:boto:auth_path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.941 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.941 DEBUG:boto:Path: /ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.941 DEBUG:boto:Data: 2024-03-04T17:39:09.941 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.941 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.941 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.941 DEBUG:boto:Params: {} 2024-03-04T17:39:09.941 DEBUG:boto:Token: None 2024-03-04T17:39:09.941 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.941 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2yylMH5eCJkMdK6rHsGzJG+/fNA= 2024-03-04T17:39:09.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2yylMH5eCJkMdK6rHsGzJG+/fNA=', 'Content-Length': '0'} 2024-03-04T17:39:09.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000267e6c65d204b59-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.945 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:39:09.945 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:39:09.945 DEBUG:boto:path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.945 DEBUG:boto:auth_path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.945 DEBUG:boto:Method: HEAD 2024-03-04T17:39:09.945 DEBUG:boto:Path: /ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.945 DEBUG:boto:Data: 2024-03-04T17:39:09.945 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.945 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.945 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.946 DEBUG:boto:Params: {} 2024-03-04T17:39:09.946 DEBUG:boto:Token: None 2024-03-04T17:39:09.946 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.946 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2yylMH5eCJkMdK6rHsGzJG+/fNA= 2024-03-04T17:39:09.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2yylMH5eCJkMdK6rHsGzJG+/fNA=', 'Content-Length': '0'} 2024-03-04T17:39:09.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044057caf78652599-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:09.949 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:39:09.949 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:39:09.949 DEBUG:boto:path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.949 DEBUG:boto:auth_path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.950 DEBUG:boto:Method: GET 2024-03-04T17:39:09.950 DEBUG:boto:Path: /ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.950 DEBUG:boto:Data: 2024-03-04T17:39:09.950 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.950 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:09.950 DEBUG:boto:Port: 8001 2024-03-04T17:39:09.950 DEBUG:boto:Params: {} 2024-03-04T17:39:09.950 DEBUG:boto:Token: None 2024-03-04T17:39:09.950 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.950 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M3CvQL7eh2lQ9cUqq1RbUgyx6Vw= 2024-03-04T17:39:09.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M3CvQL7eh2lQ9cUqq1RbUgyx6Vw=', 'Content-Length': '0'} 2024-03-04T17:39:09.953 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014d431ce1ad3268f-0065e6073d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:09.996 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:39:09.996 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:39:09.996 DEBUG:boto:path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.996 DEBUG:boto:auth_path=/ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.996 DEBUG:boto:Method: GET 2024-03-04T17:39:09.996 DEBUG:boto:Path: /ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.996 DEBUG:boto:Data: 2024-03-04T17:39:09.997 DEBUG:boto:Headers: {} 2024-03-04T17:39:09.997 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:09.997 DEBUG:boto:Port: 8000 2024-03-04T17:39:09.997 DEBUG:boto:Params: {} 2024-03-04T17:39:09.997 DEBUG:boto:Token: None 2024-03-04T17:39:09.997 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:09 GMT /ppfahn-10/.o.?versionId=null 2024-03-04T17:39:09.997 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M3CvQL7eh2lQ9cUqq1RbUgyx6Vw= 2024-03-04T17:39:09.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:09 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M3CvQL7eh2lQ9cUqq1RbUgyx6Vw=', 'Content-Length': '0'} 2024-03-04T17:39:10.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006986d435ff92ae18-0065e6073d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.001 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.001 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.001 DEBUG:boto:path=/ppfahn-10/?prefix=.o. 2024-03-04T17:39:10.001 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o. 2024-03-04T17:39:10.001 DEBUG:boto:Method: GET 2024-03-04T17:39:10.001 DEBUG:boto:Path: /ppfahn-10/?prefix=.o. 2024-03-04T17:39:10.001 DEBUG:boto:Data: 2024-03-04T17:39:10.001 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.002 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.002 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.002 DEBUG:boto:Params: {} 2024-03-04T17:39:10.002 DEBUG:boto:Token: None 2024-03-04T17:39:10.002 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.002 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e6363f364e248c5-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.006 DEBUG:boto:b'ppfahn-10.o.1000false.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.006 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.006 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.007 DEBUG:boto:path=/ppfahn-10/?prefix=.o. 2024-03-04T17:39:10.007 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o. 2024-03-04T17:39:10.007 DEBUG:boto:Method: GET 2024-03-04T17:39:10.007 DEBUG:boto:Path: /ppfahn-10/?prefix=.o. 2024-03-04T17:39:10.007 DEBUG:boto:Data: 2024-03-04T17:39:10.007 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.007 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.007 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.007 DEBUG:boto:Params: {} 2024-03-04T17:39:10.007 DEBUG:boto:Token: None 2024-03-04T17:39:10.008 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.008 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ad197b261388b08-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.012 DEBUG:boto:b'ppfahn-10.o.1000false.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.012 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:39:10.012 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:39:10.012 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.013 DEBUG:boto:Path: /ppfahn-10/.o. 2024-03-04T17:39:10.013 DEBUG:boto:Data: 2024-03-04T17:39:10.013 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.013 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.013 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.013 DEBUG:boto:Params: {} 2024-03-04T17:39:10.013 DEBUG:boto:Token: None 2024-03-04T17:39:10.013 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/.o. 2024-03-04T17:39:10.013 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6hEa7jCy3HtioBtzcL3ES7hKFRw= 2024-03-04T17:39:10.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6hEa7jCy3HtioBtzcL3ES7hKFRw=', 'Content-Length': '0'} 2024-03-04T17:39:10.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a878e0ac8debf15c-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.016 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:39:10.016 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:39:10.016 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.017 DEBUG:boto:Path: /ppfahn-10/.o. 2024-03-04T17:39:10.017 DEBUG:boto:Data: 2024-03-04T17:39:10.017 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.017 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.017 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.017 DEBUG:boto:Params: {} 2024-03-04T17:39:10.017 DEBUG:boto:Token: None 2024-03-04T17:39:10.017 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/.o. 2024-03-04T17:39:10.017 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6hEa7jCy3HtioBtzcL3ES7hKFRw= 2024-03-04T17:39:10.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6hEa7jCy3HtioBtzcL3ES7hKFRw=', 'Content-Length': '0'} 2024-03-04T17:39:10.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6e9072e1e0b661a-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:10.020 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:39:10.020 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:39:10.020 DEBUG:boto:Method: GET 2024-03-04T17:39:10.020 DEBUG:boto:Path: /ppfahn-10/.o. 2024-03-04T17:39:10.020 DEBUG:boto:Data: 2024-03-04T17:39:10.020 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.021 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.021 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.021 DEBUG:boto:Params: {} 2024-03-04T17:39:10.021 DEBUG:boto:Token: None 2024-03-04T17:39:10.021 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/.o. 2024-03-04T17:39:10.021 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2Bi7G8ZiYD1ck2tfaxONC6KYl8= 2024-03-04T17:39:10.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2Bi7G8ZiYD1ck2tfaxONC6KYl8=', 'Content-Length': '0'} 2024-03-04T17:39:10.024 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000262ab3b2624cb1e5-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.064 DEBUG:boto:path=/ppfahn-10/.o. 2024-03-04T17:39:10.064 DEBUG:boto:auth_path=/ppfahn-10/.o. 2024-03-04T17:39:10.064 DEBUG:boto:Method: GET 2024-03-04T17:39:10.064 DEBUG:boto:Path: /ppfahn-10/.o. 2024-03-04T17:39:10.064 DEBUG:boto:Data: 2024-03-04T17:39:10.064 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.065 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.065 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.065 DEBUG:boto:Params: {} 2024-03-04T17:39:10.065 DEBUG:boto:Token: None 2024-03-04T17:39:10.065 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/.o. 2024-03-04T17:39:10.065 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2Bi7G8ZiYD1ck2tfaxONC6KYl8= 2024-03-04T17:39:10.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2Bi7G8ZiYD1ck2tfaxONC6KYl8=', 'Content-Length': '0'} 2024-03-04T17:39:10.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078f38abb9665d101-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.068 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.068 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.068 DEBUG:boto:path=/ppfahn-10/?prefix=.o. 2024-03-04T17:39:10.068 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o. 2024-03-04T17:39:10.068 DEBUG:boto:Method: GET 2024-03-04T17:39:10.069 DEBUG:boto:Path: /ppfahn-10/?prefix=.o. 2024-03-04T17:39:10.069 DEBUG:boto:Data: 2024-03-04T17:39:10.069 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.069 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.069 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.069 DEBUG:boto:Params: {} 2024-03-04T17:39:10.069 DEBUG:boto:Token: None 2024-03-04T17:39:10.069 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.069 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1d4722bfdd837e7-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.074 DEBUG:boto:b'ppfahn-10.o.1000false.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.074 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.074 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.074 DEBUG:boto:path=/ppfahn-10/?prefix=.o. 2024-03-04T17:39:10.074 DEBUG:boto:auth_path=/ppfahn-10/?prefix=.o. 2024-03-04T17:39:10.074 DEBUG:boto:Method: GET 2024-03-04T17:39:10.075 DEBUG:boto:Path: /ppfahn-10/?prefix=.o. 2024-03-04T17:39:10.075 DEBUG:boto:Data: 2024-03-04T17:39:10.075 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.075 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.075 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.075 DEBUG:boto:Params: {} 2024-03-04T17:39:10.075 DEBUG:boto:Token: None 2024-03-04T17:39:10.075 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.075 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027e41d1542d2d348-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.079 DEBUG:boto:b'ppfahn-10.o.1000false.o.2024-03-04T17:36:30.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:10.080 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:39:10.080 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:39:10.080 DEBUG:boto:path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.080 DEBUG:boto:auth_path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.080 DEBUG:boto:Method: GET 2024-03-04T17:39:10.080 DEBUG:boto:Path: /ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.080 DEBUG:boto:Data: 2024-03-04T17:39:10.080 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.080 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.080 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.081 DEBUG:boto:Params: {} 2024-03-04T17:39:10.081 DEBUG:boto:Token: None 2024-03-04T17:39:10.081 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.081 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RLzt6XgZJv9PRF+wf6TucsG1irk= 2024-03-04T17:39:10.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RLzt6XgZJv9PRF+wf6TucsG1irk=', 'Content-Length': '0'} 2024-03-04T17:39:10.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006860cdf535619b20-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.124 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:39:10.124 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:39:10.124 DEBUG:boto:path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.124 DEBUG:boto:auth_path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.125 DEBUG:boto:Method: GET 2024-03-04T17:39:10.125 DEBUG:boto:Path: /ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.125 DEBUG:boto:Data: 2024-03-04T17:39:10.125 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.125 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.125 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.125 DEBUG:boto:Params: {} 2024-03-04T17:39:10.125 DEBUG:boto:Token: None 2024-03-04T17:39:10.125 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.125 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RLzt6XgZJv9PRF+wf6TucsG1irk= 2024-03-04T17:39:10.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RLzt6XgZJv9PRF+wf6TucsG1irk=', 'Content-Length': '0'} 2024-03-04T17:39:10.128 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f07d67b0b0531355-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.129 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:39:10.129 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:39:10.129 DEBUG:boto:path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.129 DEBUG:boto:auth_path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.129 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.129 DEBUG:boto:Path: /ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.129 DEBUG:boto:Data: 2024-03-04T17:39:10.129 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.129 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.129 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.130 DEBUG:boto:Params: {} 2024-03-04T17:39:10.130 DEBUG:boto:Token: None 2024-03-04T17:39:10.130 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.130 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LAXR7h6Z2fpicSVbFo5SiMhJzaY= 2024-03-04T17:39:10.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LAXR7h6Z2fpicSVbFo5SiMhJzaY=', 'Content-Length': '0'} 2024-03-04T17:39:10.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7389c774e71deb4-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.133 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:39:10.133 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:39:10.133 DEBUG:boto:path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.133 DEBUG:boto:auth_path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.133 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.133 DEBUG:boto:Path: /ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.133 DEBUG:boto:Data: 2024-03-04T17:39:10.134 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.134 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.134 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.134 DEBUG:boto:Params: {} 2024-03-04T17:39:10.134 DEBUG:boto:Token: None 2024-03-04T17:39:10.134 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.134 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LAXR7h6Z2fpicSVbFo5SiMhJzaY= 2024-03-04T17:39:10.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LAXR7h6Z2fpicSVbFo5SiMhJzaY=', 'Content-Length': '0'} 2024-03-04T17:39:10.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085b8430cf1955d76-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:10.137 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:39:10.137 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:39:10.137 DEBUG:boto:path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.137 DEBUG:boto:auth_path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.138 DEBUG:boto:Method: GET 2024-03-04T17:39:10.138 DEBUG:boto:Path: /ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.138 DEBUG:boto:Data: 2024-03-04T17:39:10.138 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.138 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.138 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.138 DEBUG:boto:Params: {} 2024-03-04T17:39:10.138 DEBUG:boto:Token: None 2024-03-04T17:39:10.138 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.138 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RLzt6XgZJv9PRF+wf6TucsG1irk= 2024-03-04T17:39:10.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RLzt6XgZJv9PRF+wf6TucsG1irk=', 'Content-Length': '0'} 2024-03-04T17:39:10.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043af155ca88bce42-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.188 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:39:10.188 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:39:10.188 DEBUG:boto:path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.188 DEBUG:boto:auth_path=/ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.188 DEBUG:boto:Method: GET 2024-03-04T17:39:10.188 DEBUG:boto:Path: /ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.189 DEBUG:boto:Data: 2024-03-04T17:39:10.189 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.189 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.189 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.189 DEBUG:boto:Params: {} 2024-03-04T17:39:10.189 DEBUG:boto:Token: None 2024-03-04T17:39:10.189 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/%3A?versionId=null 2024-03-04T17:39:10.189 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RLzt6XgZJv9PRF+wf6TucsG1irk= 2024-03-04T17:39:10.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RLzt6XgZJv9PRF+wf6TucsG1irk=', 'Content-Length': '0'} 2024-03-04T17:39:10.192 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad53137e9d001d0a-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.192 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.192 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.193 DEBUG:boto:path=/ppfahn-10/?prefix=%3A 2024-03-04T17:39:10.193 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%3A 2024-03-04T17:39:10.193 DEBUG:boto:Method: GET 2024-03-04T17:39:10.193 DEBUG:boto:Path: /ppfahn-10/?prefix=%3A 2024-03-04T17:39:10.193 DEBUG:boto:Data: 2024-03-04T17:39:10.193 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.193 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.193 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.193 DEBUG:boto:Params: {} 2024-03-04T17:39:10.193 DEBUG:boto:Token: None 2024-03-04T17:39:10.193 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.194 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b72b75efafecb073-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.198 DEBUG:boto:b'ppfahn-10:1000false:2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.198 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.198 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.198 DEBUG:boto:path=/ppfahn-10/?prefix=%3A 2024-03-04T17:39:10.198 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%3A 2024-03-04T17:39:10.199 DEBUG:boto:Method: GET 2024-03-04T17:39:10.199 DEBUG:boto:Path: /ppfahn-10/?prefix=%3A 2024-03-04T17:39:10.199 DEBUG:boto:Data: 2024-03-04T17:39:10.199 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.199 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.199 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.199 DEBUG:boto:Params: {} 2024-03-04T17:39:10.199 DEBUG:boto:Token: None 2024-03-04T17:39:10.199 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.199 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009b13de2ffe75835-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.244 DEBUG:boto:b'ppfahn-10:1000false:2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.244 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:39:10.244 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:39:10.245 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.245 DEBUG:boto:Path: /ppfahn-10/%3A 2024-03-04T17:39:10.245 DEBUG:boto:Data: 2024-03-04T17:39:10.245 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.245 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.245 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.245 DEBUG:boto:Params: {} 2024-03-04T17:39:10.245 DEBUG:boto:Token: None 2024-03-04T17:39:10.245 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/%3A 2024-03-04T17:39:10.245 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3fIRV9fle7+14gL6s1bpXdTIcoI= 2024-03-04T17:39:10.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3fIRV9fle7+14gL6s1bpXdTIcoI=', 'Content-Length': '0'} 2024-03-04T17:39:10.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1d79f911eff0cb4-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.248 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:39:10.249 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:39:10.249 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.249 DEBUG:boto:Path: /ppfahn-10/%3A 2024-03-04T17:39:10.249 DEBUG:boto:Data: 2024-03-04T17:39:10.249 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.249 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.249 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.249 DEBUG:boto:Params: {} 2024-03-04T17:39:10.249 DEBUG:boto:Token: None 2024-03-04T17:39:10.249 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/%3A 2024-03-04T17:39:10.250 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3fIRV9fle7+14gL6s1bpXdTIcoI= 2024-03-04T17:39:10.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3fIRV9fle7+14gL6s1bpXdTIcoI=', 'Content-Length': '0'} 2024-03-04T17:39:10.252 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e54fcedc1ff478f-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:10.253 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:39:10.253 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:39:10.253 DEBUG:boto:Method: GET 2024-03-04T17:39:10.253 DEBUG:boto:Path: /ppfahn-10/%3A 2024-03-04T17:39:10.253 DEBUG:boto:Data: 2024-03-04T17:39:10.253 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.253 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.253 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.253 DEBUG:boto:Params: {} 2024-03-04T17:39:10.254 DEBUG:boto:Token: None 2024-03-04T17:39:10.254 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/%3A 2024-03-04T17:39:10.254 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f+Tllbkq5k8w4E2MjrTRGj0XyEU= 2024-03-04T17:39:10.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f+Tllbkq5k8w4E2MjrTRGj0XyEU=', 'Content-Length': '0'} 2024-03-04T17:39:10.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f53988df53c7d7b-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.300 DEBUG:boto:path=/ppfahn-10/%3A 2024-03-04T17:39:10.300 DEBUG:boto:auth_path=/ppfahn-10/%3A 2024-03-04T17:39:10.300 DEBUG:boto:Method: GET 2024-03-04T17:39:10.300 DEBUG:boto:Path: /ppfahn-10/%3A 2024-03-04T17:39:10.300 DEBUG:boto:Data: 2024-03-04T17:39:10.300 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.300 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.301 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.301 DEBUG:boto:Params: {} 2024-03-04T17:39:10.301 DEBUG:boto:Token: None 2024-03-04T17:39:10.301 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/%3A 2024-03-04T17:39:10.301 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f+Tllbkq5k8w4E2MjrTRGj0XyEU= 2024-03-04T17:39:10.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f+Tllbkq5k8w4E2MjrTRGj0XyEU=', 'Content-Length': '0'} 2024-03-04T17:39:10.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac87e23ee810615f-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.304 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.304 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.304 DEBUG:boto:path=/ppfahn-10/?prefix=%3A 2024-03-04T17:39:10.304 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%3A 2024-03-04T17:39:10.304 DEBUG:boto:Method: GET 2024-03-04T17:39:10.305 DEBUG:boto:Path: /ppfahn-10/?prefix=%3A 2024-03-04T17:39:10.305 DEBUG:boto:Data: 2024-03-04T17:39:10.305 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.305 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.305 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.305 DEBUG:boto:Params: {} 2024-03-04T17:39:10.305 DEBUG:boto:Token: None 2024-03-04T17:39:10.305 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.305 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f035a5a5e3499c83-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.352 DEBUG:boto:b'ppfahn-10:1000false:2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.352 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.352 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.353 DEBUG:boto:path=/ppfahn-10/?prefix=%3A 2024-03-04T17:39:10.353 DEBUG:boto:auth_path=/ppfahn-10/?prefix=%3A 2024-03-04T17:39:10.353 DEBUG:boto:Method: GET 2024-03-04T17:39:10.353 DEBUG:boto:Path: /ppfahn-10/?prefix=%3A 2024-03-04T17:39:10.353 DEBUG:boto:Data: 2024-03-04T17:39:10.353 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.353 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.353 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.353 DEBUG:boto:Params: {} 2024-03-04T17:39:10.353 DEBUG:boto:Token: None 2024-03-04T17:39:10.354 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.354 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015423c0789dd881f-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.359 DEBUG:boto:b'ppfahn-10:1000false:2024-03-04T17:36:30.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:10.360 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:39:10.360 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:39:10.360 DEBUG:boto:path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.360 DEBUG:boto:auth_path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.360 DEBUG:boto:Method: GET 2024-03-04T17:39:10.360 DEBUG:boto:Path: /ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.360 DEBUG:boto:Data: 2024-03-04T17:39:10.360 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.361 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.361 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.361 DEBUG:boto:Params: {} 2024-03-04T17:39:10.361 DEBUG:boto:Token: None 2024-03-04T17:39:10.361 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.361 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VohZmJ/YjQ45HSAorGrFMcck/Bs= 2024-03-04T17:39:10.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VohZmJ/YjQ45HSAorGrFMcck/Bs=', 'Content-Length': '0'} 2024-03-04T17:39:10.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000585a3b27365bb124-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.412 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:39:10.412 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:39:10.412 DEBUG:boto:path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.412 DEBUG:boto:auth_path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.413 DEBUG:boto:Method: GET 2024-03-04T17:39:10.413 DEBUG:boto:Path: /ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.413 DEBUG:boto:Data: 2024-03-04T17:39:10.413 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.413 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.413 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.413 DEBUG:boto:Params: {} 2024-03-04T17:39:10.413 DEBUG:boto:Token: None 2024-03-04T17:39:10.413 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.414 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VohZmJ/YjQ45HSAorGrFMcck/Bs= 2024-03-04T17:39:10.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VohZmJ/YjQ45HSAorGrFMcck/Bs=', 'Content-Length': '0'} 2024-03-04T17:39:10.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009204818a8399b2da-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.417 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:39:10.417 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:39:10.417 DEBUG:boto:path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.417 DEBUG:boto:auth_path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.417 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.417 DEBUG:boto:Path: /ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.417 DEBUG:boto:Data: 2024-03-04T17:39:10.417 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.418 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.418 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.418 DEBUG:boto:Params: {} 2024-03-04T17:39:10.418 DEBUG:boto:Token: None 2024-03-04T17:39:10.418 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.418 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wIFZp5WUNWtBiPzCbiwiV7vCawo= 2024-03-04T17:39:10.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wIFZp5WUNWtBiPzCbiwiV7vCawo=', 'Content-Length': '0'} 2024-03-04T17:39:10.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000066abdb579eeff32-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.421 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:39:10.422 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:39:10.422 DEBUG:boto:path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.422 DEBUG:boto:auth_path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.422 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.422 DEBUG:boto:Path: /ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.422 DEBUG:boto:Data: 2024-03-04T17:39:10.422 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.422 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.422 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.422 DEBUG:boto:Params: {} 2024-03-04T17:39:10.422 DEBUG:boto:Token: None 2024-03-04T17:39:10.423 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.423 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wIFZp5WUNWtBiPzCbiwiV7vCawo= 2024-03-04T17:39:10.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wIFZp5WUNWtBiPzCbiwiV7vCawo=', 'Content-Length': '0'} 2024-03-04T17:39:10.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5d5a3e3e2a1d39c-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:10.426 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:39:10.426 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:39:10.426 DEBUG:boto:path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.426 DEBUG:boto:auth_path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.427 DEBUG:boto:Method: GET 2024-03-04T17:39:10.427 DEBUG:boto:Path: /ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.427 DEBUG:boto:Data: 2024-03-04T17:39:10.427 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.427 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.427 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.427 DEBUG:boto:Params: {} 2024-03-04T17:39:10.427 DEBUG:boto:Token: None 2024-03-04T17:39:10.427 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.427 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VohZmJ/YjQ45HSAorGrFMcck/Bs= 2024-03-04T17:39:10.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VohZmJ/YjQ45HSAorGrFMcck/Bs=', 'Content-Length': '0'} 2024-03-04T17:39:10.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c30c4b5fac2300a9-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.472 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:39:10.472 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:39:10.473 DEBUG:boto:path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.473 DEBUG:boto:auth_path=/ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.473 DEBUG:boto:Method: GET 2024-03-04T17:39:10.473 DEBUG:boto:Path: /ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.473 DEBUG:boto:Data: 2024-03-04T17:39:10.473 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.473 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.473 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.473 DEBUG:boto:Params: {} 2024-03-04T17:39:10.473 DEBUG:boto:Token: None 2024-03-04T17:39:10.473 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/_myobj?versionId=null 2024-03-04T17:39:10.474 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VohZmJ/YjQ45HSAorGrFMcck/Bs= 2024-03-04T17:39:10.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VohZmJ/YjQ45HSAorGrFMcck/Bs=', 'Content-Length': '0'} 2024-03-04T17:39:10.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1797769153a964b-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.477 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.477 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.477 DEBUG:boto:path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:39:10.477 DEBUG:boto:auth_path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:39:10.477 DEBUG:boto:Method: GET 2024-03-04T17:39:10.477 DEBUG:boto:Path: /ppfahn-10/?prefix=_myobj 2024-03-04T17:39:10.477 DEBUG:boto:Data: 2024-03-04T17:39:10.477 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.477 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.478 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.478 DEBUG:boto:Params: {} 2024-03-04T17:39:10.478 DEBUG:boto:Token: None 2024-03-04T17:39:10.478 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.478 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb4f5c6ac13e4540-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.483 DEBUG:boto:b'ppfahn-10_myobj1000false_myobj2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.483 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.483 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.483 DEBUG:boto:path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:39:10.483 DEBUG:boto:auth_path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:39:10.483 DEBUG:boto:Method: GET 2024-03-04T17:39:10.483 DEBUG:boto:Path: /ppfahn-10/?prefix=_myobj 2024-03-04T17:39:10.484 DEBUG:boto:Data: 2024-03-04T17:39:10.484 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.484 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.484 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.484 DEBUG:boto:Params: {} 2024-03-04T17:39:10.484 DEBUG:boto:Token: None 2024-03-04T17:39:10.484 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.484 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eb39120a32be59f-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.532 DEBUG:boto:b'ppfahn-10_myobj1000false_myobj2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.532 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:39:10.532 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:39:10.533 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.533 DEBUG:boto:Path: /ppfahn-10/_myobj 2024-03-04T17:39:10.533 DEBUG:boto:Data: 2024-03-04T17:39:10.533 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.533 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.533 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.533 DEBUG:boto:Params: {} 2024-03-04T17:39:10.533 DEBUG:boto:Token: None 2024-03-04T17:39:10.533 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/_myobj 2024-03-04T17:39:10.533 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2FgyyIpF31M9H/P/QtZFeY9Bt+k= 2024-03-04T17:39:10.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2FgyyIpF31M9H/P/QtZFeY9Bt+k=', 'Content-Length': '0'} 2024-03-04T17:39:10.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbaae798b3ee1182-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.536 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:39:10.537 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:39:10.537 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.537 DEBUG:boto:Path: /ppfahn-10/_myobj 2024-03-04T17:39:10.537 DEBUG:boto:Data: 2024-03-04T17:39:10.537 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.537 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.537 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.537 DEBUG:boto:Params: {} 2024-03-04T17:39:10.537 DEBUG:boto:Token: None 2024-03-04T17:39:10.537 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/_myobj 2024-03-04T17:39:10.538 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2FgyyIpF31M9H/P/QtZFeY9Bt+k= 2024-03-04T17:39:10.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2FgyyIpF31M9H/P/QtZFeY9Bt+k=', 'Content-Length': '0'} 2024-03-04T17:39:10.540 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051b2583ee0b7199b-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:10.541 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:39:10.541 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:39:10.541 DEBUG:boto:Method: GET 2024-03-04T17:39:10.541 DEBUG:boto:Path: /ppfahn-10/_myobj 2024-03-04T17:39:10.541 DEBUG:boto:Data: 2024-03-04T17:39:10.541 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.541 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.541 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.541 DEBUG:boto:Params: {} 2024-03-04T17:39:10.541 DEBUG:boto:Token: None 2024-03-04T17:39:10.541 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/_myobj 2024-03-04T17:39:10.542 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Ah1gzpdR7+fpFz7amaybtf+vz/0= 2024-03-04T17:39:10.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Ah1gzpdR7+fpFz7amaybtf+vz/0=', 'Content-Length': '0'} 2024-03-04T17:39:10.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b04c1fc1ea55a63-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.588 DEBUG:boto:path=/ppfahn-10/_myobj 2024-03-04T17:39:10.588 DEBUG:boto:auth_path=/ppfahn-10/_myobj 2024-03-04T17:39:10.588 DEBUG:boto:Method: GET 2024-03-04T17:39:10.588 DEBUG:boto:Path: /ppfahn-10/_myobj 2024-03-04T17:39:10.588 DEBUG:boto:Data: 2024-03-04T17:39:10.588 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.589 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.589 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.589 DEBUG:boto:Params: {} 2024-03-04T17:39:10.589 DEBUG:boto:Token: None 2024-03-04T17:39:10.589 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/_myobj 2024-03-04T17:39:10.589 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Ah1gzpdR7+fpFz7amaybtf+vz/0= 2024-03-04T17:39:10.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Ah1gzpdR7+fpFz7amaybtf+vz/0=', 'Content-Length': '0'} 2024-03-04T17:39:10.592 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000110519f887c3fe70-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.592 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.592 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.592 DEBUG:boto:path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:39:10.592 DEBUG:boto:auth_path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:39:10.593 DEBUG:boto:Method: GET 2024-03-04T17:39:10.593 DEBUG:boto:Path: /ppfahn-10/?prefix=_myobj 2024-03-04T17:39:10.593 DEBUG:boto:Data: 2024-03-04T17:39:10.593 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.593 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.593 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.593 DEBUG:boto:Params: {} 2024-03-04T17:39:10.593 DEBUG:boto:Token: None 2024-03-04T17:39:10.593 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.594 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017abf5015614f0dd-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.640 DEBUG:boto:b'ppfahn-10_myobj1000false_myobj2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.641 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.641 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.641 DEBUG:boto:path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:39:10.641 DEBUG:boto:auth_path=/ppfahn-10/?prefix=_myobj 2024-03-04T17:39:10.641 DEBUG:boto:Method: GET 2024-03-04T17:39:10.641 DEBUG:boto:Path: /ppfahn-10/?prefix=_myobj 2024-03-04T17:39:10.641 DEBUG:boto:Data: 2024-03-04T17:39:10.641 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.641 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.641 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.641 DEBUG:boto:Params: {} 2024-03-04T17:39:10.642 DEBUG:boto:Token: None 2024-03-04T17:39:10.642 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.642 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c9222e18a18df3b-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.646 DEBUG:boto:b'ppfahn-10_myobj1000false_myobj2024-03-04T17:36:30.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:10.646 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:39:10.646 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:39:10.646 DEBUG:boto:path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.647 DEBUG:boto:auth_path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.647 DEBUG:boto:Method: GET 2024-03-04T17:39:10.647 DEBUG:boto:Path: /ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.647 DEBUG:boto:Data: 2024-03-04T17:39:10.647 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.647 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.647 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.647 DEBUG:boto:Params: {} 2024-03-04T17:39:10.647 DEBUG:boto:Token: None 2024-03-04T17:39:10.647 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.648 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xSJOfM9m2kHtyBTJAPUSG8Q07Pk= 2024-03-04T17:39:10.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xSJOfM9m2kHtyBTJAPUSG8Q07Pk=', 'Content-Length': '0'} 2024-03-04T17:39:10.651 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db28373dede82335-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.692 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:39:10.692 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:39:10.692 DEBUG:boto:path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.692 DEBUG:boto:auth_path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.692 DEBUG:boto:Method: GET 2024-03-04T17:39:10.692 DEBUG:boto:Path: /ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.693 DEBUG:boto:Data: 2024-03-04T17:39:10.693 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.693 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.693 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.693 DEBUG:boto:Params: {} 2024-03-04T17:39:10.693 DEBUG:boto:Token: None 2024-03-04T17:39:10.693 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.693 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xSJOfM9m2kHtyBTJAPUSG8Q07Pk= 2024-03-04T17:39:10.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xSJOfM9m2kHtyBTJAPUSG8Q07Pk=', 'Content-Length': '0'} 2024-03-04T17:39:10.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d90f49239101703-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.696 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:39:10.696 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:39:10.697 DEBUG:boto:path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.697 DEBUG:boto:auth_path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.697 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.697 DEBUG:boto:Path: /ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.697 DEBUG:boto:Data: 2024-03-04T17:39:10.697 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.697 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.697 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.697 DEBUG:boto:Params: {} 2024-03-04T17:39:10.697 DEBUG:boto:Token: None 2024-03-04T17:39:10.698 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.698 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mKRsZGfa8ldplUUjIfv1UPH12dc= 2024-03-04T17:39:10.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mKRsZGfa8ldplUUjIfv1UPH12dc=', 'Content-Length': '0'} 2024-03-04T17:39:10.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e6460dbc6964d12-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.701 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:39:10.701 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:39:10.701 DEBUG:boto:path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.701 DEBUG:boto:auth_path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.701 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.701 DEBUG:boto:Path: /ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.701 DEBUG:boto:Data: 2024-03-04T17:39:10.701 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.701 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.701 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.701 DEBUG:boto:Params: {} 2024-03-04T17:39:10.702 DEBUG:boto:Token: None 2024-03-04T17:39:10.702 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.702 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mKRsZGfa8ldplUUjIfv1UPH12dc= 2024-03-04T17:39:10.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mKRsZGfa8ldplUUjIfv1UPH12dc=', 'Content-Length': '0'} 2024-03-04T17:39:10.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072fa508cd3952df0-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:10.705 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:39:10.705 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:39:10.705 DEBUG:boto:path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.705 DEBUG:boto:auth_path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.705 DEBUG:boto:Method: GET 2024-03-04T17:39:10.705 DEBUG:boto:Path: /ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.705 DEBUG:boto:Data: 2024-03-04T17:39:10.705 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.705 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.706 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.706 DEBUG:boto:Params: {} 2024-03-04T17:39:10.706 DEBUG:boto:Token: None 2024-03-04T17:39:10.706 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.706 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xSJOfM9m2kHtyBTJAPUSG8Q07Pk= 2024-03-04T17:39:10.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xSJOfM9m2kHtyBTJAPUSG8Q07Pk=', 'Content-Length': '0'} 2024-03-04T17:39:10.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067faeda4762591f3-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.752 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:39:10.752 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:39:10.752 DEBUG:boto:path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.752 DEBUG:boto:auth_path=/ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.752 DEBUG:boto:Method: GET 2024-03-04T17:39:10.753 DEBUG:boto:Path: /ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.753 DEBUG:boto:Data: 2024-03-04T17:39:10.753 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.753 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.753 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.753 DEBUG:boto:Params: {} 2024-03-04T17:39:10.753 DEBUG:boto:Token: None 2024-03-04T17:39:10.753 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/myobj?versionId=null 2024-03-04T17:39:10.753 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xSJOfM9m2kHtyBTJAPUSG8Q07Pk= 2024-03-04T17:39:10.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xSJOfM9m2kHtyBTJAPUSG8Q07Pk=', 'Content-Length': '0'} 2024-03-04T17:39:10.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b04b64338a19760-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.756 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.756 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.756 DEBUG:boto:path=/ppfahn-10/?prefix=myobj 2024-03-04T17:39:10.756 DEBUG:boto:auth_path=/ppfahn-10/?prefix=myobj 2024-03-04T17:39:10.757 DEBUG:boto:Method: GET 2024-03-04T17:39:10.757 DEBUG:boto:Path: /ppfahn-10/?prefix=myobj 2024-03-04T17:39:10.757 DEBUG:boto:Data: 2024-03-04T17:39:10.757 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.757 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.757 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.757 DEBUG:boto:Params: {} 2024-03-04T17:39:10.757 DEBUG:boto:Token: None 2024-03-04T17:39:10.757 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.757 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd1410ffac835d42-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.761 DEBUG:boto:b'ppfahn-10myobj1000falsemyobj2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.762 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.762 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.762 DEBUG:boto:path=/ppfahn-10/?prefix=myobj 2024-03-04T17:39:10.762 DEBUG:boto:auth_path=/ppfahn-10/?prefix=myobj 2024-03-04T17:39:10.762 DEBUG:boto:Method: GET 2024-03-04T17:39:10.762 DEBUG:boto:Path: /ppfahn-10/?prefix=myobj 2024-03-04T17:39:10.762 DEBUG:boto:Data: 2024-03-04T17:39:10.762 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.762 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.762 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.762 DEBUG:boto:Params: {} 2024-03-04T17:39:10.762 DEBUG:boto:Token: None 2024-03-04T17:39:10.763 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.763 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c2d66ad40ac9055-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.808 DEBUG:boto:b'ppfahn-10myobj1000falsemyobj2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.808 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:39:10.809 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:39:10.809 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.809 DEBUG:boto:Path: /ppfahn-10/myobj 2024-03-04T17:39:10.809 DEBUG:boto:Data: 2024-03-04T17:39:10.809 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.809 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.809 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.809 DEBUG:boto:Params: {} 2024-03-04T17:39:10.809 DEBUG:boto:Token: None 2024-03-04T17:39:10.809 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/myobj 2024-03-04T17:39:10.810 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IS1Fs6wd/RDlVZEUt8rvNtAnru4= 2024-03-04T17:39:10.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IS1Fs6wd/RDlVZEUt8rvNtAnru4=', 'Content-Length': '0'} 2024-03-04T17:39:10.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091fc2a193c719928-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.813 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:39:10.813 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:39:10.813 DEBUG:boto:Method: HEAD 2024-03-04T17:39:10.813 DEBUG:boto:Path: /ppfahn-10/myobj 2024-03-04T17:39:10.813 DEBUG:boto:Data: 2024-03-04T17:39:10.813 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.813 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.813 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.813 DEBUG:boto:Params: {} 2024-03-04T17:39:10.813 DEBUG:boto:Token: None 2024-03-04T17:39:10.814 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/myobj 2024-03-04T17:39:10.814 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IS1Fs6wd/RDlVZEUt8rvNtAnru4= 2024-03-04T17:39:10.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IS1Fs6wd/RDlVZEUt8rvNtAnru4=', 'Content-Length': '0'} 2024-03-04T17:39:10.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b5b13d7837b6080-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:10.817 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:39:10.817 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:39:10.817 DEBUG:boto:Method: GET 2024-03-04T17:39:10.817 DEBUG:boto:Path: /ppfahn-10/myobj 2024-03-04T17:39:10.817 DEBUG:boto:Data: 2024-03-04T17:39:10.817 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.817 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.817 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.817 DEBUG:boto:Params: {} 2024-03-04T17:39:10.818 DEBUG:boto:Token: None 2024-03-04T17:39:10.818 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/myobj 2024-03-04T17:39:10.818 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YVaIfCbeXxLVuMzMKOIjT0F+tVI= 2024-03-04T17:39:10.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YVaIfCbeXxLVuMzMKOIjT0F+tVI=', 'Content-Length': '0'} 2024-03-04T17:39:10.820 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-10:d96636ac-054d-40e3-90f3-e3912da784b2.4349.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b2e2da77cb74d0a-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.864 DEBUG:boto:path=/ppfahn-10/myobj 2024-03-04T17:39:10.864 DEBUG:boto:auth_path=/ppfahn-10/myobj 2024-03-04T17:39:10.864 DEBUG:boto:Method: GET 2024-03-04T17:39:10.864 DEBUG:boto:Path: /ppfahn-10/myobj 2024-03-04T17:39:10.864 DEBUG:boto:Data: 2024-03-04T17:39:10.864 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.865 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.865 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.865 DEBUG:boto:Params: {} 2024-03-04T17:39:10.865 DEBUG:boto:Token: None 2024-03-04T17:39:10.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/myobj 2024-03-04T17:39:10.865 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YVaIfCbeXxLVuMzMKOIjT0F+tVI= 2024-03-04T17:39:10.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YVaIfCbeXxLVuMzMKOIjT0F+tVI=', 'Content-Length': '0'} 2024-03-04T17:39:10.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9a9f73984a69e57-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.869 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.869 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.869 DEBUG:boto:path=/ppfahn-10/?prefix=myobj 2024-03-04T17:39:10.869 DEBUG:boto:auth_path=/ppfahn-10/?prefix=myobj 2024-03-04T17:39:10.869 DEBUG:boto:Method: GET 2024-03-04T17:39:10.869 DEBUG:boto:Path: /ppfahn-10/?prefix=myobj 2024-03-04T17:39:10.869 DEBUG:boto:Data: 2024-03-04T17:39:10.869 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.869 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:10.869 DEBUG:boto:Port: 8001 2024-03-04T17:39:10.869 DEBUG:boto:Params: {} 2024-03-04T17:39:10.869 DEBUG:boto:Token: None 2024-03-04T17:39:10.870 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.870 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093cbae3df30c7b10-0065e6073e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.916 DEBUG:boto:b'ppfahn-10myobj1000falsemyobj2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.916 DEBUG:boto:path=/ppfahn-10/ 2024-03-04T17:39:10.917 DEBUG:boto:auth_path=/ppfahn-10/ 2024-03-04T17:39:10.917 DEBUG:boto:path=/ppfahn-10/?prefix=myobj 2024-03-04T17:39:10.917 DEBUG:boto:auth_path=/ppfahn-10/?prefix=myobj 2024-03-04T17:39:10.917 DEBUG:boto:Method: GET 2024-03-04T17:39:10.917 DEBUG:boto:Path: /ppfahn-10/?prefix=myobj 2024-03-04T17:39:10.917 DEBUG:boto:Data: 2024-03-04T17:39:10.917 DEBUG:boto:Headers: {} 2024-03-04T17:39:10.917 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:10.917 DEBUG:boto:Port: 8000 2024-03-04T17:39:10.917 DEBUG:boto:Params: {} 2024-03-04T17:39:10.917 DEBUG:boto:Token: None 2024-03-04T17:39:10.917 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:10 GMT /ppfahn-10/ 2024-03-04T17:39:10.918 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw= 2024-03-04T17:39:10.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+iWqJO72U8FxkxeAgrLH2HkbrIw=', 'Content-Length': '0'} 2024-03-04T17:39:10.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fce20b654530115e-0065e6073e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:10.922 DEBUG:boto:b'ppfahn-10myobj1000falsemyobj2024-03-04T17:36:30.010Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:10.922 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-10 zones={test-zone3, test-zone1} 2024-03-04T17:39:10.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:10.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:10.923 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:39:10.971 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:10.972 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:11.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:11.036+0000 7f73686e9980 1 bucket sync caught up with source: 2024-03-04T17:39:11.037 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000002.12.6, , 00000000001.11.6, , 00000000001.9.6, 00000000002.7.6, 00000000001.16.6, 00000000001.17.6, 00000000001.17.6, ] 2024-03-04T17:39:11.037 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.12.6, , 00000000001.11.6, , 00000000001.9.6, 00000000002.7.6, 00000000001.16.6, 00000000001.17.6, 00000000001.17.6, ] 2024-03-04T17:39:11.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:11.036+0000 7f73686e9980 0 bucket checkpoint complete 2024-03-04T17:39:11.059 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-11 zones={test-zone1, test-zone2} 2024-03-04T17:39:11.059 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.059 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.059 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.059 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:39:11.059 DEBUG:boto:Data: 2024-03-04T17:39:11.059 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.059 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.059 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.059 DEBUG:boto:Params: {} 2024-03-04T17:39:11.060 DEBUG:boto:Token: None 2024-03-04T17:39:11.060 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.060 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W1U8D8N+aSL0IsWUStS7wQt6b9k= 2024-03-04T17:39:11.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W1U8D8N+aSL0IsWUStS7wQt6b9k=', 'Content-Length': '0'} 2024-03-04T17:39:11.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecd3b0a6c89a08af-0065e6073f-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.063 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.063 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.063 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.063 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:39:11.063 DEBUG:boto:Data: 2024-03-04T17:39:11.063 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.063 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.063 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.063 DEBUG:boto:Params: {} 2024-03-04T17:39:11.064 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:39:11.064 DEBUG:boto:Token: None 2024-03-04T17:39:11.064 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.064 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W1U8D8N+aSL0IsWUStS7wQt6b9k= 2024-03-04T17:39:11.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W1U8D8N+aSL0IsWUStS7wQt6b9k=', 'Content-Length': '0'} 2024-03-04T17:39:11.070 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033d345d617cf6a50-0065e6073f-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.070 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:39:11.070 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.070 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.070 DEBUG:boto:path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:11.071 DEBUG:boto:auth_path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:11.071 DEBUG:boto:Method: GET 2024-03-04T17:39:11.071 DEBUG:boto:Path: /ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:11.071 DEBUG:boto:Data: 2024-03-04T17:39:11.071 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.071 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.071 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.071 DEBUG:boto:Params: {} 2024-03-04T17:39:11.071 DEBUG:boto:Token: None 2024-03-04T17:39:11.071 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/?versions 2024-03-04T17:39:11.071 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:b1EKRcGG9nZIuX0pBZEEmNyo6J4= 2024-03-04T17:39:11.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:b1EKRcGG9nZIuX0pBZEEmNyo6J4=', 'Content-Length': '0'} 2024-03-04T17:39:11.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5b3c697849abc74-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.074 DEBUG:boto:b'ppfahn-11999false&nulltrue2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.074 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T17:39:11.075 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T17:39:11.075 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T17:39:11.075 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T17:39:11.075 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T17:39:11.075 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T17:39:11.075 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T17:39:11.075 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T17:39:11.075 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:11.075 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:39:11.075 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.076 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.076 DEBUG:boto:path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:11.076 DEBUG:boto:auth_path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:11.076 DEBUG:boto:Method: GET 2024-03-04T17:39:11.076 DEBUG:boto:Path: /ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:11.076 DEBUG:boto:Data: 2024-03-04T17:39:11.076 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.076 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.076 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.076 DEBUG:boto:Params: {} 2024-03-04T17:39:11.076 DEBUG:boto:Token: None 2024-03-04T17:39:11.077 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/?versions 2024-03-04T17:39:11.077 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:b1EKRcGG9nZIuX0pBZEEmNyo6J4= 2024-03-04T17:39:11.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:b1EKRcGG9nZIuX0pBZEEmNyo6J4=', 'Content-Length': '0'} 2024-03-04T17:39:11.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088f06a2f1bd71697-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.082 DEBUG:boto:b'ppfahn-11999false&nulltrue2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.082 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T17:39:11.083 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T17:39:11.083 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T17:39:11.083 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T17:39:11.083 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T17:39:11.083 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T17:39:11.083 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T17:39:11.083 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T17:39:11.083 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:11.083 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.083 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.084 DEBUG:boto:path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:11.084 DEBUG:boto:auth_path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:11.084 DEBUG:boto:Method: GET 2024-03-04T17:39:11.084 DEBUG:boto:Path: /ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:11.084 DEBUG:boto:Data: 2024-03-04T17:39:11.084 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.084 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.084 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.084 DEBUG:boto:Params: {} 2024-03-04T17:39:11.084 DEBUG:boto:Token: None 2024-03-04T17:39:11.084 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/?versions 2024-03-04T17:39:11.085 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:b1EKRcGG9nZIuX0pBZEEmNyo6J4= 2024-03-04T17:39:11.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:b1EKRcGG9nZIuX0pBZEEmNyo6J4=', 'Content-Length': '0'} 2024-03-04T17:39:11.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db74bf2ead97c22e-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.088 DEBUG:boto:b'ppfahn-11999false&nulltrue2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.088 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.088 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.089 DEBUG:boto:path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:11.089 DEBUG:boto:auth_path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:11.089 DEBUG:boto:Method: GET 2024-03-04T17:39:11.089 DEBUG:boto:Path: /ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:11.089 DEBUG:boto:Data: 2024-03-04T17:39:11.089 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.089 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.089 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.089 DEBUG:boto:Params: {} 2024-03-04T17:39:11.089 DEBUG:boto:Token: None 2024-03-04T17:39:11.090 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/?versions 2024-03-04T17:39:11.090 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:b1EKRcGG9nZIuX0pBZEEmNyo6J4= 2024-03-04T17:39:11.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:b1EKRcGG9nZIuX0pBZEEmNyo6J4=', 'Content-Length': '0'} 2024-03-04T17:39:11.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e51752a4280b8c87-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.094 DEBUG:boto:b'ppfahn-11999false&nulltrue2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:11.095 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:11.095 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:11.095 DEBUG:boto:path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.095 DEBUG:boto:auth_path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.095 DEBUG:boto:Method: GET 2024-03-04T17:39:11.096 DEBUG:boto:Path: /ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.096 DEBUG:boto:Data: 2024-03-04T17:39:11.096 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.096 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.096 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.096 DEBUG:boto:Params: {} 2024-03-04T17:39:11.096 DEBUG:boto:Token: None 2024-03-04T17:39:11.096 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.096 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S8oYa72uaF9T6a+pYqx3cxs56R0= 2024-03-04T17:39:11.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S8oYa72uaF9T6a+pYqx3cxs56R0=', 'Content-Length': '0'} 2024-03-04T17:39:11.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003952b800e6127162-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.099 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:11.099 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:11.099 DEBUG:boto:path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.099 DEBUG:boto:auth_path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.099 DEBUG:boto:Method: GET 2024-03-04T17:39:11.099 DEBUG:boto:Path: /ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.100 DEBUG:boto:Data: 2024-03-04T17:39:11.100 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.100 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.100 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.100 DEBUG:boto:Params: {} 2024-03-04T17:39:11.100 DEBUG:boto:Token: None 2024-03-04T17:39:11.100 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.100 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S8oYa72uaF9T6a+pYqx3cxs56R0= 2024-03-04T17:39:11.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S8oYa72uaF9T6a+pYqx3cxs56R0=', 'Content-Length': '0'} 2024-03-04T17:39:11.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f4912470f7a5152-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.144 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:11.144 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:11.144 DEBUG:boto:path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.144 DEBUG:boto:auth_path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.144 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.145 DEBUG:boto:Path: /ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.145 DEBUG:boto:Data: 2024-03-04T17:39:11.145 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.145 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.145 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.145 DEBUG:boto:Params: {} 2024-03-04T17:39:11.145 DEBUG:boto:Token: None 2024-03-04T17:39:11.145 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.145 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hyma4Urmk0ihgMBkLYlbGaXzS0E= 2024-03-04T17:39:11.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hyma4Urmk0ihgMBkLYlbGaXzS0E=', 'Content-Length': '0'} 2024-03-04T17:39:11.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094bd17e7c1e7052c-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.149 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:11.149 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:11.149 DEBUG:boto:path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.149 DEBUG:boto:auth_path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.149 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.149 DEBUG:boto:Path: /ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.149 DEBUG:boto:Data: 2024-03-04T17:39:11.149 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.149 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.149 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.149 DEBUG:boto:Params: {} 2024-03-04T17:39:11.150 DEBUG:boto:Token: None 2024-03-04T17:39:11.150 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.150 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hyma4Urmk0ihgMBkLYlbGaXzS0E= 2024-03-04T17:39:11.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hyma4Urmk0ihgMBkLYlbGaXzS0E=', 'Content-Length': '0'} 2024-03-04T17:39:11.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a1e87a44c3dd67f-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:11.153 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:11.153 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:11.153 DEBUG:boto:path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.153 DEBUG:boto:auth_path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.153 DEBUG:boto:Method: GET 2024-03-04T17:39:11.153 DEBUG:boto:Path: /ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.153 DEBUG:boto:Data: 2024-03-04T17:39:11.154 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.154 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.154 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.154 DEBUG:boto:Params: {} 2024-03-04T17:39:11.154 DEBUG:boto:Token: None 2024-03-04T17:39:11.154 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.154 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S8oYa72uaF9T6a+pYqx3cxs56R0= 2024-03-04T17:39:11.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S8oYa72uaF9T6a+pYqx3cxs56R0=', 'Content-Length': '0'} 2024-03-04T17:39:11.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e85852f7dc006280-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.200 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:11.200 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:11.200 DEBUG:boto:path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.200 DEBUG:boto:auth_path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.200 DEBUG:boto:Method: GET 2024-03-04T17:39:11.201 DEBUG:boto:Path: /ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.201 DEBUG:boto:Data: 2024-03-04T17:39:11.201 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.201 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.201 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.201 DEBUG:boto:Params: {} 2024-03-04T17:39:11.201 DEBUG:boto:Token: None 2024-03-04T17:39:11.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/%26?versionId=null 2024-03-04T17:39:11.201 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S8oYa72uaF9T6a+pYqx3cxs56R0= 2024-03-04T17:39:11.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S8oYa72uaF9T6a+pYqx3cxs56R0=', 'Content-Length': '0'} 2024-03-04T17:39:11.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013068bbd5966b271-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.204 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.205 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.205 DEBUG:boto:path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:11.205 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:11.205 DEBUG:boto:Method: GET 2024-03-04T17:39:11.205 DEBUG:boto:Path: /ppfahn-11/?prefix=%26 2024-03-04T17:39:11.205 DEBUG:boto:Data: 2024-03-04T17:39:11.205 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.205 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.205 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.205 DEBUG:boto:Params: {} 2024-03-04T17:39:11.206 DEBUG:boto:Token: None 2024-03-04T17:39:11.206 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.206 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065c1114e35ab9b29-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.256 DEBUG:boto:b'ppfahn-11&1000false&2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.256 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.256 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.257 DEBUG:boto:path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:11.257 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:11.257 DEBUG:boto:Method: GET 2024-03-04T17:39:11.257 DEBUG:boto:Path: /ppfahn-11/?prefix=%26 2024-03-04T17:39:11.257 DEBUG:boto:Data: 2024-03-04T17:39:11.257 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.257 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.257 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.257 DEBUG:boto:Params: {} 2024-03-04T17:39:11.257 DEBUG:boto:Token: None 2024-03-04T17:39:11.258 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.258 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002c12c8da5a38de0-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.262 DEBUG:boto:b'ppfahn-11&1000false&2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.262 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:11.263 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:11.263 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.263 DEBUG:boto:Path: /ppfahn-11/%26 2024-03-04T17:39:11.263 DEBUG:boto:Data: 2024-03-04T17:39:11.263 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.263 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.263 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.263 DEBUG:boto:Params: {} 2024-03-04T17:39:11.264 DEBUG:boto:Token: None 2024-03-04T17:39:11.264 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/%26 2024-03-04T17:39:11.264 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BLJ9udxMtUgF4iHNADDI7pOFFA8= 2024-03-04T17:39:11.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BLJ9udxMtUgF4iHNADDI7pOFFA8=', 'Content-Length': '0'} 2024-03-04T17:39:11.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bcb6f4b22c22c7c-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.267 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:11.267 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:11.267 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.267 DEBUG:boto:Path: /ppfahn-11/%26 2024-03-04T17:39:11.267 DEBUG:boto:Data: 2024-03-04T17:39:11.267 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.267 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.268 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.268 DEBUG:boto:Params: {} 2024-03-04T17:39:11.268 DEBUG:boto:Token: None 2024-03-04T17:39:11.268 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/%26 2024-03-04T17:39:11.268 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BLJ9udxMtUgF4iHNADDI7pOFFA8= 2024-03-04T17:39:11.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BLJ9udxMtUgF4iHNADDI7pOFFA8=', 'Content-Length': '0'} 2024-03-04T17:39:11.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b27bcc717169f193-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:11.271 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:11.271 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:11.271 DEBUG:boto:Method: GET 2024-03-04T17:39:11.271 DEBUG:boto:Path: /ppfahn-11/%26 2024-03-04T17:39:11.272 DEBUG:boto:Data: 2024-03-04T17:39:11.272 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.272 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.272 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.272 DEBUG:boto:Params: {} 2024-03-04T17:39:11.272 DEBUG:boto:Token: None 2024-03-04T17:39:11.272 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/%26 2024-03-04T17:39:11.272 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3iVz5nOpzw2BOZQhT2mjhcJ7UlE= 2024-03-04T17:39:11.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3iVz5nOpzw2BOZQhT2mjhcJ7UlE=', 'Content-Length': '0'} 2024-03-04T17:39:11.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b57d5f579e22fd0-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.276 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:11.276 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:11.276 DEBUG:boto:Method: GET 2024-03-04T17:39:11.277 DEBUG:boto:Path: /ppfahn-11/%26 2024-03-04T17:39:11.277 DEBUG:boto:Data: 2024-03-04T17:39:11.277 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.277 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.277 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.277 DEBUG:boto:Params: {} 2024-03-04T17:39:11.277 DEBUG:boto:Token: None 2024-03-04T17:39:11.277 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/%26 2024-03-04T17:39:11.277 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3iVz5nOpzw2BOZQhT2mjhcJ7UlE= 2024-03-04T17:39:11.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3iVz5nOpzw2BOZQhT2mjhcJ7UlE=', 'Content-Length': '0'} 2024-03-04T17:39:11.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b25399dfd3eadedb-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.280 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.280 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.281 DEBUG:boto:path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:11.281 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:11.281 DEBUG:boto:Method: GET 2024-03-04T17:39:11.281 DEBUG:boto:Path: /ppfahn-11/?prefix=%26 2024-03-04T17:39:11.281 DEBUG:boto:Data: 2024-03-04T17:39:11.281 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.281 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.281 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.281 DEBUG:boto:Params: {} 2024-03-04T17:39:11.281 DEBUG:boto:Token: None 2024-03-04T17:39:11.281 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.282 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000078195867af955ba-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.287 DEBUG:boto:b'ppfahn-11&1000false&2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.287 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.287 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.287 DEBUG:boto:path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:11.287 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:11.287 DEBUG:boto:Method: GET 2024-03-04T17:39:11.287 DEBUG:boto:Path: /ppfahn-11/?prefix=%26 2024-03-04T17:39:11.287 DEBUG:boto:Data: 2024-03-04T17:39:11.288 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.288 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.288 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.288 DEBUG:boto:Params: {} 2024-03-04T17:39:11.288 DEBUG:boto:Token: None 2024-03-04T17:39:11.288 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.288 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbcf193fc3cbda55-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.336 DEBUG:boto:b'ppfahn-11&1000false&2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:11.336 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:11.336 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:11.337 DEBUG:boto:path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:11.337 DEBUG:boto:auth_path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:11.337 DEBUG:boto:Method: GET 2024-03-04T17:39:11.337 DEBUG:boto:Path: /ppfahn-11/.?versionId=null 2024-03-04T17:39:11.337 DEBUG:boto:Data: 2024-03-04T17:39:11.337 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.337 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.337 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.337 DEBUG:boto:Params: {} 2024-03-04T17:39:11.337 DEBUG:boto:Token: None 2024-03-04T17:39:11.338 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.?versionId=null 2024-03-04T17:39:11.338 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MVXzluPMOhWsJrsR8LQQsoFn53w= 2024-03-04T17:39:11.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MVXzluPMOhWsJrsR8LQQsoFn53w=', 'Content-Length': '0'} 2024-03-04T17:39:11.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1dd050b637bcffd-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.341 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:11.341 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:11.341 DEBUG:boto:path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:11.341 DEBUG:boto:auth_path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:11.341 DEBUG:boto:Method: GET 2024-03-04T17:39:11.341 DEBUG:boto:Path: /ppfahn-11/.?versionId=null 2024-03-04T17:39:11.341 DEBUG:boto:Data: 2024-03-04T17:39:11.341 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.342 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.342 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.342 DEBUG:boto:Params: {} 2024-03-04T17:39:11.342 DEBUG:boto:Token: None 2024-03-04T17:39:11.342 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.?versionId=null 2024-03-04T17:39:11.342 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MVXzluPMOhWsJrsR8LQQsoFn53w= 2024-03-04T17:39:11.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MVXzluPMOhWsJrsR8LQQsoFn53w=', 'Content-Length': '0'} 2024-03-04T17:39:11.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000275ad20159b7e2f3-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.345 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:11.346 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:11.346 DEBUG:boto:path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:11.346 DEBUG:boto:auth_path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:11.346 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.346 DEBUG:boto:Path: /ppfahn-11/.?versionId=null 2024-03-04T17:39:11.346 DEBUG:boto:Data: 2024-03-04T17:39:11.346 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.346 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.346 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.346 DEBUG:boto:Params: {} 2024-03-04T17:39:11.346 DEBUG:boto:Token: None 2024-03-04T17:39:11.347 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.?versionId=null 2024-03-04T17:39:11.347 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DDOckbV6H+qJ3RhZyzNYm0du6yg= 2024-03-04T17:39:11.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DDOckbV6H+qJ3RhZyzNYm0du6yg=', 'Content-Length': '0'} 2024-03-04T17:39:11.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2af7dab340c3d06-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.350 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:11.350 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:11.350 DEBUG:boto:path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:11.350 DEBUG:boto:auth_path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:11.350 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.350 DEBUG:boto:Path: /ppfahn-11/.?versionId=null 2024-03-04T17:39:11.350 DEBUG:boto:Data: 2024-03-04T17:39:11.350 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.351 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.351 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.351 DEBUG:boto:Params: {} 2024-03-04T17:39:11.351 DEBUG:boto:Token: None 2024-03-04T17:39:11.351 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.?versionId=null 2024-03-04T17:39:11.351 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DDOckbV6H+qJ3RhZyzNYm0du6yg= 2024-03-04T17:39:11.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DDOckbV6H+qJ3RhZyzNYm0du6yg=', 'Content-Length': '0'} 2024-03-04T17:39:11.354 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae58e6b9e3c69fca-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:11.354 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:11.354 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:11.354 DEBUG:boto:path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:11.354 DEBUG:boto:auth_path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:11.355 DEBUG:boto:Method: GET 2024-03-04T17:39:11.355 DEBUG:boto:Path: /ppfahn-11/.?versionId=null 2024-03-04T17:39:11.355 DEBUG:boto:Data: 2024-03-04T17:39:11.355 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.355 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.355 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.355 DEBUG:boto:Params: {} 2024-03-04T17:39:11.355 DEBUG:boto:Token: None 2024-03-04T17:39:11.355 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.?versionId=null 2024-03-04T17:39:11.355 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MVXzluPMOhWsJrsR8LQQsoFn53w= 2024-03-04T17:39:11.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MVXzluPMOhWsJrsR8LQQsoFn53w=', 'Content-Length': '0'} 2024-03-04T17:39:11.359 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f13f16e304fb6f8-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.400 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:11.400 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:11.400 DEBUG:boto:path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:11.400 DEBUG:boto:auth_path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:11.401 DEBUG:boto:Method: GET 2024-03-04T17:39:11.401 DEBUG:boto:Path: /ppfahn-11/.?versionId=null 2024-03-04T17:39:11.401 DEBUG:boto:Data: 2024-03-04T17:39:11.401 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.401 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.401 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.401 DEBUG:boto:Params: {} 2024-03-04T17:39:11.401 DEBUG:boto:Token: None 2024-03-04T17:39:11.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.?versionId=null 2024-03-04T17:39:11.401 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MVXzluPMOhWsJrsR8LQQsoFn53w= 2024-03-04T17:39:11.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MVXzluPMOhWsJrsR8LQQsoFn53w=', 'Content-Length': '0'} 2024-03-04T17:39:11.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000faf9b25c5276b7cf-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.404 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.404 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.405 DEBUG:boto:path=/ppfahn-11/?prefix=. 2024-03-04T17:39:11.405 DEBUG:boto:auth_path=/ppfahn-11/?prefix=. 2024-03-04T17:39:11.405 DEBUG:boto:Method: GET 2024-03-04T17:39:11.405 DEBUG:boto:Path: /ppfahn-11/?prefix=. 2024-03-04T17:39:11.405 DEBUG:boto:Data: 2024-03-04T17:39:11.405 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.405 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.405 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.405 DEBUG:boto:Params: {} 2024-03-04T17:39:11.405 DEBUG:boto:Token: None 2024-03-04T17:39:11.405 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.406 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d7d95a969e4b80c-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.411 DEBUG:boto:b'ppfahn-11.1000false.2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.411 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.412 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.412 DEBUG:boto:path=/ppfahn-11/?prefix=. 2024-03-04T17:39:11.412 DEBUG:boto:auth_path=/ppfahn-11/?prefix=. 2024-03-04T17:39:11.412 DEBUG:boto:Method: GET 2024-03-04T17:39:11.412 DEBUG:boto:Path: /ppfahn-11/?prefix=. 2024-03-04T17:39:11.412 DEBUG:boto:Data: 2024-03-04T17:39:11.412 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.412 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.412 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.412 DEBUG:boto:Params: {} 2024-03-04T17:39:11.413 DEBUG:boto:Token: None 2024-03-04T17:39:11.413 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.413 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4b51ade691c665d-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.418 DEBUG:boto:b'ppfahn-11.1000false.2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.419 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:11.419 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:11.419 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.419 DEBUG:boto:Path: /ppfahn-11/. 2024-03-04T17:39:11.419 DEBUG:boto:Data: 2024-03-04T17:39:11.419 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.419 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.419 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.419 DEBUG:boto:Params: {} 2024-03-04T17:39:11.420 DEBUG:boto:Token: None 2024-03-04T17:39:11.420 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/. 2024-03-04T17:39:11.420 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:imh94xlAZZibgfild6bLKGXx+R4= 2024-03-04T17:39:11.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:imh94xlAZZibgfild6bLKGXx+R4=', 'Content-Length': '0'} 2024-03-04T17:39:11.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aafa2bc1f0b7a8a5-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.423 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:11.423 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:11.423 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.423 DEBUG:boto:Path: /ppfahn-11/. 2024-03-04T17:39:11.423 DEBUG:boto:Data: 2024-03-04T17:39:11.423 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.424 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.424 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.424 DEBUG:boto:Params: {} 2024-03-04T17:39:11.424 DEBUG:boto:Token: None 2024-03-04T17:39:11.424 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/. 2024-03-04T17:39:11.424 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:imh94xlAZZibgfild6bLKGXx+R4= 2024-03-04T17:39:11.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:imh94xlAZZibgfild6bLKGXx+R4=', 'Content-Length': '0'} 2024-03-04T17:39:11.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036b2c698cb4e6ea1-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:11.427 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:11.427 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:11.427 DEBUG:boto:Method: GET 2024-03-04T17:39:11.427 DEBUG:boto:Path: /ppfahn-11/. 2024-03-04T17:39:11.428 DEBUG:boto:Data: 2024-03-04T17:39:11.428 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.428 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.428 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.428 DEBUG:boto:Params: {} 2024-03-04T17:39:11.428 DEBUG:boto:Token: None 2024-03-04T17:39:11.428 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/. 2024-03-04T17:39:11.428 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:j99QJcKXunZ/nzZLK7a9+UFI994= 2024-03-04T17:39:11.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:j99QJcKXunZ/nzZLK7a9+UFI994=', 'Content-Length': '0'} 2024-03-04T17:39:11.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3362915e4b19be8-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.432 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:11.432 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:11.432 DEBUG:boto:Method: GET 2024-03-04T17:39:11.432 DEBUG:boto:Path: /ppfahn-11/. 2024-03-04T17:39:11.432 DEBUG:boto:Data: 2024-03-04T17:39:11.432 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.432 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.432 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.432 DEBUG:boto:Params: {} 2024-03-04T17:39:11.433 DEBUG:boto:Token: None 2024-03-04T17:39:11.433 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/. 2024-03-04T17:39:11.433 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:j99QJcKXunZ/nzZLK7a9+UFI994= 2024-03-04T17:39:11.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:j99QJcKXunZ/nzZLK7a9+UFI994=', 'Content-Length': '0'} 2024-03-04T17:39:11.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083a55110a64732d8-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.476 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.477 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.477 DEBUG:boto:path=/ppfahn-11/?prefix=. 2024-03-04T17:39:11.477 DEBUG:boto:auth_path=/ppfahn-11/?prefix=. 2024-03-04T17:39:11.477 DEBUG:boto:Method: GET 2024-03-04T17:39:11.477 DEBUG:boto:Path: /ppfahn-11/?prefix=. 2024-03-04T17:39:11.477 DEBUG:boto:Data: 2024-03-04T17:39:11.477 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.477 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.477 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.477 DEBUG:boto:Params: {} 2024-03-04T17:39:11.477 DEBUG:boto:Token: None 2024-03-04T17:39:11.478 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.478 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ba6d15b9b5b5914-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.483 DEBUG:boto:b'ppfahn-11.1000false.2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.483 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.483 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.483 DEBUG:boto:path=/ppfahn-11/?prefix=. 2024-03-04T17:39:11.484 DEBUG:boto:auth_path=/ppfahn-11/?prefix=. 2024-03-04T17:39:11.484 DEBUG:boto:Method: GET 2024-03-04T17:39:11.484 DEBUG:boto:Path: /ppfahn-11/?prefix=. 2024-03-04T17:39:11.484 DEBUG:boto:Data: 2024-03-04T17:39:11.484 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.484 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.484 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.484 DEBUG:boto:Params: {} 2024-03-04T17:39:11.484 DEBUG:boto:Token: None 2024-03-04T17:39:11.484 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.485 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bcd3cb6ef657fa7-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.489 DEBUG:boto:b'ppfahn-11.1000false.2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:11.490 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:11.490 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:11.490 DEBUG:boto:path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:11.490 DEBUG:boto:auth_path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:11.490 DEBUG:boto:Method: GET 2024-03-04T17:39:11.491 DEBUG:boto:Path: /ppfahn-11/..?versionId=null 2024-03-04T17:39:11.491 DEBUG:boto:Data: 2024-03-04T17:39:11.491 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.491 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.491 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.491 DEBUG:boto:Params: {} 2024-03-04T17:39:11.491 DEBUG:boto:Token: None 2024-03-04T17:39:11.491 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/..?versionId=null 2024-03-04T17:39:11.491 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:L2zRa8uxnXrhWksGozdkpzTA508= 2024-03-04T17:39:11.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:L2zRa8uxnXrhWksGozdkpzTA508=', 'Content-Length': '0'} 2024-03-04T17:39:11.494 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b683862c0fd6091-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.536 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:11.536 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:11.536 DEBUG:boto:path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:11.536 DEBUG:boto:auth_path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:11.536 DEBUG:boto:Method: GET 2024-03-04T17:39:11.536 DEBUG:boto:Path: /ppfahn-11/..?versionId=null 2024-03-04T17:39:11.537 DEBUG:boto:Data: 2024-03-04T17:39:11.537 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.537 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.537 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.537 DEBUG:boto:Params: {} 2024-03-04T17:39:11.537 DEBUG:boto:Token: None 2024-03-04T17:39:11.537 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/..?versionId=null 2024-03-04T17:39:11.537 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:L2zRa8uxnXrhWksGozdkpzTA508= 2024-03-04T17:39:11.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:L2zRa8uxnXrhWksGozdkpzTA508=', 'Content-Length': '0'} 2024-03-04T17:39:11.540 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032f7c4d8055a2603-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.540 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:11.540 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:11.541 DEBUG:boto:path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:11.541 DEBUG:boto:auth_path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:11.541 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.541 DEBUG:boto:Path: /ppfahn-11/..?versionId=null 2024-03-04T17:39:11.541 DEBUG:boto:Data: 2024-03-04T17:39:11.541 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.541 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.541 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.541 DEBUG:boto:Params: {} 2024-03-04T17:39:11.541 DEBUG:boto:Token: None 2024-03-04T17:39:11.542 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/..?versionId=null 2024-03-04T17:39:11.542 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:blYunooSBUm1xyFC2bwZrGSAr2U= 2024-03-04T17:39:11.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:blYunooSBUm1xyFC2bwZrGSAr2U=', 'Content-Length': '0'} 2024-03-04T17:39:11.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a900091fdeb0bdca-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.545 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:11.545 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:11.545 DEBUG:boto:path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:11.545 DEBUG:boto:auth_path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:11.545 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.545 DEBUG:boto:Path: /ppfahn-11/..?versionId=null 2024-03-04T17:39:11.545 DEBUG:boto:Data: 2024-03-04T17:39:11.545 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.545 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.545 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.546 DEBUG:boto:Params: {} 2024-03-04T17:39:11.546 DEBUG:boto:Token: None 2024-03-04T17:39:11.546 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/..?versionId=null 2024-03-04T17:39:11.546 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:blYunooSBUm1xyFC2bwZrGSAr2U= 2024-03-04T17:39:11.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:blYunooSBUm1xyFC2bwZrGSAr2U=', 'Content-Length': '0'} 2024-03-04T17:39:11.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000244718d0de408646-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:11.550 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:11.550 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:11.550 DEBUG:boto:path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:11.550 DEBUG:boto:auth_path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:11.550 DEBUG:boto:Method: GET 2024-03-04T17:39:11.550 DEBUG:boto:Path: /ppfahn-11/..?versionId=null 2024-03-04T17:39:11.550 DEBUG:boto:Data: 2024-03-04T17:39:11.550 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.550 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.550 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.551 DEBUG:boto:Params: {} 2024-03-04T17:39:11.551 DEBUG:boto:Token: None 2024-03-04T17:39:11.551 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/..?versionId=null 2024-03-04T17:39:11.551 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:L2zRa8uxnXrhWksGozdkpzTA508= 2024-03-04T17:39:11.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:L2zRa8uxnXrhWksGozdkpzTA508=', 'Content-Length': '0'} 2024-03-04T17:39:11.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cfb451a14fef8f3-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.554 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:11.554 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:11.554 DEBUG:boto:path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:11.555 DEBUG:boto:auth_path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:11.555 DEBUG:boto:Method: GET 2024-03-04T17:39:11.555 DEBUG:boto:Path: /ppfahn-11/..?versionId=null 2024-03-04T17:39:11.555 DEBUG:boto:Data: 2024-03-04T17:39:11.555 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.555 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.555 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.555 DEBUG:boto:Params: {} 2024-03-04T17:39:11.555 DEBUG:boto:Token: None 2024-03-04T17:39:11.555 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/..?versionId=null 2024-03-04T17:39:11.556 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:L2zRa8uxnXrhWksGozdkpzTA508= 2024-03-04T17:39:11.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:L2zRa8uxnXrhWksGozdkpzTA508=', 'Content-Length': '0'} 2024-03-04T17:39:11.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074dd705618803af9-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.600 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.600 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.600 DEBUG:boto:path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:11.600 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:11.600 DEBUG:boto:Method: GET 2024-03-04T17:39:11.601 DEBUG:boto:Path: /ppfahn-11/?prefix=.. 2024-03-04T17:39:11.601 DEBUG:boto:Data: 2024-03-04T17:39:11.601 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.601 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.601 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.601 DEBUG:boto:Params: {} 2024-03-04T17:39:11.601 DEBUG:boto:Token: None 2024-03-04T17:39:11.601 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.601 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c017edc3c598d8e-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.606 DEBUG:boto:b'ppfahn-11..1000false..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.606 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.606 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.606 DEBUG:boto:path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:11.606 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:11.607 DEBUG:boto:Method: GET 2024-03-04T17:39:11.607 DEBUG:boto:Path: /ppfahn-11/?prefix=.. 2024-03-04T17:39:11.607 DEBUG:boto:Data: 2024-03-04T17:39:11.607 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.607 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.607 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.607 DEBUG:boto:Params: {} 2024-03-04T17:39:11.607 DEBUG:boto:Token: None 2024-03-04T17:39:11.607 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.607 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000160e478faebfb7e6-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.615 DEBUG:boto:b'ppfahn-11..1000false..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.615 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:11.616 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:11.616 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.616 DEBUG:boto:Path: /ppfahn-11/.. 2024-03-04T17:39:11.616 DEBUG:boto:Data: 2024-03-04T17:39:11.616 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.616 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.616 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.616 DEBUG:boto:Params: {} 2024-03-04T17:39:11.616 DEBUG:boto:Token: None 2024-03-04T17:39:11.617 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.. 2024-03-04T17:39:11.617 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xo+3zLU4WQDCzgJiXu0XGuDJK80= 2024-03-04T17:39:11.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xo+3zLU4WQDCzgJiXu0XGuDJK80=', 'Content-Length': '0'} 2024-03-04T17:39:11.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d74c9c48702a2a2-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.620 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:11.620 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:11.620 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.620 DEBUG:boto:Path: /ppfahn-11/.. 2024-03-04T17:39:11.620 DEBUG:boto:Data: 2024-03-04T17:39:11.620 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.620 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.620 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.620 DEBUG:boto:Params: {} 2024-03-04T17:39:11.621 DEBUG:boto:Token: None 2024-03-04T17:39:11.621 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.. 2024-03-04T17:39:11.621 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xo+3zLU4WQDCzgJiXu0XGuDJK80= 2024-03-04T17:39:11.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xo+3zLU4WQDCzgJiXu0XGuDJK80=', 'Content-Length': '0'} 2024-03-04T17:39:11.624 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3929d3d6edab1a9-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:11.624 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:11.624 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:11.624 DEBUG:boto:Method: GET 2024-03-04T17:39:11.624 DEBUG:boto:Path: /ppfahn-11/.. 2024-03-04T17:39:11.624 DEBUG:boto:Data: 2024-03-04T17:39:11.624 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.624 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.625 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.625 DEBUG:boto:Params: {} 2024-03-04T17:39:11.625 DEBUG:boto:Token: None 2024-03-04T17:39:11.625 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.. 2024-03-04T17:39:11.625 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:euEtA8OeLIEga2p8BM9Jb+1APh8= 2024-03-04T17:39:11.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:euEtA8OeLIEga2p8BM9Jb+1APh8=', 'Content-Length': '0'} 2024-03-04T17:39:11.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019ea712675abc4dc-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.672 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:11.672 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:11.672 DEBUG:boto:Method: GET 2024-03-04T17:39:11.672 DEBUG:boto:Path: /ppfahn-11/.. 2024-03-04T17:39:11.672 DEBUG:boto:Data: 2024-03-04T17:39:11.673 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.673 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.673 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.673 DEBUG:boto:Params: {} 2024-03-04T17:39:11.673 DEBUG:boto:Token: None 2024-03-04T17:39:11.673 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.. 2024-03-04T17:39:11.673 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:euEtA8OeLIEga2p8BM9Jb+1APh8= 2024-03-04T17:39:11.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:euEtA8OeLIEga2p8BM9Jb+1APh8=', 'Content-Length': '0'} 2024-03-04T17:39:11.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078b149ad1bf4970c-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.677 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.677 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.677 DEBUG:boto:path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:11.677 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:11.677 DEBUG:boto:Method: GET 2024-03-04T17:39:11.677 DEBUG:boto:Path: /ppfahn-11/?prefix=.. 2024-03-04T17:39:11.677 DEBUG:boto:Data: 2024-03-04T17:39:11.677 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.677 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.677 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.678 DEBUG:boto:Params: {} 2024-03-04T17:39:11.678 DEBUG:boto:Token: None 2024-03-04T17:39:11.678 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.678 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0dcbc66943ab253-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.682 DEBUG:boto:b'ppfahn-11..1000false..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.682 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.683 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.683 DEBUG:boto:path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:11.683 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:11.683 DEBUG:boto:Method: GET 2024-03-04T17:39:11.683 DEBUG:boto:Path: /ppfahn-11/?prefix=.. 2024-03-04T17:39:11.683 DEBUG:boto:Data: 2024-03-04T17:39:11.683 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.683 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.683 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.683 DEBUG:boto:Params: {} 2024-03-04T17:39:11.684 DEBUG:boto:Token: None 2024-03-04T17:39:11.684 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.684 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022de0fa93634a8e9-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.732 DEBUG:boto:b'ppfahn-11..1000false..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:11.732 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:11.733 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:11.733 DEBUG:boto:path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:11.733 DEBUG:boto:auth_path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:11.733 DEBUG:boto:Method: GET 2024-03-04T17:39:11.733 DEBUG:boto:Path: /ppfahn-11/...?versionId=null 2024-03-04T17:39:11.733 DEBUG:boto:Data: 2024-03-04T17:39:11.733 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.733 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.733 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.733 DEBUG:boto:Params: {} 2024-03-04T17:39:11.733 DEBUG:boto:Token: None 2024-03-04T17:39:11.734 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/...?versionId=null 2024-03-04T17:39:11.734 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Cy8ROW1PpPSoZkGnos3CU/+ZKBQ= 2024-03-04T17:39:11.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Cy8ROW1PpPSoZkGnos3CU/+ZKBQ=', 'Content-Length': '0'} 2024-03-04T17:39:11.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003589cd6a7df39a7f-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.737 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:11.737 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:11.737 DEBUG:boto:path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:11.737 DEBUG:boto:auth_path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:11.737 DEBUG:boto:Method: GET 2024-03-04T17:39:11.738 DEBUG:boto:Path: /ppfahn-11/...?versionId=null 2024-03-04T17:39:11.738 DEBUG:boto:Data: 2024-03-04T17:39:11.738 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.738 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.738 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.738 DEBUG:boto:Params: {} 2024-03-04T17:39:11.738 DEBUG:boto:Token: None 2024-03-04T17:39:11.738 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/...?versionId=null 2024-03-04T17:39:11.738 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Cy8ROW1PpPSoZkGnos3CU/+ZKBQ= 2024-03-04T17:39:11.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Cy8ROW1PpPSoZkGnos3CU/+ZKBQ=', 'Content-Length': '0'} 2024-03-04T17:39:11.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c11ddc831a7c1d71-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.742 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:11.742 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:11.742 DEBUG:boto:path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:11.742 DEBUG:boto:auth_path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:11.742 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.742 DEBUG:boto:Path: /ppfahn-11/...?versionId=null 2024-03-04T17:39:11.742 DEBUG:boto:Data: 2024-03-04T17:39:11.742 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.743 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.743 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.743 DEBUG:boto:Params: {} 2024-03-04T17:39:11.743 DEBUG:boto:Token: None 2024-03-04T17:39:11.743 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/...?versionId=null 2024-03-04T17:39:11.743 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DINTvfsxR/7ZhWV12j/DSoz+MEQ= 2024-03-04T17:39:11.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DINTvfsxR/7ZhWV12j/DSoz+MEQ=', 'Content-Length': '0'} 2024-03-04T17:39:11.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c7fd9bfa96396fa-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.746 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:11.746 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:11.746 DEBUG:boto:path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:11.746 DEBUG:boto:auth_path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:11.747 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.747 DEBUG:boto:Path: /ppfahn-11/...?versionId=null 2024-03-04T17:39:11.747 DEBUG:boto:Data: 2024-03-04T17:39:11.747 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.747 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.747 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.747 DEBUG:boto:Params: {} 2024-03-04T17:39:11.747 DEBUG:boto:Token: None 2024-03-04T17:39:11.747 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/...?versionId=null 2024-03-04T17:39:11.747 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DINTvfsxR/7ZhWV12j/DSoz+MEQ= 2024-03-04T17:39:11.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DINTvfsxR/7ZhWV12j/DSoz+MEQ=', 'Content-Length': '0'} 2024-03-04T17:39:11.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069c98ba6b6ba4661-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:11.751 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:11.751 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:11.751 DEBUG:boto:path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:11.751 DEBUG:boto:auth_path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:11.751 DEBUG:boto:Method: GET 2024-03-04T17:39:11.751 DEBUG:boto:Path: /ppfahn-11/...?versionId=null 2024-03-04T17:39:11.751 DEBUG:boto:Data: 2024-03-04T17:39:11.751 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.751 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.751 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.751 DEBUG:boto:Params: {} 2024-03-04T17:39:11.752 DEBUG:boto:Token: None 2024-03-04T17:39:11.752 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/...?versionId=null 2024-03-04T17:39:11.752 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Cy8ROW1PpPSoZkGnos3CU/+ZKBQ= 2024-03-04T17:39:11.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Cy8ROW1PpPSoZkGnos3CU/+ZKBQ=', 'Content-Length': '0'} 2024-03-04T17:39:11.755 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a14255d98f1dcff8-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.796 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:11.796 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:11.796 DEBUG:boto:path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:11.796 DEBUG:boto:auth_path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:11.796 DEBUG:boto:Method: GET 2024-03-04T17:39:11.796 DEBUG:boto:Path: /ppfahn-11/...?versionId=null 2024-03-04T17:39:11.797 DEBUG:boto:Data: 2024-03-04T17:39:11.797 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.797 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.797 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.797 DEBUG:boto:Params: {} 2024-03-04T17:39:11.797 DEBUG:boto:Token: None 2024-03-04T17:39:11.797 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/...?versionId=null 2024-03-04T17:39:11.797 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Cy8ROW1PpPSoZkGnos3CU/+ZKBQ= 2024-03-04T17:39:11.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Cy8ROW1PpPSoZkGnos3CU/+ZKBQ=', 'Content-Length': '0'} 2024-03-04T17:39:11.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6cf83956855489a-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.801 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.801 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.801 DEBUG:boto:path=/ppfahn-11/?prefix=... 2024-03-04T17:39:11.801 DEBUG:boto:auth_path=/ppfahn-11/?prefix=... 2024-03-04T17:39:11.801 DEBUG:boto:Method: GET 2024-03-04T17:39:11.801 DEBUG:boto:Path: /ppfahn-11/?prefix=... 2024-03-04T17:39:11.801 DEBUG:boto:Data: 2024-03-04T17:39:11.801 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.801 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.801 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.801 DEBUG:boto:Params: {} 2024-03-04T17:39:11.802 DEBUG:boto:Token: None 2024-03-04T17:39:11.802 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.802 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f45355b7450a744-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.806 DEBUG:boto:b'ppfahn-11...1000false...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.806 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.806 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.807 DEBUG:boto:path=/ppfahn-11/?prefix=... 2024-03-04T17:39:11.807 DEBUG:boto:auth_path=/ppfahn-11/?prefix=... 2024-03-04T17:39:11.807 DEBUG:boto:Method: GET 2024-03-04T17:39:11.807 DEBUG:boto:Path: /ppfahn-11/?prefix=... 2024-03-04T17:39:11.807 DEBUG:boto:Data: 2024-03-04T17:39:11.807 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.807 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.807 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.807 DEBUG:boto:Params: {} 2024-03-04T17:39:11.807 DEBUG:boto:Token: None 2024-03-04T17:39:11.808 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.808 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f1cdb8f22b1f57c-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.856 DEBUG:boto:b'ppfahn-11...1000false...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.856 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:11.857 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:11.857 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.857 DEBUG:boto:Path: /ppfahn-11/... 2024-03-04T17:39:11.857 DEBUG:boto:Data: 2024-03-04T17:39:11.857 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.857 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.857 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.857 DEBUG:boto:Params: {} 2024-03-04T17:39:11.857 DEBUG:boto:Token: None 2024-03-04T17:39:11.858 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/... 2024-03-04T17:39:11.858 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NGVHxqoU/1i3r1+yMweirmB32Ms= 2024-03-04T17:39:11.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NGVHxqoU/1i3r1+yMweirmB32Ms=', 'Content-Length': '0'} 2024-03-04T17:39:11.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ba4ad884268678e-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.861 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:11.862 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:11.862 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.862 DEBUG:boto:Path: /ppfahn-11/... 2024-03-04T17:39:11.862 DEBUG:boto:Data: 2024-03-04T17:39:11.862 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.862 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.862 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.862 DEBUG:boto:Params: {} 2024-03-04T17:39:11.862 DEBUG:boto:Token: None 2024-03-04T17:39:11.862 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/... 2024-03-04T17:39:11.863 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NGVHxqoU/1i3r1+yMweirmB32Ms= 2024-03-04T17:39:11.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NGVHxqoU/1i3r1+yMweirmB32Ms=', 'Content-Length': '0'} 2024-03-04T17:39:11.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c9c53bc640a5b0b-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:11.866 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:11.866 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:11.866 DEBUG:boto:Method: GET 2024-03-04T17:39:11.866 DEBUG:boto:Path: /ppfahn-11/... 2024-03-04T17:39:11.866 DEBUG:boto:Data: 2024-03-04T17:39:11.866 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.866 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.867 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.867 DEBUG:boto:Params: {} 2024-03-04T17:39:11.867 DEBUG:boto:Token: None 2024-03-04T17:39:11.867 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/... 2024-03-04T17:39:11.867 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Ebi4j3+CI/mJjX/Wn1nB4PpLPm4= 2024-03-04T17:39:11.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Ebi4j3+CI/mJjX/Wn1nB4PpLPm4=', 'Content-Length': '0'} 2024-03-04T17:39:11.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f0b336cd48f5ea7-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.870 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:11.871 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:11.871 DEBUG:boto:Method: GET 2024-03-04T17:39:11.871 DEBUG:boto:Path: /ppfahn-11/... 2024-03-04T17:39:11.871 DEBUG:boto:Data: 2024-03-04T17:39:11.871 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.871 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.871 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.871 DEBUG:boto:Params: {} 2024-03-04T17:39:11.871 DEBUG:boto:Token: None 2024-03-04T17:39:11.871 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/... 2024-03-04T17:39:11.872 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Ebi4j3+CI/mJjX/Wn1nB4PpLPm4= 2024-03-04T17:39:11.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Ebi4j3+CI/mJjX/Wn1nB4PpLPm4=', 'Content-Length': '0'} 2024-03-04T17:39:11.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008caf18bd4448397a-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.875 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.875 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.875 DEBUG:boto:path=/ppfahn-11/?prefix=... 2024-03-04T17:39:11.875 DEBUG:boto:auth_path=/ppfahn-11/?prefix=... 2024-03-04T17:39:11.875 DEBUG:boto:Method: GET 2024-03-04T17:39:11.875 DEBUG:boto:Path: /ppfahn-11/?prefix=... 2024-03-04T17:39:11.875 DEBUG:boto:Data: 2024-03-04T17:39:11.875 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.875 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.875 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.875 DEBUG:boto:Params: {} 2024-03-04T17:39:11.876 DEBUG:boto:Token: None 2024-03-04T17:39:11.876 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.876 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000963c9923efe4160d-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.924 DEBUG:boto:b'ppfahn-11...1000false...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.924 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.925 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.925 DEBUG:boto:path=/ppfahn-11/?prefix=... 2024-03-04T17:39:11.925 DEBUG:boto:auth_path=/ppfahn-11/?prefix=... 2024-03-04T17:39:11.925 DEBUG:boto:Method: GET 2024-03-04T17:39:11.925 DEBUG:boto:Path: /ppfahn-11/?prefix=... 2024-03-04T17:39:11.925 DEBUG:boto:Data: 2024-03-04T17:39:11.925 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.925 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.925 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.925 DEBUG:boto:Params: {} 2024-03-04T17:39:11.925 DEBUG:boto:Token: None 2024-03-04T17:39:11.925 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.926 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:11.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cf177a6ae07cd15-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.930 DEBUG:boto:b'ppfahn-11...1000false...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:11.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:11.930 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:11.931 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:11.931 DEBUG:boto:path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.931 DEBUG:boto:auth_path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.931 DEBUG:boto:Method: GET 2024-03-04T17:39:11.931 DEBUG:boto:Path: /ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.931 DEBUG:boto:Data: 2024-03-04T17:39:11.931 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.931 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.931 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.931 DEBUG:boto:Params: {} 2024-03-04T17:39:11.932 DEBUG:boto:Token: None 2024-03-04T17:39:11.932 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.932 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Zg3E+/RdT3E/wAJYoWCOZQgOdAs= 2024-03-04T17:39:11.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Zg3E+/RdT3E/wAJYoWCOZQgOdAs=', 'Content-Length': '0'} 2024-03-04T17:39:11.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e071002ab3e4572-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.935 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:11.936 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:11.936 DEBUG:boto:path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.936 DEBUG:boto:auth_path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.936 DEBUG:boto:Method: GET 2024-03-04T17:39:11.936 DEBUG:boto:Path: /ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.936 DEBUG:boto:Data: 2024-03-04T17:39:11.936 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.936 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.936 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.936 DEBUG:boto:Params: {} 2024-03-04T17:39:11.936 DEBUG:boto:Token: None 2024-03-04T17:39:11.937 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.937 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Zg3E+/RdT3E/wAJYoWCOZQgOdAs= 2024-03-04T17:39:11.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Zg3E+/RdT3E/wAJYoWCOZQgOdAs=', 'Content-Length': '0'} 2024-03-04T17:39:11.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006e6f4a92df57c94-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.980 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:11.980 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:11.980 DEBUG:boto:path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.981 DEBUG:boto:auth_path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.981 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.981 DEBUG:boto:Path: /ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.981 DEBUG:boto:Data: 2024-03-04T17:39:11.981 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.981 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.981 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.981 DEBUG:boto:Params: {} 2024-03-04T17:39:11.981 DEBUG:boto:Token: None 2024-03-04T17:39:11.981 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.982 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:d/ylUac6N0VrH69qGxtm3q8x1u8= 2024-03-04T17:39:11.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:d/ylUac6N0VrH69qGxtm3q8x1u8=', 'Content-Length': '0'} 2024-03-04T17:39:11.984 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f66298b5acac6ae-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.985 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:11.985 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:11.985 DEBUG:boto:path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.985 DEBUG:boto:auth_path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.985 DEBUG:boto:Method: HEAD 2024-03-04T17:39:11.985 DEBUG:boto:Path: /ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.985 DEBUG:boto:Data: 2024-03-04T17:39:11.985 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.985 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.985 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.985 DEBUG:boto:Params: {} 2024-03-04T17:39:11.986 DEBUG:boto:Token: None 2024-03-04T17:39:11.986 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.986 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:d/ylUac6N0VrH69qGxtm3q8x1u8= 2024-03-04T17:39:11.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:d/ylUac6N0VrH69qGxtm3q8x1u8=', 'Content-Length': '0'} 2024-03-04T17:39:11.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7680dc2a3213aa0-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:11.989 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:11.989 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:11.989 DEBUG:boto:path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.989 DEBUG:boto:auth_path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.989 DEBUG:boto:Method: GET 2024-03-04T17:39:11.989 DEBUG:boto:Path: /ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.990 DEBUG:boto:Data: 2024-03-04T17:39:11.990 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.990 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.990 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.990 DEBUG:boto:Params: {} 2024-03-04T17:39:11.990 DEBUG:boto:Token: None 2024-03-04T17:39:11.990 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.990 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Zg3E+/RdT3E/wAJYoWCOZQgOdAs= 2024-03-04T17:39:11.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Zg3E+/RdT3E/wAJYoWCOZQgOdAs=', 'Content-Length': '0'} 2024-03-04T17:39:11.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c56af65f4d012f72-0065e6073f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.993 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:11.993 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:11.994 DEBUG:boto:path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.994 DEBUG:boto:auth_path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.994 DEBUG:boto:Method: GET 2024-03-04T17:39:11.994 DEBUG:boto:Path: /ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.994 DEBUG:boto:Data: 2024-03-04T17:39:11.994 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.994 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:11.994 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.994 DEBUG:boto:Params: {} 2024-03-04T17:39:11.994 DEBUG:boto:Token: None 2024-03-04T17:39:11.994 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/.o?versionId=null 2024-03-04T17:39:11.995 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Zg3E+/RdT3E/wAJYoWCOZQgOdAs= 2024-03-04T17:39:11.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Zg3E+/RdT3E/wAJYoWCOZQgOdAs=', 'Content-Length': '0'} 2024-03-04T17:39:11.997 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdc46d425eb47328-0065e6073f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:11.998 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:11.998 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:11.998 DEBUG:boto:path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:11.998 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:11.998 DEBUG:boto:Method: GET 2024-03-04T17:39:11.998 DEBUG:boto:Path: /ppfahn-11/?prefix=.o 2024-03-04T17:39:11.998 DEBUG:boto:Data: 2024-03-04T17:39:11.998 DEBUG:boto:Headers: {} 2024-03-04T17:39:11.998 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:11.999 DEBUG:boto:Port: 8000 2024-03-04T17:39:11.999 DEBUG:boto:Params: {} 2024-03-04T17:39:11.999 DEBUG:boto:Token: None 2024-03-04T17:39:11.999 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:11 GMT /ppfahn-11/ 2024-03-04T17:39:11.999 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY= 2024-03-04T17:39:11.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iPFa6k4hMdFDfO5nnvlVAQnCKIY=', 'Content-Length': '0'} 2024-03-04T17:39:12.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003856a2e59e6743ec-0065e60740-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.044 DEBUG:boto:b'ppfahn-11.o1000false.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.044 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.045 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.045 DEBUG:boto:path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:12.045 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:12.045 DEBUG:boto:Method: GET 2024-03-04T17:39:12.045 DEBUG:boto:Path: /ppfahn-11/?prefix=.o 2024-03-04T17:39:12.045 DEBUG:boto:Data: 2024-03-04T17:39:12.045 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.045 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.045 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.045 DEBUG:boto:Params: {} 2024-03-04T17:39:12.045 DEBUG:boto:Token: None 2024-03-04T17:39:12.046 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.046 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009792681ebb9c5cfa-0065e60740-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.050 DEBUG:boto:b'ppfahn-11.o1000false.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.051 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:12.051 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:12.051 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.051 DEBUG:boto:Path: /ppfahn-11/.o 2024-03-04T17:39:12.051 DEBUG:boto:Data: 2024-03-04T17:39:12.051 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.052 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.052 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.052 DEBUG:boto:Params: {} 2024-03-04T17:39:12.052 DEBUG:boto:Token: None 2024-03-04T17:39:12.052 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o 2024-03-04T17:39:12.052 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:j12twtmddh3OiOyo9JyGZFdmQHM= 2024-03-04T17:39:12.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:j12twtmddh3OiOyo9JyGZFdmQHM=', 'Content-Length': '0'} 2024-03-04T17:39:12.055 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004439e4acd8c1abf0-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.055 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:12.055 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:12.055 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.056 DEBUG:boto:Path: /ppfahn-11/.o 2024-03-04T17:39:12.056 DEBUG:boto:Data: 2024-03-04T17:39:12.056 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.056 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.056 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.056 DEBUG:boto:Params: {} 2024-03-04T17:39:12.056 DEBUG:boto:Token: None 2024-03-04T17:39:12.056 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o 2024-03-04T17:39:12.057 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:j12twtmddh3OiOyo9JyGZFdmQHM= 2024-03-04T17:39:12.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:j12twtmddh3OiOyo9JyGZFdmQHM=', 'Content-Length': '0'} 2024-03-04T17:39:12.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2d9bc469e4c4b1d-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:12.060 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:12.060 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:12.060 DEBUG:boto:Method: GET 2024-03-04T17:39:12.060 DEBUG:boto:Path: /ppfahn-11/.o 2024-03-04T17:39:12.060 DEBUG:boto:Data: 2024-03-04T17:39:12.060 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.060 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.061 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.061 DEBUG:boto:Params: {} 2024-03-04T17:39:12.061 DEBUG:boto:Token: None 2024-03-04T17:39:12.061 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o 2024-03-04T17:39:12.061 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GAbSb6oAHtwSlHJYJKNj3P5zD7o= 2024-03-04T17:39:12.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GAbSb6oAHtwSlHJYJKNj3P5zD7o=', 'Content-Length': '0'} 2024-03-04T17:39:12.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4b57006d4c961b2-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.064 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:12.064 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:12.064 DEBUG:boto:Method: GET 2024-03-04T17:39:12.064 DEBUG:boto:Path: /ppfahn-11/.o 2024-03-04T17:39:12.065 DEBUG:boto:Data: 2024-03-04T17:39:12.065 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.065 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.065 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.065 DEBUG:boto:Params: {} 2024-03-04T17:39:12.065 DEBUG:boto:Token: None 2024-03-04T17:39:12.065 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o 2024-03-04T17:39:12.065 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GAbSb6oAHtwSlHJYJKNj3P5zD7o= 2024-03-04T17:39:12.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GAbSb6oAHtwSlHJYJKNj3P5zD7o=', 'Content-Length': '0'} 2024-03-04T17:39:12.069 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000129eff543e9e60ad-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.116 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.116 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.116 DEBUG:boto:path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:12.116 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:12.116 DEBUG:boto:Method: GET 2024-03-04T17:39:12.117 DEBUG:boto:Path: /ppfahn-11/?prefix=.o 2024-03-04T17:39:12.117 DEBUG:boto:Data: 2024-03-04T17:39:12.117 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.117 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.117 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.117 DEBUG:boto:Params: {} 2024-03-04T17:39:12.117 DEBUG:boto:Token: None 2024-03-04T17:39:12.117 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.117 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5931bbde3bb586c-0065e60740-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.122 DEBUG:boto:b'ppfahn-11.o1000false.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.123 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.123 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.123 DEBUG:boto:path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:12.123 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:12.123 DEBUG:boto:Method: GET 2024-03-04T17:39:12.123 DEBUG:boto:Path: /ppfahn-11/?prefix=.o 2024-03-04T17:39:12.123 DEBUG:boto:Data: 2024-03-04T17:39:12.123 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.124 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.124 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.124 DEBUG:boto:Params: {} 2024-03-04T17:39:12.124 DEBUG:boto:Token: None 2024-03-04T17:39:12.124 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.124 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce3b785e8bcc8ad8-0065e60740-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.129 DEBUG:boto:b'ppfahn-11.o1000false.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:12.129 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:12.129 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:12.129 DEBUG:boto:path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.129 DEBUG:boto:auth_path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.130 DEBUG:boto:Method: GET 2024-03-04T17:39:12.130 DEBUG:boto:Path: /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.130 DEBUG:boto:Data: 2024-03-04T17:39:12.130 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.130 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.130 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.130 DEBUG:boto:Params: {} 2024-03-04T17:39:12.130 DEBUG:boto:Token: None 2024-03-04T17:39:12.130 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.130 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cO2KIy7q0Gup4RS4sLfNhGyitQU= 2024-03-04T17:39:12.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cO2KIy7q0Gup4RS4sLfNhGyitQU=', 'Content-Length': '0'} 2024-03-04T17:39:12.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000427cec98a52fb860-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.134 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:12.134 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:12.134 DEBUG:boto:path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.134 DEBUG:boto:auth_path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.134 DEBUG:boto:Method: GET 2024-03-04T17:39:12.134 DEBUG:boto:Path: /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.134 DEBUG:boto:Data: 2024-03-04T17:39:12.134 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.134 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.134 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.134 DEBUG:boto:Params: {} 2024-03-04T17:39:12.135 DEBUG:boto:Token: None 2024-03-04T17:39:12.135 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.135 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cO2KIy7q0Gup4RS4sLfNhGyitQU= 2024-03-04T17:39:12.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cO2KIy7q0Gup4RS4sLfNhGyitQU=', 'Content-Length': '0'} 2024-03-04T17:39:12.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6bc404b6754a513-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.138 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:12.139 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:12.139 DEBUG:boto:path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.139 DEBUG:boto:auth_path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.139 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.139 DEBUG:boto:Path: /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.139 DEBUG:boto:Data: 2024-03-04T17:39:12.139 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.139 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.139 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.139 DEBUG:boto:Params: {} 2024-03-04T17:39:12.140 DEBUG:boto:Token: None 2024-03-04T17:39:12.140 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.140 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cy7BH4IqLvXzt3QYUbcZjhyII20= 2024-03-04T17:39:12.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cy7BH4IqLvXzt3QYUbcZjhyII20=', 'Content-Length': '0'} 2024-03-04T17:39:12.142 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c06d38eb03b60acc-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.143 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:12.143 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:12.143 DEBUG:boto:path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.143 DEBUG:boto:auth_path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.143 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.143 DEBUG:boto:Path: /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.143 DEBUG:boto:Data: 2024-03-04T17:39:12.143 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.143 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.144 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.144 DEBUG:boto:Params: {} 2024-03-04T17:39:12.144 DEBUG:boto:Token: None 2024-03-04T17:39:12.144 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.144 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cy7BH4IqLvXzt3QYUbcZjhyII20= 2024-03-04T17:39:12.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cy7BH4IqLvXzt3QYUbcZjhyII20=', 'Content-Length': '0'} 2024-03-04T17:39:12.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000964e25b76fe63342-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:12.147 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:12.147 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:12.147 DEBUG:boto:path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.148 DEBUG:boto:auth_path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.148 DEBUG:boto:Method: GET 2024-03-04T17:39:12.148 DEBUG:boto:Path: /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.148 DEBUG:boto:Data: 2024-03-04T17:39:12.148 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.148 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.148 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.148 DEBUG:boto:Params: {} 2024-03-04T17:39:12.148 DEBUG:boto:Token: None 2024-03-04T17:39:12.148 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.149 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cO2KIy7q0Gup4RS4sLfNhGyitQU= 2024-03-04T17:39:12.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cO2KIy7q0Gup4RS4sLfNhGyitQU=', 'Content-Length': '0'} 2024-03-04T17:39:12.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003797882a9c7e26f-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.192 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:12.192 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:12.192 DEBUG:boto:path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.192 DEBUG:boto:auth_path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.192 DEBUG:boto:Method: GET 2024-03-04T17:39:12.193 DEBUG:boto:Path: /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.193 DEBUG:boto:Data: 2024-03-04T17:39:12.193 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.193 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.193 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.193 DEBUG:boto:Params: {} 2024-03-04T17:39:12.193 DEBUG:boto:Token: None 2024-03-04T17:39:12.193 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:12.193 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cO2KIy7q0Gup4RS4sLfNhGyitQU= 2024-03-04T17:39:12.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cO2KIy7q0Gup4RS4sLfNhGyitQU=', 'Content-Length': '0'} 2024-03-04T17:39:12.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001057d22529ec001b-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.196 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.196 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.197 DEBUG:boto:path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:12.197 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:12.197 DEBUG:boto:Method: GET 2024-03-04T17:39:12.197 DEBUG:boto:Path: /ppfahn-11/?prefix=.o. 2024-03-04T17:39:12.197 DEBUG:boto:Data: 2024-03-04T17:39:12.197 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.197 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.197 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.197 DEBUG:boto:Params: {} 2024-03-04T17:39:12.197 DEBUG:boto:Token: None 2024-03-04T17:39:12.198 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.198 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006243451553cd641a-0065e60740-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.202 DEBUG:boto:b'ppfahn-11.o.1000false.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.202 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.202 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.202 DEBUG:boto:path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:12.202 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:12.203 DEBUG:boto:Method: GET 2024-03-04T17:39:12.203 DEBUG:boto:Path: /ppfahn-11/?prefix=.o. 2024-03-04T17:39:12.203 DEBUG:boto:Data: 2024-03-04T17:39:12.203 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.203 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.203 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.203 DEBUG:boto:Params: {} 2024-03-04T17:39:12.203 DEBUG:boto:Token: None 2024-03-04T17:39:12.203 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.203 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e94a8ade94b33e8-0065e60740-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.209 DEBUG:boto:b'ppfahn-11.o.1000false.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.209 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:12.209 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:12.209 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.210 DEBUG:boto:Path: /ppfahn-11/.o. 2024-03-04T17:39:12.210 DEBUG:boto:Data: 2024-03-04T17:39:12.210 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.210 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.210 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.210 DEBUG:boto:Params: {} 2024-03-04T17:39:12.210 DEBUG:boto:Token: None 2024-03-04T17:39:12.210 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o. 2024-03-04T17:39:12.210 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WpljsgbzvofaLryqOWlMrOYxuCk= 2024-03-04T17:39:12.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WpljsgbzvofaLryqOWlMrOYxuCk=', 'Content-Length': '0'} 2024-03-04T17:39:12.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000435fd1e8021db0c4-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.213 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:12.213 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:12.213 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.214 DEBUG:boto:Path: /ppfahn-11/.o. 2024-03-04T17:39:12.214 DEBUG:boto:Data: 2024-03-04T17:39:12.214 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.214 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.214 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.214 DEBUG:boto:Params: {} 2024-03-04T17:39:12.214 DEBUG:boto:Token: None 2024-03-04T17:39:12.214 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o. 2024-03-04T17:39:12.215 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WpljsgbzvofaLryqOWlMrOYxuCk= 2024-03-04T17:39:12.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WpljsgbzvofaLryqOWlMrOYxuCk=', 'Content-Length': '0'} 2024-03-04T17:39:12.217 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065df05611239e1c6-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:12.218 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:12.218 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:12.218 DEBUG:boto:Method: GET 2024-03-04T17:39:12.218 DEBUG:boto:Path: /ppfahn-11/.o. 2024-03-04T17:39:12.218 DEBUG:boto:Data: 2024-03-04T17:39:12.218 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.218 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.218 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.218 DEBUG:boto:Params: {} 2024-03-04T17:39:12.219 DEBUG:boto:Token: None 2024-03-04T17:39:12.219 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o. 2024-03-04T17:39:12.219 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Cp20ihsI0bvc3Oy5Mfy22i2d9W8= 2024-03-04T17:39:12.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Cp20ihsI0bvc3Oy5Mfy22i2d9W8=', 'Content-Length': '0'} 2024-03-04T17:39:12.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c0382e284ff7402-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.264 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:12.264 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:12.264 DEBUG:boto:Method: GET 2024-03-04T17:39:12.264 DEBUG:boto:Path: /ppfahn-11/.o. 2024-03-04T17:39:12.264 DEBUG:boto:Data: 2024-03-04T17:39:12.264 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.265 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.265 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.265 DEBUG:boto:Params: {} 2024-03-04T17:39:12.265 DEBUG:boto:Token: None 2024-03-04T17:39:12.265 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/.o. 2024-03-04T17:39:12.265 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Cp20ihsI0bvc3Oy5Mfy22i2d9W8= 2024-03-04T17:39:12.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Cp20ihsI0bvc3Oy5Mfy22i2d9W8=', 'Content-Length': '0'} 2024-03-04T17:39:12.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ac6d3339b8d168f-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.268 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.268 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.268 DEBUG:boto:path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:12.269 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:12.269 DEBUG:boto:Method: GET 2024-03-04T17:39:12.269 DEBUG:boto:Path: /ppfahn-11/?prefix=.o. 2024-03-04T17:39:12.269 DEBUG:boto:Data: 2024-03-04T17:39:12.269 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.269 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.269 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.269 DEBUG:boto:Params: {} 2024-03-04T17:39:12.269 DEBUG:boto:Token: None 2024-03-04T17:39:12.269 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.270 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009526303b27ba8695-0065e60740-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.274 DEBUG:boto:b'ppfahn-11.o.1000false.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.274 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.275 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.275 DEBUG:boto:path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:12.275 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:12.275 DEBUG:boto:Method: GET 2024-03-04T17:39:12.275 DEBUG:boto:Path: /ppfahn-11/?prefix=.o. 2024-03-04T17:39:12.275 DEBUG:boto:Data: 2024-03-04T17:39:12.275 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.275 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.275 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.275 DEBUG:boto:Params: {} 2024-03-04T17:39:12.276 DEBUG:boto:Token: None 2024-03-04T17:39:12.276 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.276 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007459a57901987f90-0065e60740-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.324 DEBUG:boto:b'ppfahn-11.o.1000false.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:12.324 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:12.324 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:12.325 DEBUG:boto:path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.325 DEBUG:boto:auth_path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.325 DEBUG:boto:Method: GET 2024-03-04T17:39:12.325 DEBUG:boto:Path: /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.325 DEBUG:boto:Data: 2024-03-04T17:39:12.325 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.325 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.325 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.325 DEBUG:boto:Params: {} 2024-03-04T17:39:12.325 DEBUG:boto:Token: None 2024-03-04T17:39:12.325 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.326 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e+WFLcgdXbQiFl8gUdI2fiXtT7w= 2024-03-04T17:39:12.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e+WFLcgdXbQiFl8gUdI2fiXtT7w=', 'Content-Length': '0'} 2024-03-04T17:39:12.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005dd3019418ced943-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.329 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:12.329 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:12.329 DEBUG:boto:path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.329 DEBUG:boto:auth_path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.329 DEBUG:boto:Method: GET 2024-03-04T17:39:12.329 DEBUG:boto:Path: /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.329 DEBUG:boto:Data: 2024-03-04T17:39:12.330 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.330 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.330 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.330 DEBUG:boto:Params: {} 2024-03-04T17:39:12.330 DEBUG:boto:Token: None 2024-03-04T17:39:12.330 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.330 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e+WFLcgdXbQiFl8gUdI2fiXtT7w= 2024-03-04T17:39:12.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e+WFLcgdXbQiFl8gUdI2fiXtT7w=', 'Content-Length': '0'} 2024-03-04T17:39:12.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7baff586cbb7671-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.334 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:12.334 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:12.334 DEBUG:boto:path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.334 DEBUG:boto:auth_path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.334 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.334 DEBUG:boto:Path: /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.334 DEBUG:boto:Data: 2024-03-04T17:39:12.335 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.335 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.335 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.335 DEBUG:boto:Params: {} 2024-03-04T17:39:12.335 DEBUG:boto:Token: None 2024-03-04T17:39:12.335 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.335 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:g3kKEcCJ064iO9WIc7Y8X2X7yaQ= 2024-03-04T17:39:12.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:g3kKEcCJ064iO9WIc7Y8X2X7yaQ=', 'Content-Length': '0'} 2024-03-04T17:39:12.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aaff3620867b037b-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.338 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:12.338 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:12.338 DEBUG:boto:path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.339 DEBUG:boto:auth_path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.339 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.339 DEBUG:boto:Path: /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.339 DEBUG:boto:Data: 2024-03-04T17:39:12.339 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.339 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.339 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.339 DEBUG:boto:Params: {} 2024-03-04T17:39:12.339 DEBUG:boto:Token: None 2024-03-04T17:39:12.339 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.340 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:g3kKEcCJ064iO9WIc7Y8X2X7yaQ= 2024-03-04T17:39:12.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:g3kKEcCJ064iO9WIc7Y8X2X7yaQ=', 'Content-Length': '0'} 2024-03-04T17:39:12.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073113d4215563959-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:12.343 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:12.343 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:12.343 DEBUG:boto:path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.343 DEBUG:boto:auth_path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.344 DEBUG:boto:Method: GET 2024-03-04T17:39:12.344 DEBUG:boto:Path: /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.344 DEBUG:boto:Data: 2024-03-04T17:39:12.344 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.344 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.344 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.344 DEBUG:boto:Params: {} 2024-03-04T17:39:12.344 DEBUG:boto:Token: None 2024-03-04T17:39:12.344 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.344 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e+WFLcgdXbQiFl8gUdI2fiXtT7w= 2024-03-04T17:39:12.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e+WFLcgdXbQiFl8gUdI2fiXtT7w=', 'Content-Length': '0'} 2024-03-04T17:39:12.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba4e1d0fd37662ea-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.388 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:12.388 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:12.388 DEBUG:boto:path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.388 DEBUG:boto:auth_path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.389 DEBUG:boto:Method: GET 2024-03-04T17:39:12.389 DEBUG:boto:Path: /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.389 DEBUG:boto:Data: 2024-03-04T17:39:12.389 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.389 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.389 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.389 DEBUG:boto:Params: {} 2024-03-04T17:39:12.389 DEBUG:boto:Token: None 2024-03-04T17:39:12.390 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:12.390 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e+WFLcgdXbQiFl8gUdI2fiXtT7w= 2024-03-04T17:39:12.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e+WFLcgdXbQiFl8gUdI2fiXtT7w=', 'Content-Length': '0'} 2024-03-04T17:39:12.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3808d6fab8cbca2-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.393 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.393 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.393 DEBUG:boto:path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:12.393 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:12.394 DEBUG:boto:Method: GET 2024-03-04T17:39:12.394 DEBUG:boto:Path: /ppfahn-11/?prefix=%3A 2024-03-04T17:39:12.394 DEBUG:boto:Data: 2024-03-04T17:39:12.394 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.394 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.394 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.394 DEBUG:boto:Params: {} 2024-03-04T17:39:12.394 DEBUG:boto:Token: None 2024-03-04T17:39:12.394 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.395 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009bb14770b8d1b07-0065e60740-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.399 DEBUG:boto:b'ppfahn-11:1000false:2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.399 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.400 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.400 DEBUG:boto:path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:12.400 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:12.400 DEBUG:boto:Method: GET 2024-03-04T17:39:12.400 DEBUG:boto:Path: /ppfahn-11/?prefix=%3A 2024-03-04T17:39:12.400 DEBUG:boto:Data: 2024-03-04T17:39:12.400 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.400 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.400 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.401 DEBUG:boto:Params: {} 2024-03-04T17:39:12.401 DEBUG:boto:Token: None 2024-03-04T17:39:12.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.401 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003093ae609cec7e84-0065e60740-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.448 DEBUG:boto:b'ppfahn-11:1000false:2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.448 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:12.449 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:12.449 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.449 DEBUG:boto:Path: /ppfahn-11/%3A 2024-03-04T17:39:12.449 DEBUG:boto:Data: 2024-03-04T17:39:12.449 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.449 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.449 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.449 DEBUG:boto:Params: {} 2024-03-04T17:39:12.449 DEBUG:boto:Token: None 2024-03-04T17:39:12.450 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/%3A 2024-03-04T17:39:12.450 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8sg9klMRnddluLbBXPbDGpKHWyE= 2024-03-04T17:39:12.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8sg9klMRnddluLbBXPbDGpKHWyE=', 'Content-Length': '0'} 2024-03-04T17:39:12.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000869ae8362d9d1738-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.453 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:12.453 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:12.453 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.453 DEBUG:boto:Path: /ppfahn-11/%3A 2024-03-04T17:39:12.453 DEBUG:boto:Data: 2024-03-04T17:39:12.453 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.454 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.454 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.454 DEBUG:boto:Params: {} 2024-03-04T17:39:12.454 DEBUG:boto:Token: None 2024-03-04T17:39:12.454 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/%3A 2024-03-04T17:39:12.454 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8sg9klMRnddluLbBXPbDGpKHWyE= 2024-03-04T17:39:12.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8sg9klMRnddluLbBXPbDGpKHWyE=', 'Content-Length': '0'} 2024-03-04T17:39:12.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029b03c1d3bb4e570-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:12.457 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:12.457 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:12.457 DEBUG:boto:Method: GET 2024-03-04T17:39:12.458 DEBUG:boto:Path: /ppfahn-11/%3A 2024-03-04T17:39:12.458 DEBUG:boto:Data: 2024-03-04T17:39:12.458 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.458 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.458 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.458 DEBUG:boto:Params: {} 2024-03-04T17:39:12.458 DEBUG:boto:Token: None 2024-03-04T17:39:12.458 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/%3A 2024-03-04T17:39:12.458 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JM123ze6mlH2Dvaig2HDwqgOhCk= 2024-03-04T17:39:12.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JM123ze6mlH2Dvaig2HDwqgOhCk=', 'Content-Length': '0'} 2024-03-04T17:39:12.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bad19843c1839c79-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.462 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:12.462 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:12.462 DEBUG:boto:Method: GET 2024-03-04T17:39:12.462 DEBUG:boto:Path: /ppfahn-11/%3A 2024-03-04T17:39:12.462 DEBUG:boto:Data: 2024-03-04T17:39:12.462 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.462 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.462 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.462 DEBUG:boto:Params: {} 2024-03-04T17:39:12.463 DEBUG:boto:Token: None 2024-03-04T17:39:12.463 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/%3A 2024-03-04T17:39:12.463 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JM123ze6mlH2Dvaig2HDwqgOhCk= 2024-03-04T17:39:12.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JM123ze6mlH2Dvaig2HDwqgOhCk=', 'Content-Length': '0'} 2024-03-04T17:39:12.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000581781b67e2d4335-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.466 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.466 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.466 DEBUG:boto:path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:12.466 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:12.467 DEBUG:boto:Method: GET 2024-03-04T17:39:12.467 DEBUG:boto:Path: /ppfahn-11/?prefix=%3A 2024-03-04T17:39:12.467 DEBUG:boto:Data: 2024-03-04T17:39:12.467 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.467 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.467 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.467 DEBUG:boto:Params: {} 2024-03-04T17:39:12.467 DEBUG:boto:Token: None 2024-03-04T17:39:12.467 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.468 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0a56b2dd5ac99be-0065e60740-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.516 DEBUG:boto:b'ppfahn-11:1000false:2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.517 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.517 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.517 DEBUG:boto:path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:12.517 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:12.517 DEBUG:boto:Method: GET 2024-03-04T17:39:12.517 DEBUG:boto:Path: /ppfahn-11/?prefix=%3A 2024-03-04T17:39:12.517 DEBUG:boto:Data: 2024-03-04T17:39:12.517 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.517 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.517 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.518 DEBUG:boto:Params: {} 2024-03-04T17:39:12.518 DEBUG:boto:Token: None 2024-03-04T17:39:12.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.518 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b27deeab098b42b4-0065e60740-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.522 DEBUG:boto:b'ppfahn-11:1000false:2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:12.523 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:12.523 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:12.523 DEBUG:boto:path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.523 DEBUG:boto:auth_path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.523 DEBUG:boto:Method: GET 2024-03-04T17:39:12.523 DEBUG:boto:Path: /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.524 DEBUG:boto:Data: 2024-03-04T17:39:12.524 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.524 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.524 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.524 DEBUG:boto:Params: {} 2024-03-04T17:39:12.524 DEBUG:boto:Token: None 2024-03-04T17:39:12.524 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.525 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gYY5cXOvPK0z8z5VvFYyy3r7cgw= 2024-03-04T17:39:12.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gYY5cXOvPK0z8z5VvFYyy3r7cgw=', 'Content-Length': '0'} 2024-03-04T17:39:12.528 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021b0182fd947f214-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.528 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:12.528 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:12.528 DEBUG:boto:path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.528 DEBUG:boto:auth_path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.528 DEBUG:boto:Method: GET 2024-03-04T17:39:12.529 DEBUG:boto:Path: /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.529 DEBUG:boto:Data: 2024-03-04T17:39:12.529 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.529 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.529 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.529 DEBUG:boto:Params: {} 2024-03-04T17:39:12.529 DEBUG:boto:Token: None 2024-03-04T17:39:12.529 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.530 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gYY5cXOvPK0z8z5VvFYyy3r7cgw= 2024-03-04T17:39:12.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gYY5cXOvPK0z8z5VvFYyy3r7cgw=', 'Content-Length': '0'} 2024-03-04T17:39:12.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e99eeca4db0c9877-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.576 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:12.576 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:12.576 DEBUG:boto:path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.576 DEBUG:boto:auth_path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.577 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.577 DEBUG:boto:Path: /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.577 DEBUG:boto:Data: 2024-03-04T17:39:12.577 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.577 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.577 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.577 DEBUG:boto:Params: {} 2024-03-04T17:39:12.577 DEBUG:boto:Token: None 2024-03-04T17:39:12.577 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.578 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Fd0wT4eWArApHE9D6D4bcUyHfAM= 2024-03-04T17:39:12.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Fd0wT4eWArApHE9D6D4bcUyHfAM=', 'Content-Length': '0'} 2024-03-04T17:39:12.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a622998bbab6911-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.581 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:12.581 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:12.581 DEBUG:boto:path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.581 DEBUG:boto:auth_path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.581 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.581 DEBUG:boto:Path: /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.582 DEBUG:boto:Data: 2024-03-04T17:39:12.582 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.582 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.582 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.582 DEBUG:boto:Params: {} 2024-03-04T17:39:12.582 DEBUG:boto:Token: None 2024-03-04T17:39:12.582 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.582 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Fd0wT4eWArApHE9D6D4bcUyHfAM= 2024-03-04T17:39:12.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Fd0wT4eWArApHE9D6D4bcUyHfAM=', 'Content-Length': '0'} 2024-03-04T17:39:12.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b01bbe141a4bef9c-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:12.586 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:12.586 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:12.586 DEBUG:boto:path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.586 DEBUG:boto:auth_path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.586 DEBUG:boto:Method: GET 2024-03-04T17:39:12.586 DEBUG:boto:Path: /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.586 DEBUG:boto:Data: 2024-03-04T17:39:12.586 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.587 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.587 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.587 DEBUG:boto:Params: {} 2024-03-04T17:39:12.587 DEBUG:boto:Token: None 2024-03-04T17:39:12.587 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.587 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gYY5cXOvPK0z8z5VvFYyy3r7cgw= 2024-03-04T17:39:12.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gYY5cXOvPK0z8z5VvFYyy3r7cgw=', 'Content-Length': '0'} 2024-03-04T17:39:12.590 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f56f96378c23a06e-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.590 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:12.590 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:12.591 DEBUG:boto:path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.591 DEBUG:boto:auth_path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.591 DEBUG:boto:Method: GET 2024-03-04T17:39:12.591 DEBUG:boto:Path: /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.591 DEBUG:boto:Data: 2024-03-04T17:39:12.591 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.591 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.591 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.591 DEBUG:boto:Params: {} 2024-03-04T17:39:12.591 DEBUG:boto:Token: None 2024-03-04T17:39:12.592 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:12.592 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gYY5cXOvPK0z8z5VvFYyy3r7cgw= 2024-03-04T17:39:12.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gYY5cXOvPK0z8z5VvFYyy3r7cgw=', 'Content-Length': '0'} 2024-03-04T17:39:12.595 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070c00ad85d11a7fa-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.595 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.595 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.595 DEBUG:boto:path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:12.595 DEBUG:boto:auth_path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:12.596 DEBUG:boto:Method: GET 2024-03-04T17:39:12.596 DEBUG:boto:Path: /ppfahn-11/?prefix=_myobj 2024-03-04T17:39:12.596 DEBUG:boto:Data: 2024-03-04T17:39:12.596 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.596 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.596 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.596 DEBUG:boto:Params: {} 2024-03-04T17:39:12.596 DEBUG:boto:Token: None 2024-03-04T17:39:12.596 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.597 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062ba73d8f32ef397-0065e60740-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.644 DEBUG:boto:b'ppfahn-11_myobj1000false_myobj2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.644 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.645 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.645 DEBUG:boto:path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:12.645 DEBUG:boto:auth_path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:12.645 DEBUG:boto:Method: GET 2024-03-04T17:39:12.645 DEBUG:boto:Path: /ppfahn-11/?prefix=_myobj 2024-03-04T17:39:12.645 DEBUG:boto:Data: 2024-03-04T17:39:12.645 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.645 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.645 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.646 DEBUG:boto:Params: {} 2024-03-04T17:39:12.646 DEBUG:boto:Token: None 2024-03-04T17:39:12.646 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.646 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e3ff22df76af291-0065e60740-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.651 DEBUG:boto:b'ppfahn-11_myobj1000false_myobj2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.651 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:12.652 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:12.652 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.652 DEBUG:boto:Path: /ppfahn-11/_myobj 2024-03-04T17:39:12.652 DEBUG:boto:Data: 2024-03-04T17:39:12.652 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.652 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.652 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.652 DEBUG:boto:Params: {} 2024-03-04T17:39:12.652 DEBUG:boto:Token: None 2024-03-04T17:39:12.653 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/_myobj 2024-03-04T17:39:12.653 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rQISNpMzLnAx4IRM5L2Dqh30uJE= 2024-03-04T17:39:12.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rQISNpMzLnAx4IRM5L2Dqh30uJE=', 'Content-Length': '0'} 2024-03-04T17:39:12.656 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035f6c2475ebf79d1-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.656 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:12.656 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:12.656 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.656 DEBUG:boto:Path: /ppfahn-11/_myobj 2024-03-04T17:39:12.657 DEBUG:boto:Data: 2024-03-04T17:39:12.657 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.657 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.657 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.657 DEBUG:boto:Params: {} 2024-03-04T17:39:12.657 DEBUG:boto:Token: None 2024-03-04T17:39:12.657 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/_myobj 2024-03-04T17:39:12.657 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rQISNpMzLnAx4IRM5L2Dqh30uJE= 2024-03-04T17:39:12.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rQISNpMzLnAx4IRM5L2Dqh30uJE=', 'Content-Length': '0'} 2024-03-04T17:39:12.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050f3c442a3e61c5a-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:12.661 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:12.661 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:12.661 DEBUG:boto:Method: GET 2024-03-04T17:39:12.661 DEBUG:boto:Path: /ppfahn-11/_myobj 2024-03-04T17:39:12.661 DEBUG:boto:Data: 2024-03-04T17:39:12.661 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.661 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.661 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.661 DEBUG:boto:Params: {} 2024-03-04T17:39:12.661 DEBUG:boto:Token: None 2024-03-04T17:39:12.662 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/_myobj 2024-03-04T17:39:12.662 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lXk5cSgg3IRoXxCuzNIVKKvuG3M= 2024-03-04T17:39:12.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lXk5cSgg3IRoXxCuzNIVKKvuG3M=', 'Content-Length': '0'} 2024-03-04T17:39:12.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3531d89de26ed05-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.665 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:12.665 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:12.665 DEBUG:boto:Method: GET 2024-03-04T17:39:12.665 DEBUG:boto:Path: /ppfahn-11/_myobj 2024-03-04T17:39:12.665 DEBUG:boto:Data: 2024-03-04T17:39:12.665 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.666 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.666 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.666 DEBUG:boto:Params: {} 2024-03-04T17:39:12.666 DEBUG:boto:Token: None 2024-03-04T17:39:12.666 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/_myobj 2024-03-04T17:39:12.666 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lXk5cSgg3IRoXxCuzNIVKKvuG3M= 2024-03-04T17:39:12.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lXk5cSgg3IRoXxCuzNIVKKvuG3M=', 'Content-Length': '0'} 2024-03-04T17:39:12.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000881803e420f1230b-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.712 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.712 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.712 DEBUG:boto:path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:12.712 DEBUG:boto:auth_path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:12.712 DEBUG:boto:Method: GET 2024-03-04T17:39:12.713 DEBUG:boto:Path: /ppfahn-11/?prefix=_myobj 2024-03-04T17:39:12.713 DEBUG:boto:Data: 2024-03-04T17:39:12.713 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.713 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.713 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.713 DEBUG:boto:Params: {} 2024-03-04T17:39:12.713 DEBUG:boto:Token: None 2024-03-04T17:39:12.713 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.713 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b4f660aa6647c3-0065e60740-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.719 DEBUG:boto:b'ppfahn-11_myobj1000false_myobj2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.719 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.719 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.719 DEBUG:boto:path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:12.719 DEBUG:boto:auth_path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:12.719 DEBUG:boto:Method: GET 2024-03-04T17:39:12.720 DEBUG:boto:Path: /ppfahn-11/?prefix=_myobj 2024-03-04T17:39:12.720 DEBUG:boto:Data: 2024-03-04T17:39:12.720 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.720 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.720 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.720 DEBUG:boto:Params: {} 2024-03-04T17:39:12.720 DEBUG:boto:Token: None 2024-03-04T17:39:12.720 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.720 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035f7f5d8fc7bd661-0065e60740-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.725 DEBUG:boto:b'ppfahn-11_myobj1000false_myobj2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:12.726 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:12.726 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:12.726 DEBUG:boto:path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.726 DEBUG:boto:auth_path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.726 DEBUG:boto:Method: GET 2024-03-04T17:39:12.726 DEBUG:boto:Path: /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.726 DEBUG:boto:Data: 2024-03-04T17:39:12.726 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.726 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.726 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.727 DEBUG:boto:Params: {} 2024-03-04T17:39:12.727 DEBUG:boto:Token: None 2024-03-04T17:39:12.727 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.727 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:y/nhjSZqcIjaGVxDy3CsNRzCDiM= 2024-03-04T17:39:12.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:y/nhjSZqcIjaGVxDy3CsNRzCDiM=', 'Content-Length': '0'} 2024-03-04T17:39:12.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba05bd6a35990c49-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.730 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:12.730 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:12.730 DEBUG:boto:path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.730 DEBUG:boto:auth_path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.731 DEBUG:boto:Method: GET 2024-03-04T17:39:12.731 DEBUG:boto:Path: /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.731 DEBUG:boto:Data: 2024-03-04T17:39:12.731 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.731 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.731 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.731 DEBUG:boto:Params: {} 2024-03-04T17:39:12.731 DEBUG:boto:Token: None 2024-03-04T17:39:12.731 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.731 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:y/nhjSZqcIjaGVxDy3CsNRzCDiM= 2024-03-04T17:39:12.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:y/nhjSZqcIjaGVxDy3CsNRzCDiM=', 'Content-Length': '0'} 2024-03-04T17:39:12.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5d3541813214e38-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.735 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:12.735 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:12.735 DEBUG:boto:path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.735 DEBUG:boto:auth_path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.735 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.735 DEBUG:boto:Path: /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.735 DEBUG:boto:Data: 2024-03-04T17:39:12.736 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.736 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.736 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.736 DEBUG:boto:Params: {} 2024-03-04T17:39:12.736 DEBUG:boto:Token: None 2024-03-04T17:39:12.736 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.736 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iRxHM0cB+lAjQyjUl1A5Mw89rDY= 2024-03-04T17:39:12.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iRxHM0cB+lAjQyjUl1A5Mw89rDY=', 'Content-Length': '0'} 2024-03-04T17:39:12.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba18f71011a766b3-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.739 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:12.739 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:12.739 DEBUG:boto:path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.740 DEBUG:boto:auth_path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.740 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.740 DEBUG:boto:Path: /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.740 DEBUG:boto:Data: 2024-03-04T17:39:12.740 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.740 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.740 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.740 DEBUG:boto:Params: {} 2024-03-04T17:39:12.740 DEBUG:boto:Token: None 2024-03-04T17:39:12.740 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.741 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iRxHM0cB+lAjQyjUl1A5Mw89rDY= 2024-03-04T17:39:12.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iRxHM0cB+lAjQyjUl1A5Mw89rDY=', 'Content-Length': '0'} 2024-03-04T17:39:12.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c734ee41ab4ceeb-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:12.744 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:12.744 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:12.744 DEBUG:boto:path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.744 DEBUG:boto:auth_path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.744 DEBUG:boto:Method: GET 2024-03-04T17:39:12.744 DEBUG:boto:Path: /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.745 DEBUG:boto:Data: 2024-03-04T17:39:12.745 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.745 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.745 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.745 DEBUG:boto:Params: {} 2024-03-04T17:39:12.745 DEBUG:boto:Token: None 2024-03-04T17:39:12.745 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.745 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:y/nhjSZqcIjaGVxDy3CsNRzCDiM= 2024-03-04T17:39:12.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:y/nhjSZqcIjaGVxDy3CsNRzCDiM=', 'Content-Length': '0'} 2024-03-04T17:39:12.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000376262ef5b11e3d6-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.792 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:12.792 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:12.792 DEBUG:boto:path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.792 DEBUG:boto:auth_path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.792 DEBUG:boto:Method: GET 2024-03-04T17:39:12.793 DEBUG:boto:Path: /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.793 DEBUG:boto:Data: 2024-03-04T17:39:12.793 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.793 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.793 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.793 DEBUG:boto:Params: {} 2024-03-04T17:39:12.793 DEBUG:boto:Token: None 2024-03-04T17:39:12.793 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:12.793 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:y/nhjSZqcIjaGVxDy3CsNRzCDiM= 2024-03-04T17:39:12.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:y/nhjSZqcIjaGVxDy3CsNRzCDiM=', 'Content-Length': '0'} 2024-03-04T17:39:12.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a2a322607dd43a7-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.797 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.797 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.797 DEBUG:boto:path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:12.797 DEBUG:boto:auth_path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:12.797 DEBUG:boto:Method: GET 2024-03-04T17:39:12.797 DEBUG:boto:Path: /ppfahn-11/?prefix=myobj 2024-03-04T17:39:12.797 DEBUG:boto:Data: 2024-03-04T17:39:12.798 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.798 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.798 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.798 DEBUG:boto:Params: {} 2024-03-04T17:39:12.798 DEBUG:boto:Token: None 2024-03-04T17:39:12.798 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.798 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e866e2b353c2e142-0065e60740-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.803 DEBUG:boto:b'ppfahn-11myobj1000falsemyobj2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.803 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.803 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.804 DEBUG:boto:path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:12.804 DEBUG:boto:auth_path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:12.804 DEBUG:boto:Method: GET 2024-03-04T17:39:12.804 DEBUG:boto:Path: /ppfahn-11/?prefix=myobj 2024-03-04T17:39:12.804 DEBUG:boto:Data: 2024-03-04T17:39:12.804 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.804 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.804 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.804 DEBUG:boto:Params: {} 2024-03-04T17:39:12.804 DEBUG:boto:Token: None 2024-03-04T17:39:12.805 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.805 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4a8ead40901135e-0065e60740-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.852 DEBUG:boto:b'ppfahn-11myobj1000falsemyobj2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.852 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:12.852 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:12.853 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.853 DEBUG:boto:Path: /ppfahn-11/myobj 2024-03-04T17:39:12.853 DEBUG:boto:Data: 2024-03-04T17:39:12.853 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.853 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.853 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.853 DEBUG:boto:Params: {} 2024-03-04T17:39:12.853 DEBUG:boto:Token: None 2024-03-04T17:39:12.853 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/myobj 2024-03-04T17:39:12.853 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W2vzXJ9C2xYUIL8MhwtGVDME53w= 2024-03-04T17:39:12.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W2vzXJ9C2xYUIL8MhwtGVDME53w=', 'Content-Length': '0'} 2024-03-04T17:39:12.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c15834ac067c38c-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.856 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:12.857 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:12.857 DEBUG:boto:Method: HEAD 2024-03-04T17:39:12.857 DEBUG:boto:Path: /ppfahn-11/myobj 2024-03-04T17:39:12.857 DEBUG:boto:Data: 2024-03-04T17:39:12.857 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.857 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.857 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.857 DEBUG:boto:Params: {} 2024-03-04T17:39:12.857 DEBUG:boto:Token: None 2024-03-04T17:39:12.857 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/myobj 2024-03-04T17:39:12.858 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W2vzXJ9C2xYUIL8MhwtGVDME53w= 2024-03-04T17:39:12.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W2vzXJ9C2xYUIL8MhwtGVDME53w=', 'Content-Length': '0'} 2024-03-04T17:39:12.860 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6db02744d93d076-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:12.861 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:12.861 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:12.861 DEBUG:boto:Method: GET 2024-03-04T17:39:12.861 DEBUG:boto:Path: /ppfahn-11/myobj 2024-03-04T17:39:12.861 DEBUG:boto:Data: 2024-03-04T17:39:12.861 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.861 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.861 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.861 DEBUG:boto:Params: {} 2024-03-04T17:39:12.861 DEBUG:boto:Token: None 2024-03-04T17:39:12.862 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/myobj 2024-03-04T17:39:12.862 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Lp3IPNUJXbFSQcyWJ3OMvlGCr6w= 2024-03-04T17:39:12.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Lp3IPNUJXbFSQcyWJ3OMvlGCr6w=', 'Content-Length': '0'} 2024-03-04T17:39:12.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d6c9d3435d7beeb-0065e60740-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.865 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:12.865 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:12.865 DEBUG:boto:Method: GET 2024-03-04T17:39:12.865 DEBUG:boto:Path: /ppfahn-11/myobj 2024-03-04T17:39:12.865 DEBUG:boto:Data: 2024-03-04T17:39:12.866 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.866 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.866 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.866 DEBUG:boto:Params: {} 2024-03-04T17:39:12.866 DEBUG:boto:Token: None 2024-03-04T17:39:12.866 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/myobj 2024-03-04T17:39:12.866 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Lp3IPNUJXbFSQcyWJ3OMvlGCr6w= 2024-03-04T17:39:12.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Lp3IPNUJXbFSQcyWJ3OMvlGCr6w=', 'Content-Length': '0'} 2024-03-04T17:39:12.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003950ca6468ecaca3-0065e60740-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.870 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.870 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.870 DEBUG:boto:path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:12.870 DEBUG:boto:auth_path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:12.870 DEBUG:boto:Method: GET 2024-03-04T17:39:12.870 DEBUG:boto:Path: /ppfahn-11/?prefix=myobj 2024-03-04T17:39:12.870 DEBUG:boto:Data: 2024-03-04T17:39:12.870 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.870 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:12.870 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.871 DEBUG:boto:Params: {} 2024-03-04T17:39:12.871 DEBUG:boto:Token: None 2024-03-04T17:39:12.871 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.871 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e9701b67d8732de-0065e60740-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.916 DEBUG:boto:b'ppfahn-11myobj1000falsemyobj2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.916 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:12.916 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:12.917 DEBUG:boto:path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:12.917 DEBUG:boto:auth_path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:12.917 DEBUG:boto:Method: GET 2024-03-04T17:39:12.917 DEBUG:boto:Path: /ppfahn-11/?prefix=myobj 2024-03-04T17:39:12.917 DEBUG:boto:Data: 2024-03-04T17:39:12.917 DEBUG:boto:Headers: {} 2024-03-04T17:39:12.917 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:12.917 DEBUG:boto:Port: 8000 2024-03-04T17:39:12.917 DEBUG:boto:Params: {} 2024-03-04T17:39:12.917 DEBUG:boto:Token: None 2024-03-04T17:39:12.917 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:12 GMT /ppfahn-11/ 2024-03-04T17:39:12.918 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50= 2024-03-04T17:39:12.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNInUdRipSrjMcCrbj/cSgEzL50=', 'Content-Length': '0'} 2024-03-04T17:39:12.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd6293a1500db458-0065e60740-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:12.922 DEBUG:boto:b'ppfahn-11myobj1000falsemyobj2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:12.922 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-11 zones={test-zone1, test-zone2} 2024-03-04T17:39:12.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:12.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:12.923 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:39:12.973 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:12.973 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:13.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:13.036+0000 7fc56c63e980 1 bucket sync caught up with source: 2024-03-04T17:39:13.037 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000002.12.6, , 00000000001.11.6, , 00000000001.9.6, 00000000002.7.6, 00000000001.16.6, 00000000001.17.6, 00000000001.17.6, ] 2024-03-04T17:39:13.037 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.12.6, , 00000000001.11.6, , 00000000001.9.6, 00000000002.7.6, 00000000001.16.6, 00000000001.17.6, 00000000001.17.6, ] 2024-03-04T17:39:13.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:13.036+0000 7fc56c63e980 0 bucket checkpoint complete 2024-03-04T17:39:13.059 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-11 zones={test-zone3, test-zone2} 2024-03-04T17:39:13.059 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.059 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.060 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.060 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:39:13.060 DEBUG:boto:Data: 2024-03-04T17:39:13.060 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.060 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.060 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.060 DEBUG:boto:Params: {} 2024-03-04T17:39:13.060 DEBUG:boto:Token: None 2024-03-04T17:39:13.060 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.061 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:P2CRsMOW/2p/YVfXaJnRYKxOfP8= 2024-03-04T17:39:13.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:P2CRsMOW/2p/YVfXaJnRYKxOfP8=', 'Content-Length': '0'} 2024-03-04T17:39:13.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b187281b7a9aaf6-0065e60741-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.063 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.063 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.063 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.063 DEBUG:boto:Path: /ppfahn-11/ 2024-03-04T17:39:13.063 DEBUG:boto:Data: 2024-03-04T17:39:13.064 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.064 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.064 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.064 DEBUG:boto:Params: {} 2024-03-04T17:39:13.064 DEBUG:boto:Token: None 2024-03-04T17:39:13.064 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.064 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:P2CRsMOW/2p/YVfXaJnRYKxOfP8= 2024-03-04T17:39:13.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:P2CRsMOW/2p/YVfXaJnRYKxOfP8=', 'Content-Length': '0'} 2024-03-04T17:39:13.069 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005adee2a2d2684fec-0065e60741-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.069 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:39:13.069 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.069 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.069 DEBUG:boto:path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:13.069 DEBUG:boto:auth_path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:13.069 DEBUG:boto:Method: GET 2024-03-04T17:39:13.069 DEBUG:boto:Path: /ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:13.069 DEBUG:boto:Data: 2024-03-04T17:39:13.070 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.070 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.070 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.070 DEBUG:boto:Params: {} 2024-03-04T17:39:13.070 DEBUG:boto:Token: None 2024-03-04T17:39:13.070 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/?versions 2024-03-04T17:39:13.070 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wpl7sDaX+K0NU98/4VTo7A4Oku0= 2024-03-04T17:39:13.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wpl7sDaX+K0NU98/4VTo7A4Oku0=', 'Content-Length': '0'} 2024-03-04T17:39:13.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036ee5eca5b69c1ad-0065e60741-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.073 DEBUG:boto:b'ppfahn-11999false&nulltrue2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.074 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T17:39:13.074 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T17:39:13.074 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T17:39:13.074 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T17:39:13.074 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T17:39:13.074 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T17:39:13.074 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T17:39:13.074 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T17:39:13.074 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:13.075 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:39:13.075 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.075 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.075 DEBUG:boto:path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:13.075 DEBUG:boto:auth_path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:13.075 DEBUG:boto:Method: GET 2024-03-04T17:39:13.075 DEBUG:boto:Path: /ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:13.075 DEBUG:boto:Data: 2024-03-04T17:39:13.075 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.075 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.075 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.076 DEBUG:boto:Params: {} 2024-03-04T17:39:13.076 DEBUG:boto:Token: None 2024-03-04T17:39:13.076 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/?versions 2024-03-04T17:39:13.076 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wpl7sDaX+K0NU98/4VTo7A4Oku0= 2024-03-04T17:39:13.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wpl7sDaX+K0NU98/4VTo7A4Oku0=', 'Content-Length': '0'} 2024-03-04T17:39:13.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c4d90005f1e4d83-0065e60741-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.081 DEBUG:boto:b'ppfahn-11999false&nulltrue2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.082 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T17:39:13.082 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T17:39:13.082 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T17:39:13.082 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T17:39:13.082 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T17:39:13.082 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T17:39:13.082 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T17:39:13.082 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T17:39:13.082 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:13.082 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.083 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.083 DEBUG:boto:path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:13.083 DEBUG:boto:auth_path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:13.083 DEBUG:boto:Method: GET 2024-03-04T17:39:13.083 DEBUG:boto:Path: /ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:13.083 DEBUG:boto:Data: 2024-03-04T17:39:13.083 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.083 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.083 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.083 DEBUG:boto:Params: {} 2024-03-04T17:39:13.083 DEBUG:boto:Token: None 2024-03-04T17:39:13.084 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/?versions 2024-03-04T17:39:13.084 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wpl7sDaX+K0NU98/4VTo7A4Oku0= 2024-03-04T17:39:13.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wpl7sDaX+K0NU98/4VTo7A4Oku0=', 'Content-Length': '0'} 2024-03-04T17:39:13.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a6ff25e0d63de97-0065e60741-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.086 DEBUG:boto:b'ppfahn-11999false&nulltrue2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.087 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.087 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.087 DEBUG:boto:path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:13.087 DEBUG:boto:auth_path=/ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:13.087 DEBUG:boto:Method: GET 2024-03-04T17:39:13.088 DEBUG:boto:Path: /ppfahn-11/?versions&max-keys=999 2024-03-04T17:39:13.088 DEBUG:boto:Data: 2024-03-04T17:39:13.088 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.088 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.088 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.088 DEBUG:boto:Params: {} 2024-03-04T17:39:13.088 DEBUG:boto:Token: None 2024-03-04T17:39:13.088 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/?versions 2024-03-04T17:39:13.088 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wpl7sDaX+K0NU98/4VTo7A4Oku0= 2024-03-04T17:39:13.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wpl7sDaX+K0NU98/4VTo7A4Oku0=', 'Content-Length': '0'} 2024-03-04T17:39:13.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c61b787752a24e6-0065e60741-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.093 DEBUG:boto:b'ppfahn-11999false&nulltrue2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:13.094 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:13.094 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:13.094 DEBUG:boto:path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.094 DEBUG:boto:auth_path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.094 DEBUG:boto:Method: GET 2024-03-04T17:39:13.095 DEBUG:boto:Path: /ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.095 DEBUG:boto:Data: 2024-03-04T17:39:13.095 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.095 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.095 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.095 DEBUG:boto:Params: {} 2024-03-04T17:39:13.095 DEBUG:boto:Token: None 2024-03-04T17:39:13.095 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.095 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gjysW/ZKOOShd+Y9AzLgFIJo0u8= 2024-03-04T17:39:13.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gjysW/ZKOOShd+Y9AzLgFIJo0u8=', 'Content-Length': '0'} 2024-03-04T17:39:13.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001ab57d229781c10-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.098 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:13.098 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:13.098 DEBUG:boto:path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.098 DEBUG:boto:auth_path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.098 DEBUG:boto:Method: GET 2024-03-04T17:39:13.098 DEBUG:boto:Path: /ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.098 DEBUG:boto:Data: 2024-03-04T17:39:13.098 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.099 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.099 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.099 DEBUG:boto:Params: {} 2024-03-04T17:39:13.099 DEBUG:boto:Token: None 2024-03-04T17:39:13.099 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.099 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gjysW/ZKOOShd+Y9AzLgFIJo0u8= 2024-03-04T17:39:13.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gjysW/ZKOOShd+Y9AzLgFIJo0u8=', 'Content-Length': '0'} 2024-03-04T17:39:13.102 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4a6a5837ee1ef98-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.144 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:13.144 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:13.144 DEBUG:boto:path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.144 DEBUG:boto:auth_path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.145 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.145 DEBUG:boto:Path: /ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.145 DEBUG:boto:Data: 2024-03-04T17:39:13.145 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.145 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.145 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.145 DEBUG:boto:Params: {} 2024-03-04T17:39:13.145 DEBUG:boto:Token: None 2024-03-04T17:39:13.145 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.145 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Lh93Vn6p3kvq5p5Ea9g6+ogEf4U= 2024-03-04T17:39:13.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Lh93Vn6p3kvq5p5Ea9g6+ogEf4U=', 'Content-Length': '0'} 2024-03-04T17:39:13.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081f3eaf3e1bed5b9-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.149 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:13.149 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:13.149 DEBUG:boto:path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.149 DEBUG:boto:auth_path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.149 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.149 DEBUG:boto:Path: /ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.149 DEBUG:boto:Data: 2024-03-04T17:39:13.149 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.150 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.150 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.150 DEBUG:boto:Params: {} 2024-03-04T17:39:13.150 DEBUG:boto:Token: None 2024-03-04T17:39:13.150 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.150 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Lh93Vn6p3kvq5p5Ea9g6+ogEf4U= 2024-03-04T17:39:13.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Lh93Vn6p3kvq5p5Ea9g6+ogEf4U=', 'Content-Length': '0'} 2024-03-04T17:39:13.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7e6150a65493e5b-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:13.153 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:13.153 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:13.153 DEBUG:boto:path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.154 DEBUG:boto:auth_path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.154 DEBUG:boto:Method: GET 2024-03-04T17:39:13.154 DEBUG:boto:Path: /ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.154 DEBUG:boto:Data: 2024-03-04T17:39:13.154 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.154 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.154 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.154 DEBUG:boto:Params: {} 2024-03-04T17:39:13.154 DEBUG:boto:Token: None 2024-03-04T17:39:13.154 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.154 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gjysW/ZKOOShd+Y9AzLgFIJo0u8= 2024-03-04T17:39:13.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gjysW/ZKOOShd+Y9AzLgFIJo0u8=', 'Content-Length': '0'} 2024-03-04T17:39:13.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000300a278515293004-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.200 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:13.200 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:13.200 DEBUG:boto:path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.200 DEBUG:boto:auth_path=/ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.200 DEBUG:boto:Method: GET 2024-03-04T17:39:13.201 DEBUG:boto:Path: /ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.201 DEBUG:boto:Data: 2024-03-04T17:39:13.201 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.201 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.201 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.201 DEBUG:boto:Params: {} 2024-03-04T17:39:13.201 DEBUG:boto:Token: None 2024-03-04T17:39:13.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/%26?versionId=null 2024-03-04T17:39:13.201 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gjysW/ZKOOShd+Y9AzLgFIJo0u8= 2024-03-04T17:39:13.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gjysW/ZKOOShd+Y9AzLgFIJo0u8=', 'Content-Length': '0'} 2024-03-04T17:39:13.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e41b0d28479101ac-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.204 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.205 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.205 DEBUG:boto:path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:13.205 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:13.205 DEBUG:boto:Method: GET 2024-03-04T17:39:13.205 DEBUG:boto:Path: /ppfahn-11/?prefix=%26 2024-03-04T17:39:13.205 DEBUG:boto:Data: 2024-03-04T17:39:13.205 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.205 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.205 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.205 DEBUG:boto:Params: {} 2024-03-04T17:39:13.205 DEBUG:boto:Token: None 2024-03-04T17:39:13.206 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.206 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002945d446a9d01d82-0065e60741-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.252 DEBUG:boto:b'ppfahn-11&1000false&2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.252 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.252 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.253 DEBUG:boto:path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:13.253 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:13.253 DEBUG:boto:Method: GET 2024-03-04T17:39:13.253 DEBUG:boto:Path: /ppfahn-11/?prefix=%26 2024-03-04T17:39:13.253 DEBUG:boto:Data: 2024-03-04T17:39:13.253 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.253 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.253 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.253 DEBUG:boto:Params: {} 2024-03-04T17:39:13.253 DEBUG:boto:Token: None 2024-03-04T17:39:13.254 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.254 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c2b1a2ab2daf691-0065e60741-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.258 DEBUG:boto:b'ppfahn-11&1000false&2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.259 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:13.259 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:13.259 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.259 DEBUG:boto:Path: /ppfahn-11/%26 2024-03-04T17:39:13.259 DEBUG:boto:Data: 2024-03-04T17:39:13.259 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.259 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.260 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.260 DEBUG:boto:Params: {} 2024-03-04T17:39:13.260 DEBUG:boto:Token: None 2024-03-04T17:39:13.260 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/%26 2024-03-04T17:39:13.260 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+OA886R17snr2gr6EbgHbwt7SNc= 2024-03-04T17:39:13.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+OA886R17snr2gr6EbgHbwt7SNc=', 'Content-Length': '0'} 2024-03-04T17:39:13.263 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000598b0bf0094ea5d1-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.263 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:13.263 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:13.264 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.264 DEBUG:boto:Path: /ppfahn-11/%26 2024-03-04T17:39:13.264 DEBUG:boto:Data: 2024-03-04T17:39:13.264 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.264 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.264 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.264 DEBUG:boto:Params: {} 2024-03-04T17:39:13.264 DEBUG:boto:Token: None 2024-03-04T17:39:13.264 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/%26 2024-03-04T17:39:13.265 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+OA886R17snr2gr6EbgHbwt7SNc= 2024-03-04T17:39:13.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+OA886R17snr2gr6EbgHbwt7SNc=', 'Content-Length': '0'} 2024-03-04T17:39:13.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ab22bb963e1f3bf-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:13.268 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:13.268 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:13.268 DEBUG:boto:Method: GET 2024-03-04T17:39:13.268 DEBUG:boto:Path: /ppfahn-11/%26 2024-03-04T17:39:13.268 DEBUG:boto:Data: 2024-03-04T17:39:13.268 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.268 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.268 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.268 DEBUG:boto:Params: {} 2024-03-04T17:39:13.268 DEBUG:boto:Token: None 2024-03-04T17:39:13.269 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/%26 2024-03-04T17:39:13.269 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZrVE4Z1LXG8quuJXYFixK6JgfE0= 2024-03-04T17:39:13.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZrVE4Z1LXG8quuJXYFixK6JgfE0=', 'Content-Length': '0'} 2024-03-04T17:39:13.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005aa5f981a2a90928-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.312 DEBUG:boto:path=/ppfahn-11/%26 2024-03-04T17:39:13.312 DEBUG:boto:auth_path=/ppfahn-11/%26 2024-03-04T17:39:13.312 DEBUG:boto:Method: GET 2024-03-04T17:39:13.312 DEBUG:boto:Path: /ppfahn-11/%26 2024-03-04T17:39:13.312 DEBUG:boto:Data: 2024-03-04T17:39:13.312 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.313 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.313 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.313 DEBUG:boto:Params: {} 2024-03-04T17:39:13.313 DEBUG:boto:Token: None 2024-03-04T17:39:13.313 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/%26 2024-03-04T17:39:13.313 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZrVE4Z1LXG8quuJXYFixK6JgfE0= 2024-03-04T17:39:13.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZrVE4Z1LXG8quuJXYFixK6JgfE0=', 'Content-Length': '0'} 2024-03-04T17:39:13.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc870064943da15c-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.316 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.316 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.316 DEBUG:boto:path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:13.317 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:13.317 DEBUG:boto:Method: GET 2024-03-04T17:39:13.317 DEBUG:boto:Path: /ppfahn-11/?prefix=%26 2024-03-04T17:39:13.317 DEBUG:boto:Data: 2024-03-04T17:39:13.317 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.317 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.317 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.317 DEBUG:boto:Params: {} 2024-03-04T17:39:13.317 DEBUG:boto:Token: None 2024-03-04T17:39:13.317 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.318 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d45a63c9a33a44f-0065e60741-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.364 DEBUG:boto:b'ppfahn-11&1000false&2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.364 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.364 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.365 DEBUG:boto:path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:13.365 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%26 2024-03-04T17:39:13.365 DEBUG:boto:Method: GET 2024-03-04T17:39:13.365 DEBUG:boto:Path: /ppfahn-11/?prefix=%26 2024-03-04T17:39:13.365 DEBUG:boto:Data: 2024-03-04T17:39:13.365 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.365 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.365 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.365 DEBUG:boto:Params: {} 2024-03-04T17:39:13.365 DEBUG:boto:Token: None 2024-03-04T17:39:13.365 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.365 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000526615720e7ce411-0065e60741-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.370 DEBUG:boto:b'ppfahn-11&1000false&2024-03-04T17:36:30.450Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:13.370 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:13.370 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:13.370 DEBUG:boto:path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:13.370 DEBUG:boto:auth_path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:13.371 DEBUG:boto:Method: GET 2024-03-04T17:39:13.371 DEBUG:boto:Path: /ppfahn-11/.?versionId=null 2024-03-04T17:39:13.371 DEBUG:boto:Data: 2024-03-04T17:39:13.371 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.371 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.371 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.371 DEBUG:boto:Params: {} 2024-03-04T17:39:13.371 DEBUG:boto:Token: None 2024-03-04T17:39:13.371 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/.?versionId=null 2024-03-04T17:39:13.371 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YwKv3vukGgQuVevA1lKPuXkDNpk= 2024-03-04T17:39:13.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YwKv3vukGgQuVevA1lKPuXkDNpk=', 'Content-Length': '0'} 2024-03-04T17:39:13.375 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009afb2911d2a07fb7-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.416 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:13.416 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:13.416 DEBUG:boto:path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:13.416 DEBUG:boto:auth_path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:13.416 DEBUG:boto:Method: GET 2024-03-04T17:39:13.417 DEBUG:boto:Path: /ppfahn-11/.?versionId=null 2024-03-04T17:39:13.417 DEBUG:boto:Data: 2024-03-04T17:39:13.417 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.417 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.417 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.417 DEBUG:boto:Params: {} 2024-03-04T17:39:13.417 DEBUG:boto:Token: None 2024-03-04T17:39:13.417 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/.?versionId=null 2024-03-04T17:39:13.418 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YwKv3vukGgQuVevA1lKPuXkDNpk= 2024-03-04T17:39:13.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YwKv3vukGgQuVevA1lKPuXkDNpk=', 'Content-Length': '0'} 2024-03-04T17:39:13.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007419f221d6692329-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.421 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:13.421 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:13.421 DEBUG:boto:path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:13.421 DEBUG:boto:auth_path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:13.421 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.422 DEBUG:boto:Path: /ppfahn-11/.?versionId=null 2024-03-04T17:39:13.422 DEBUG:boto:Data: 2024-03-04T17:39:13.422 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.422 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.422 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.422 DEBUG:boto:Params: {} 2024-03-04T17:39:13.422 DEBUG:boto:Token: None 2024-03-04T17:39:13.422 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/.?versionId=null 2024-03-04T17:39:13.422 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:B1s4cQ9la+6RfObOrEWlxcDnD2g= 2024-03-04T17:39:13.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:B1s4cQ9la+6RfObOrEWlxcDnD2g=', 'Content-Length': '0'} 2024-03-04T17:39:13.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080b1f2af42a20322-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.425 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:13.425 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:13.426 DEBUG:boto:path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:13.426 DEBUG:boto:auth_path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:13.426 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.426 DEBUG:boto:Path: /ppfahn-11/.?versionId=null 2024-03-04T17:39:13.426 DEBUG:boto:Data: 2024-03-04T17:39:13.426 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.426 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.426 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.426 DEBUG:boto:Params: {} 2024-03-04T17:39:13.426 DEBUG:boto:Token: None 2024-03-04T17:39:13.427 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/.?versionId=null 2024-03-04T17:39:13.427 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:B1s4cQ9la+6RfObOrEWlxcDnD2g= 2024-03-04T17:39:13.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:B1s4cQ9la+6RfObOrEWlxcDnD2g=', 'Content-Length': '0'} 2024-03-04T17:39:13.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c56f222af09f3c6f-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:13.430 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:13.430 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:13.430 DEBUG:boto:path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:13.430 DEBUG:boto:auth_path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:13.430 DEBUG:boto:Method: GET 2024-03-04T17:39:13.430 DEBUG:boto:Path: /ppfahn-11/.?versionId=null 2024-03-04T17:39:13.430 DEBUG:boto:Data: 2024-03-04T17:39:13.431 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.431 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.431 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.431 DEBUG:boto:Params: {} 2024-03-04T17:39:13.431 DEBUG:boto:Token: None 2024-03-04T17:39:13.431 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/.?versionId=null 2024-03-04T17:39:13.431 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YwKv3vukGgQuVevA1lKPuXkDNpk= 2024-03-04T17:39:13.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YwKv3vukGgQuVevA1lKPuXkDNpk=', 'Content-Length': '0'} 2024-03-04T17:39:13.434 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3f7eef564f67fd9-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.476 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:13.476 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:13.476 DEBUG:boto:path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:13.476 DEBUG:boto:auth_path=/ppfahn-11/.?versionId=null 2024-03-04T17:39:13.477 DEBUG:boto:Method: GET 2024-03-04T17:39:13.477 DEBUG:boto:Path: /ppfahn-11/.?versionId=null 2024-03-04T17:39:13.477 DEBUG:boto:Data: 2024-03-04T17:39:13.477 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.477 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.477 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.477 DEBUG:boto:Params: {} 2024-03-04T17:39:13.477 DEBUG:boto:Token: None 2024-03-04T17:39:13.477 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/.?versionId=null 2024-03-04T17:39:13.478 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YwKv3vukGgQuVevA1lKPuXkDNpk= 2024-03-04T17:39:13.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YwKv3vukGgQuVevA1lKPuXkDNpk=', 'Content-Length': '0'} 2024-03-04T17:39:13.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000264b7db333259951-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.481 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.481 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.481 DEBUG:boto:path=/ppfahn-11/?prefix=. 2024-03-04T17:39:13.481 DEBUG:boto:auth_path=/ppfahn-11/?prefix=. 2024-03-04T17:39:13.481 DEBUG:boto:Method: GET 2024-03-04T17:39:13.481 DEBUG:boto:Path: /ppfahn-11/?prefix=. 2024-03-04T17:39:13.481 DEBUG:boto:Data: 2024-03-04T17:39:13.482 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.482 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.482 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.482 DEBUG:boto:Params: {} 2024-03-04T17:39:13.482 DEBUG:boto:Token: None 2024-03-04T17:39:13.482 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.482 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000807d4abd4178a0f8-0065e60741-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.487 DEBUG:boto:b'ppfahn-11.1000false.2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.487 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.487 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.487 DEBUG:boto:path=/ppfahn-11/?prefix=. 2024-03-04T17:39:13.488 DEBUG:boto:auth_path=/ppfahn-11/?prefix=. 2024-03-04T17:39:13.488 DEBUG:boto:Method: GET 2024-03-04T17:39:13.488 DEBUG:boto:Path: /ppfahn-11/?prefix=. 2024-03-04T17:39:13.488 DEBUG:boto:Data: 2024-03-04T17:39:13.488 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.488 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.488 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.488 DEBUG:boto:Params: {} 2024-03-04T17:39:13.488 DEBUG:boto:Token: None 2024-03-04T17:39:13.488 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.489 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb7eac4924ca8952-0065e60741-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.494 DEBUG:boto:b'ppfahn-11.1000false.2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.494 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:13.494 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:13.494 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.494 DEBUG:boto:Path: /ppfahn-11/. 2024-03-04T17:39:13.494 DEBUG:boto:Data: 2024-03-04T17:39:13.495 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.495 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.495 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.495 DEBUG:boto:Params: {} 2024-03-04T17:39:13.495 DEBUG:boto:Token: None 2024-03-04T17:39:13.495 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/. 2024-03-04T17:39:13.495 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oAMkUFUW3MNDfFTG87vDJlxrQPw= 2024-03-04T17:39:13.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oAMkUFUW3MNDfFTG87vDJlxrQPw=', 'Content-Length': '0'} 2024-03-04T17:39:13.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006276e5d74dfd2263-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.498 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:13.498 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:13.498 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.498 DEBUG:boto:Path: /ppfahn-11/. 2024-03-04T17:39:13.499 DEBUG:boto:Data: 2024-03-04T17:39:13.499 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.499 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.499 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.499 DEBUG:boto:Params: {} 2024-03-04T17:39:13.499 DEBUG:boto:Token: None 2024-03-04T17:39:13.499 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/. 2024-03-04T17:39:13.499 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oAMkUFUW3MNDfFTG87vDJlxrQPw= 2024-03-04T17:39:13.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oAMkUFUW3MNDfFTG87vDJlxrQPw=', 'Content-Length': '0'} 2024-03-04T17:39:13.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb5bddc14b0b3dde-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:13.502 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:13.502 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:13.502 DEBUG:boto:Method: GET 2024-03-04T17:39:13.502 DEBUG:boto:Path: /ppfahn-11/. 2024-03-04T17:39:13.503 DEBUG:boto:Data: 2024-03-04T17:39:13.503 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.503 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.503 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.503 DEBUG:boto:Params: {} 2024-03-04T17:39:13.503 DEBUG:boto:Token: None 2024-03-04T17:39:13.503 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/. 2024-03-04T17:39:13.503 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:axldWy4/gYlzdDbGCp+Ru09lyBo= 2024-03-04T17:39:13.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:axldWy4/gYlzdDbGCp+Ru09lyBo=', 'Content-Length': '0'} 2024-03-04T17:39:13.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb060f1e297fc758-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.548 DEBUG:boto:path=/ppfahn-11/. 2024-03-04T17:39:13.548 DEBUG:boto:auth_path=/ppfahn-11/. 2024-03-04T17:39:13.548 DEBUG:boto:Method: GET 2024-03-04T17:39:13.548 DEBUG:boto:Path: /ppfahn-11/. 2024-03-04T17:39:13.548 DEBUG:boto:Data: 2024-03-04T17:39:13.549 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.549 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.549 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.549 DEBUG:boto:Params: {} 2024-03-04T17:39:13.549 DEBUG:boto:Token: None 2024-03-04T17:39:13.549 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/. 2024-03-04T17:39:13.549 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:axldWy4/gYlzdDbGCp+Ru09lyBo= 2024-03-04T17:39:13.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:axldWy4/gYlzdDbGCp+Ru09lyBo=', 'Content-Length': '0'} 2024-03-04T17:39:13.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002655d88e82a22f4f-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.553 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.553 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.553 DEBUG:boto:path=/ppfahn-11/?prefix=. 2024-03-04T17:39:13.553 DEBUG:boto:auth_path=/ppfahn-11/?prefix=. 2024-03-04T17:39:13.553 DEBUG:boto:Method: GET 2024-03-04T17:39:13.553 DEBUG:boto:Path: /ppfahn-11/?prefix=. 2024-03-04T17:39:13.553 DEBUG:boto:Data: 2024-03-04T17:39:13.553 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.553 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.553 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.553 DEBUG:boto:Params: {} 2024-03-04T17:39:13.554 DEBUG:boto:Token: None 2024-03-04T17:39:13.554 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.554 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c041da3ac5b85a1a-0065e60741-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.558 DEBUG:boto:b'ppfahn-11.1000false.2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.559 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.559 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.559 DEBUG:boto:path=/ppfahn-11/?prefix=. 2024-03-04T17:39:13.559 DEBUG:boto:auth_path=/ppfahn-11/?prefix=. 2024-03-04T17:39:13.559 DEBUG:boto:Method: GET 2024-03-04T17:39:13.559 DEBUG:boto:Path: /ppfahn-11/?prefix=. 2024-03-04T17:39:13.559 DEBUG:boto:Data: 2024-03-04T17:39:13.559 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.559 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.560 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.560 DEBUG:boto:Params: {} 2024-03-04T17:39:13.560 DEBUG:boto:Token: None 2024-03-04T17:39:13.560 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.560 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067686e7f54bca005-0065e60741-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.565 DEBUG:boto:b'ppfahn-11.1000false.2024-03-04T17:36:30.503Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:13.565 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:13.565 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:13.565 DEBUG:boto:path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:13.565 DEBUG:boto:auth_path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:13.565 DEBUG:boto:Method: GET 2024-03-04T17:39:13.566 DEBUG:boto:Path: /ppfahn-11/..?versionId=null 2024-03-04T17:39:13.566 DEBUG:boto:Data: 2024-03-04T17:39:13.566 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.566 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.566 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.566 DEBUG:boto:Params: {} 2024-03-04T17:39:13.566 DEBUG:boto:Token: None 2024-03-04T17:39:13.566 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/..?versionId=null 2024-03-04T17:39:13.566 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NIo4pY3ukUXBTByHvex78Mq/R+o= 2024-03-04T17:39:13.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NIo4pY3ukUXBTByHvex78Mq/R+o=', 'Content-Length': '0'} 2024-03-04T17:39:13.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b8badd19054299d-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.569 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:13.570 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:13.570 DEBUG:boto:path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:13.570 DEBUG:boto:auth_path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:13.570 DEBUG:boto:Method: GET 2024-03-04T17:39:13.570 DEBUG:boto:Path: /ppfahn-11/..?versionId=null 2024-03-04T17:39:13.570 DEBUG:boto:Data: 2024-03-04T17:39:13.570 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.570 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.570 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.570 DEBUG:boto:Params: {} 2024-03-04T17:39:13.571 DEBUG:boto:Token: None 2024-03-04T17:39:13.571 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/..?versionId=null 2024-03-04T17:39:13.571 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NIo4pY3ukUXBTByHvex78Mq/R+o= 2024-03-04T17:39:13.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NIo4pY3ukUXBTByHvex78Mq/R+o=', 'Content-Length': '0'} 2024-03-04T17:39:13.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062b6f5262856fc7b-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.616 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:13.616 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:13.616 DEBUG:boto:path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:13.616 DEBUG:boto:auth_path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:13.616 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.617 DEBUG:boto:Path: /ppfahn-11/..?versionId=null 2024-03-04T17:39:13.617 DEBUG:boto:Data: 2024-03-04T17:39:13.617 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.617 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.617 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.617 DEBUG:boto:Params: {} 2024-03-04T17:39:13.617 DEBUG:boto:Token: None 2024-03-04T17:39:13.617 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/..?versionId=null 2024-03-04T17:39:13.617 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wniOPXnEmP0ToaAwHSA+agq8LyA= 2024-03-04T17:39:13.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wniOPXnEmP0ToaAwHSA+agq8LyA=', 'Content-Length': '0'} 2024-03-04T17:39:13.620 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002024fb4437469bd-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.620 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:13.621 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:13.621 DEBUG:boto:path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:13.621 DEBUG:boto:auth_path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:13.621 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.621 DEBUG:boto:Path: /ppfahn-11/..?versionId=null 2024-03-04T17:39:13.621 DEBUG:boto:Data: 2024-03-04T17:39:13.621 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.621 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.621 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.621 DEBUG:boto:Params: {} 2024-03-04T17:39:13.621 DEBUG:boto:Token: None 2024-03-04T17:39:13.622 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/..?versionId=null 2024-03-04T17:39:13.622 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wniOPXnEmP0ToaAwHSA+agq8LyA= 2024-03-04T17:39:13.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wniOPXnEmP0ToaAwHSA+agq8LyA=', 'Content-Length': '0'} 2024-03-04T17:39:13.624 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000155e1d175617c6dc-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:13.625 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:13.625 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:13.625 DEBUG:boto:path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:13.625 DEBUG:boto:auth_path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:13.625 DEBUG:boto:Method: GET 2024-03-04T17:39:13.625 DEBUG:boto:Path: /ppfahn-11/..?versionId=null 2024-03-04T17:39:13.625 DEBUG:boto:Data: 2024-03-04T17:39:13.626 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.626 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.626 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.626 DEBUG:boto:Params: {} 2024-03-04T17:39:13.626 DEBUG:boto:Token: None 2024-03-04T17:39:13.626 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/..?versionId=null 2024-03-04T17:39:13.626 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NIo4pY3ukUXBTByHvex78Mq/R+o= 2024-03-04T17:39:13.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NIo4pY3ukUXBTByHvex78Mq/R+o=', 'Content-Length': '0'} 2024-03-04T17:39:13.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a06abdca7dfbec1f-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.672 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:13.672 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:13.672 DEBUG:boto:path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:13.672 DEBUG:boto:auth_path=/ppfahn-11/..?versionId=null 2024-03-04T17:39:13.672 DEBUG:boto:Method: GET 2024-03-04T17:39:13.673 DEBUG:boto:Path: /ppfahn-11/..?versionId=null 2024-03-04T17:39:13.673 DEBUG:boto:Data: 2024-03-04T17:39:13.673 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.673 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.673 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.673 DEBUG:boto:Params: {} 2024-03-04T17:39:13.673 DEBUG:boto:Token: None 2024-03-04T17:39:13.673 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/..?versionId=null 2024-03-04T17:39:13.673 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NIo4pY3ukUXBTByHvex78Mq/R+o= 2024-03-04T17:39:13.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NIo4pY3ukUXBTByHvex78Mq/R+o=', 'Content-Length': '0'} 2024-03-04T17:39:13.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000abd72f08c9497af-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.677 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.677 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.677 DEBUG:boto:path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:13.677 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:13.677 DEBUG:boto:Method: GET 2024-03-04T17:39:13.677 DEBUG:boto:Path: /ppfahn-11/?prefix=.. 2024-03-04T17:39:13.677 DEBUG:boto:Data: 2024-03-04T17:39:13.677 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.677 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.677 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.677 DEBUG:boto:Params: {} 2024-03-04T17:39:13.678 DEBUG:boto:Token: None 2024-03-04T17:39:13.678 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.678 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a23bcdad7fa0b5c-0065e60741-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.682 DEBUG:boto:b'ppfahn-11..1000false..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.683 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.683 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.683 DEBUG:boto:path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:13.683 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:13.683 DEBUG:boto:Method: GET 2024-03-04T17:39:13.683 DEBUG:boto:Path: /ppfahn-11/?prefix=.. 2024-03-04T17:39:13.683 DEBUG:boto:Data: 2024-03-04T17:39:13.683 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.683 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.683 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.684 DEBUG:boto:Params: {} 2024-03-04T17:39:13.684 DEBUG:boto:Token: None 2024-03-04T17:39:13.684 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.684 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd2ea19c8c31e438-0065e60741-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.688 DEBUG:boto:b'ppfahn-11..1000false..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.689 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:13.689 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:13.689 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.689 DEBUG:boto:Path: /ppfahn-11/.. 2024-03-04T17:39:13.689 DEBUG:boto:Data: 2024-03-04T17:39:13.689 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.689 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.689 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.690 DEBUG:boto:Params: {} 2024-03-04T17:39:13.690 DEBUG:boto:Token: None 2024-03-04T17:39:13.690 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/.. 2024-03-04T17:39:13.690 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:en9c5oyWictOlD5JRm4renSGTuo= 2024-03-04T17:39:13.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:en9c5oyWictOlD5JRm4renSGTuo=', 'Content-Length': '0'} 2024-03-04T17:39:13.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eaa08ad97979322c-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.693 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:13.693 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:13.693 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.693 DEBUG:boto:Path: /ppfahn-11/.. 2024-03-04T17:39:13.694 DEBUG:boto:Data: 2024-03-04T17:39:13.694 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.694 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.694 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.694 DEBUG:boto:Params: {} 2024-03-04T17:39:13.694 DEBUG:boto:Token: None 2024-03-04T17:39:13.694 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/.. 2024-03-04T17:39:13.694 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:en9c5oyWictOlD5JRm4renSGTuo= 2024-03-04T17:39:13.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:en9c5oyWictOlD5JRm4renSGTuo=', 'Content-Length': '0'} 2024-03-04T17:39:13.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006acd285e5af7bc9c-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:13.697 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:13.697 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:13.698 DEBUG:boto:Method: GET 2024-03-04T17:39:13.698 DEBUG:boto:Path: /ppfahn-11/.. 2024-03-04T17:39:13.698 DEBUG:boto:Data: 2024-03-04T17:39:13.698 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.698 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.698 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.698 DEBUG:boto:Params: {} 2024-03-04T17:39:13.698 DEBUG:boto:Token: None 2024-03-04T17:39:13.698 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/.. 2024-03-04T17:39:13.698 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rAy2OpW1RBYOmPSzIgSptpoLXYc= 2024-03-04T17:39:13.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rAy2OpW1RBYOmPSzIgSptpoLXYc=', 'Content-Length': '0'} 2024-03-04T17:39:13.701 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000301e27ffd73044a7-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.744 DEBUG:boto:path=/ppfahn-11/.. 2024-03-04T17:39:13.744 DEBUG:boto:auth_path=/ppfahn-11/.. 2024-03-04T17:39:13.744 DEBUG:boto:Method: GET 2024-03-04T17:39:13.744 DEBUG:boto:Path: /ppfahn-11/.. 2024-03-04T17:39:13.744 DEBUG:boto:Data: 2024-03-04T17:39:13.745 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.745 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.745 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.745 DEBUG:boto:Params: {} 2024-03-04T17:39:13.745 DEBUG:boto:Token: None 2024-03-04T17:39:13.745 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/.. 2024-03-04T17:39:13.745 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rAy2OpW1RBYOmPSzIgSptpoLXYc= 2024-03-04T17:39:13.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rAy2OpW1RBYOmPSzIgSptpoLXYc=', 'Content-Length': '0'} 2024-03-04T17:39:13.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8d6fd13f1a92e91-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.748 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.748 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.749 DEBUG:boto:path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:13.749 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:13.749 DEBUG:boto:Method: GET 2024-03-04T17:39:13.749 DEBUG:boto:Path: /ppfahn-11/?prefix=.. 2024-03-04T17:39:13.749 DEBUG:boto:Data: 2024-03-04T17:39:13.749 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.749 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.749 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.749 DEBUG:boto:Params: {} 2024-03-04T17:39:13.749 DEBUG:boto:Token: None 2024-03-04T17:39:13.750 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.750 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e169a9dc3a7a66f1-0065e60741-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.755 DEBUG:boto:b'ppfahn-11..1000false..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.755 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.755 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.755 DEBUG:boto:path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:13.755 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.. 2024-03-04T17:39:13.755 DEBUG:boto:Method: GET 2024-03-04T17:39:13.755 DEBUG:boto:Path: /ppfahn-11/?prefix=.. 2024-03-04T17:39:13.755 DEBUG:boto:Data: 2024-03-04T17:39:13.755 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.756 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.756 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.756 DEBUG:boto:Params: {} 2024-03-04T17:39:13.756 DEBUG:boto:Token: None 2024-03-04T17:39:13.756 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.756 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dab188bb904ce982-0065e60741-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.760 DEBUG:boto:b'ppfahn-11..1000false..2024-03-04T17:36:30.529Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:13.761 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:13.761 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:13.761 DEBUG:boto:path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:13.761 DEBUG:boto:auth_path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:13.761 DEBUG:boto:Method: GET 2024-03-04T17:39:13.761 DEBUG:boto:Path: /ppfahn-11/...?versionId=null 2024-03-04T17:39:13.761 DEBUG:boto:Data: 2024-03-04T17:39:13.762 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.762 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.762 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.762 DEBUG:boto:Params: {} 2024-03-04T17:39:13.762 DEBUG:boto:Token: None 2024-03-04T17:39:13.762 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/...?versionId=null 2024-03-04T17:39:13.762 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CHUWFoAIAggqcaPiRxYdkXCl/ng= 2024-03-04T17:39:13.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CHUWFoAIAggqcaPiRxYdkXCl/ng=', 'Content-Length': '0'} 2024-03-04T17:39:13.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092b3e8ad78146cac-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.766 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:13.766 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:13.766 DEBUG:boto:path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:13.766 DEBUG:boto:auth_path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:13.766 DEBUG:boto:Method: GET 2024-03-04T17:39:13.766 DEBUG:boto:Path: /ppfahn-11/...?versionId=null 2024-03-04T17:39:13.766 DEBUG:boto:Data: 2024-03-04T17:39:13.766 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.766 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.766 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.766 DEBUG:boto:Params: {} 2024-03-04T17:39:13.767 DEBUG:boto:Token: None 2024-03-04T17:39:13.767 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/...?versionId=null 2024-03-04T17:39:13.767 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CHUWFoAIAggqcaPiRxYdkXCl/ng= 2024-03-04T17:39:13.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CHUWFoAIAggqcaPiRxYdkXCl/ng=', 'Content-Length': '0'} 2024-03-04T17:39:13.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd97d3a5ff34f63d-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.770 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:13.770 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:13.770 DEBUG:boto:path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:13.770 DEBUG:boto:auth_path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:13.770 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.770 DEBUG:boto:Path: /ppfahn-11/...?versionId=null 2024-03-04T17:39:13.771 DEBUG:boto:Data: 2024-03-04T17:39:13.771 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.771 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.771 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.771 DEBUG:boto:Params: {} 2024-03-04T17:39:13.771 DEBUG:boto:Token: None 2024-03-04T17:39:13.771 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/...?versionId=null 2024-03-04T17:39:13.771 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YuFgslxsLGqFyWrfMW2mIVqX21Q= 2024-03-04T17:39:13.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YuFgslxsLGqFyWrfMW2mIVqX21Q=', 'Content-Length': '0'} 2024-03-04T17:39:13.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000952cee309ce21013-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.774 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:13.774 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:13.774 DEBUG:boto:path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:13.775 DEBUG:boto:auth_path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:13.775 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.775 DEBUG:boto:Path: /ppfahn-11/...?versionId=null 2024-03-04T17:39:13.775 DEBUG:boto:Data: 2024-03-04T17:39:13.775 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.775 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.775 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.775 DEBUG:boto:Params: {} 2024-03-04T17:39:13.775 DEBUG:boto:Token: None 2024-03-04T17:39:13.775 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/...?versionId=null 2024-03-04T17:39:13.775 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YuFgslxsLGqFyWrfMW2mIVqX21Q= 2024-03-04T17:39:13.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YuFgslxsLGqFyWrfMW2mIVqX21Q=', 'Content-Length': '0'} 2024-03-04T17:39:13.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f05c662e43031da0-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:13.779 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:13.779 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:13.779 DEBUG:boto:path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:13.779 DEBUG:boto:auth_path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:13.779 DEBUG:boto:Method: GET 2024-03-04T17:39:13.779 DEBUG:boto:Path: /ppfahn-11/...?versionId=null 2024-03-04T17:39:13.779 DEBUG:boto:Data: 2024-03-04T17:39:13.779 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.779 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.779 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.779 DEBUG:boto:Params: {} 2024-03-04T17:39:13.780 DEBUG:boto:Token: None 2024-03-04T17:39:13.780 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/...?versionId=null 2024-03-04T17:39:13.780 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CHUWFoAIAggqcaPiRxYdkXCl/ng= 2024-03-04T17:39:13.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CHUWFoAIAggqcaPiRxYdkXCl/ng=', 'Content-Length': '0'} 2024-03-04T17:39:13.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001234420800e582a2-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.824 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:13.824 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:13.824 DEBUG:boto:path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:13.824 DEBUG:boto:auth_path=/ppfahn-11/...?versionId=null 2024-03-04T17:39:13.824 DEBUG:boto:Method: GET 2024-03-04T17:39:13.825 DEBUG:boto:Path: /ppfahn-11/...?versionId=null 2024-03-04T17:39:13.825 DEBUG:boto:Data: 2024-03-04T17:39:13.825 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.825 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.825 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.825 DEBUG:boto:Params: {} 2024-03-04T17:39:13.825 DEBUG:boto:Token: None 2024-03-04T17:39:13.825 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/...?versionId=null 2024-03-04T17:39:13.825 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CHUWFoAIAggqcaPiRxYdkXCl/ng= 2024-03-04T17:39:13.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CHUWFoAIAggqcaPiRxYdkXCl/ng=', 'Content-Length': '0'} 2024-03-04T17:39:13.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079a251e5b8d6237e-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.829 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.829 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.829 DEBUG:boto:path=/ppfahn-11/?prefix=... 2024-03-04T17:39:13.829 DEBUG:boto:auth_path=/ppfahn-11/?prefix=... 2024-03-04T17:39:13.829 DEBUG:boto:Method: GET 2024-03-04T17:39:13.829 DEBUG:boto:Path: /ppfahn-11/?prefix=... 2024-03-04T17:39:13.829 DEBUG:boto:Data: 2024-03-04T17:39:13.829 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.829 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.830 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.830 DEBUG:boto:Params: {} 2024-03-04T17:39:13.830 DEBUG:boto:Token: None 2024-03-04T17:39:13.830 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.830 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aad8dcd05bc47dd-0065e60741-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.834 DEBUG:boto:b'ppfahn-11...1000false...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.835 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.835 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.835 DEBUG:boto:path=/ppfahn-11/?prefix=... 2024-03-04T17:39:13.835 DEBUG:boto:auth_path=/ppfahn-11/?prefix=... 2024-03-04T17:39:13.835 DEBUG:boto:Method: GET 2024-03-04T17:39:13.835 DEBUG:boto:Path: /ppfahn-11/?prefix=... 2024-03-04T17:39:13.835 DEBUG:boto:Data: 2024-03-04T17:39:13.835 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.835 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.836 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.836 DEBUG:boto:Params: {} 2024-03-04T17:39:13.836 DEBUG:boto:Token: None 2024-03-04T17:39:13.836 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.836 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdf28067ffd22cae-0065e60741-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.840 DEBUG:boto:b'ppfahn-11...1000false...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.840 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:13.841 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:13.841 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.841 DEBUG:boto:Path: /ppfahn-11/... 2024-03-04T17:39:13.841 DEBUG:boto:Data: 2024-03-04T17:39:13.841 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.841 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.841 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.841 DEBUG:boto:Params: {} 2024-03-04T17:39:13.841 DEBUG:boto:Token: None 2024-03-04T17:39:13.841 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/... 2024-03-04T17:39:13.841 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WhI533LoPmZ3LqnatcPACXgZ53I= 2024-03-04T17:39:13.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WhI533LoPmZ3LqnatcPACXgZ53I=', 'Content-Length': '0'} 2024-03-04T17:39:13.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c839b06465f3c86b-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.844 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:13.845 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:13.845 DEBUG:boto:Method: HEAD 2024-03-04T17:39:13.845 DEBUG:boto:Path: /ppfahn-11/... 2024-03-04T17:39:13.845 DEBUG:boto:Data: 2024-03-04T17:39:13.845 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.845 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.845 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.845 DEBUG:boto:Params: {} 2024-03-04T17:39:13.845 DEBUG:boto:Token: None 2024-03-04T17:39:13.845 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/... 2024-03-04T17:39:13.846 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WhI533LoPmZ3LqnatcPACXgZ53I= 2024-03-04T17:39:13.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WhI533LoPmZ3LqnatcPACXgZ53I=', 'Content-Length': '0'} 2024-03-04T17:39:13.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f50aed436c95ed7f-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:13.849 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:13.849 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:13.849 DEBUG:boto:Method: GET 2024-03-04T17:39:13.849 DEBUG:boto:Path: /ppfahn-11/... 2024-03-04T17:39:13.849 DEBUG:boto:Data: 2024-03-04T17:39:13.849 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.849 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.849 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.849 DEBUG:boto:Params: {} 2024-03-04T17:39:13.849 DEBUG:boto:Token: None 2024-03-04T17:39:13.850 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/... 2024-03-04T17:39:13.850 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KVsEWLbHgl4Cdvfskl3f/9yETs8= 2024-03-04T17:39:13.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KVsEWLbHgl4Cdvfskl3f/9yETs8=', 'Content-Length': '0'} 2024-03-04T17:39:13.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b48c779c11d22d7a-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.896 DEBUG:boto:path=/ppfahn-11/... 2024-03-04T17:39:13.896 DEBUG:boto:auth_path=/ppfahn-11/... 2024-03-04T17:39:13.896 DEBUG:boto:Method: GET 2024-03-04T17:39:13.897 DEBUG:boto:Path: /ppfahn-11/... 2024-03-04T17:39:13.897 DEBUG:boto:Data: 2024-03-04T17:39:13.897 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.897 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.897 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.897 DEBUG:boto:Params: {} 2024-03-04T17:39:13.897 DEBUG:boto:Token: None 2024-03-04T17:39:13.897 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/... 2024-03-04T17:39:13.898 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KVsEWLbHgl4Cdvfskl3f/9yETs8= 2024-03-04T17:39:13.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KVsEWLbHgl4Cdvfskl3f/9yETs8=', 'Content-Length': '0'} 2024-03-04T17:39:13.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000166cde48b137f4c1-0065e60741-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.901 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.901 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.902 DEBUG:boto:path=/ppfahn-11/?prefix=... 2024-03-04T17:39:13.902 DEBUG:boto:auth_path=/ppfahn-11/?prefix=... 2024-03-04T17:39:13.902 DEBUG:boto:Method: GET 2024-03-04T17:39:13.902 DEBUG:boto:Path: /ppfahn-11/?prefix=... 2024-03-04T17:39:13.902 DEBUG:boto:Data: 2024-03-04T17:39:13.902 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.902 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.902 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.903 DEBUG:boto:Params: {} 2024-03-04T17:39:13.903 DEBUG:boto:Token: None 2024-03-04T17:39:13.903 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.903 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd6a291333b7fbae-0065e60741-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.908 DEBUG:boto:b'ppfahn-11...1000false...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.908 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:13.908 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:13.908 DEBUG:boto:path=/ppfahn-11/?prefix=... 2024-03-04T17:39:13.908 DEBUG:boto:auth_path=/ppfahn-11/?prefix=... 2024-03-04T17:39:13.909 DEBUG:boto:Method: GET 2024-03-04T17:39:13.909 DEBUG:boto:Path: /ppfahn-11/?prefix=... 2024-03-04T17:39:13.909 DEBUG:boto:Data: 2024-03-04T17:39:13.909 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.909 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:13.909 DEBUG:boto:Port: 8000 2024-03-04T17:39:13.909 DEBUG:boto:Params: {} 2024-03-04T17:39:13.910 DEBUG:boto:Token: None 2024-03-04T17:39:13.910 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/ 2024-03-04T17:39:13.910 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c= 2024-03-04T17:39:13.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RujBVUuhUzg9Q7iZ3oPqCJ39b/c=', 'Content-Length': '0'} 2024-03-04T17:39:13.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b66e59dbb3e455e9-0065e60741-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:13.956 DEBUG:boto:b'ppfahn-11...1000false...2024-03-04T17:36:30.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:13.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:13.957 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:13.957 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:13.957 DEBUG:boto:path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:13.957 DEBUG:boto:auth_path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:13.957 DEBUG:boto:Method: GET 2024-03-04T17:39:13.957 DEBUG:boto:Path: /ppfahn-11/.o?versionId=null 2024-03-04T17:39:13.957 DEBUG:boto:Data: 2024-03-04T17:39:13.957 DEBUG:boto:Headers: {} 2024-03-04T17:39:13.957 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:13.957 DEBUG:boto:Port: 8001 2024-03-04T17:39:13.957 DEBUG:boto:Params: {} 2024-03-04T17:39:13.958 DEBUG:boto:Token: None 2024-03-04T17:39:13.958 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:13 GMT /ppfahn-11/.o?versionId=null 2024-03-04T17:39:13.958 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:05r1VqbXAplvkxKIQdHaeeGG0GE= 2024-03-04T17:39:13.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:05r1VqbXAplvkxKIQdHaeeGG0GE=', 'Content-Length': '0'} 2024-03-04T17:39:13.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb434d6bf21668ac-0065e60741-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.004 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:14.004 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:14.004 DEBUG:boto:path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.004 DEBUG:boto:auth_path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.005 DEBUG:boto:Method: GET 2024-03-04T17:39:14.005 DEBUG:boto:Path: /ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.005 DEBUG:boto:Data: 2024-03-04T17:39:14.005 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.005 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.005 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.005 DEBUG:boto:Params: {} 2024-03-04T17:39:14.005 DEBUG:boto:Token: None 2024-03-04T17:39:14.005 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.005 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E83s3P3iZMMkQvJwqWCdJ1sws7g= 2024-03-04T17:39:14.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E83s3P3iZMMkQvJwqWCdJ1sws7g=', 'Content-Length': '0'} 2024-03-04T17:39:14.008 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011bfe20a52230a64-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.008 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:14.009 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:14.009 DEBUG:boto:path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.009 DEBUG:boto:auth_path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.009 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.009 DEBUG:boto:Path: /ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.009 DEBUG:boto:Data: 2024-03-04T17:39:14.009 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.009 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.009 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.009 DEBUG:boto:Params: {} 2024-03-04T17:39:14.010 DEBUG:boto:Token: None 2024-03-04T17:39:14.010 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.010 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Mi6LZf9zJwhbSt3bw0cCk0Z2Mm0= 2024-03-04T17:39:14.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Mi6LZf9zJwhbSt3bw0cCk0Z2Mm0=', 'Content-Length': '0'} 2024-03-04T17:39:14.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005b7d908bb345d78-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.013 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:14.014 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:14.014 DEBUG:boto:path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.014 DEBUG:boto:auth_path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.014 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.014 DEBUG:boto:Path: /ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.014 DEBUG:boto:Data: 2024-03-04T17:39:14.014 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.014 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.014 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.014 DEBUG:boto:Params: {} 2024-03-04T17:39:14.014 DEBUG:boto:Token: None 2024-03-04T17:39:14.015 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.015 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Mi6LZf9zJwhbSt3bw0cCk0Z2Mm0= 2024-03-04T17:39:14.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Mi6LZf9zJwhbSt3bw0cCk0Z2Mm0=', 'Content-Length': '0'} 2024-03-04T17:39:14.017 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c089b16e1dbf4d4-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:14.018 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:14.018 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:14.018 DEBUG:boto:path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.018 DEBUG:boto:auth_path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.018 DEBUG:boto:Method: GET 2024-03-04T17:39:14.018 DEBUG:boto:Path: /ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.018 DEBUG:boto:Data: 2024-03-04T17:39:14.018 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.019 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.019 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.019 DEBUG:boto:Params: {} 2024-03-04T17:39:14.019 DEBUG:boto:Token: None 2024-03-04T17:39:14.019 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.019 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E83s3P3iZMMkQvJwqWCdJ1sws7g= 2024-03-04T17:39:14.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E83s3P3iZMMkQvJwqWCdJ1sws7g=', 'Content-Length': '0'} 2024-03-04T17:39:14.022 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd5ca350645cda51-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.064 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:14.064 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:14.064 DEBUG:boto:path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.064 DEBUG:boto:auth_path=/ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.064 DEBUG:boto:Method: GET 2024-03-04T17:39:14.065 DEBUG:boto:Path: /ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.065 DEBUG:boto:Data: 2024-03-04T17:39:14.065 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.065 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.065 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.065 DEBUG:boto:Params: {} 2024-03-04T17:39:14.065 DEBUG:boto:Token: None 2024-03-04T17:39:14.065 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o?versionId=null 2024-03-04T17:39:14.065 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E83s3P3iZMMkQvJwqWCdJ1sws7g= 2024-03-04T17:39:14.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E83s3P3iZMMkQvJwqWCdJ1sws7g=', 'Content-Length': '0'} 2024-03-04T17:39:14.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9e4d2d7d40d4ce6-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.068 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.068 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.069 DEBUG:boto:path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:14.069 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:14.069 DEBUG:boto:Method: GET 2024-03-04T17:39:14.069 DEBUG:boto:Path: /ppfahn-11/?prefix=.o 2024-03-04T17:39:14.069 DEBUG:boto:Data: 2024-03-04T17:39:14.069 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.069 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.069 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.069 DEBUG:boto:Params: {} 2024-03-04T17:39:14.069 DEBUG:boto:Token: None 2024-03-04T17:39:14.070 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.070 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ea04fe6833abf40-0065e60742-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.074 DEBUG:boto:b'ppfahn-11.o1000false.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.075 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.075 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.075 DEBUG:boto:path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:14.075 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:14.075 DEBUG:boto:Method: GET 2024-03-04T17:39:14.075 DEBUG:boto:Path: /ppfahn-11/?prefix=.o 2024-03-04T17:39:14.075 DEBUG:boto:Data: 2024-03-04T17:39:14.075 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.075 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.075 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.075 DEBUG:boto:Params: {} 2024-03-04T17:39:14.076 DEBUG:boto:Token: None 2024-03-04T17:39:14.076 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.076 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dfd7a5760e29164-0065e60742-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.080 DEBUG:boto:b'ppfahn-11.o1000false.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.081 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:14.081 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:14.081 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.081 DEBUG:boto:Path: /ppfahn-11/.o 2024-03-04T17:39:14.081 DEBUG:boto:Data: 2024-03-04T17:39:14.081 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.081 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.081 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.081 DEBUG:boto:Params: {} 2024-03-04T17:39:14.081 DEBUG:boto:Token: None 2024-03-04T17:39:14.082 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o 2024-03-04T17:39:14.082 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://Zm5EwVO1D+qv6r/ldcvkZl6mY= 2024-03-04T17:39:14.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://Zm5EwVO1D+qv6r/ldcvkZl6mY=', 'Content-Length': '0'} 2024-03-04T17:39:14.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5e5e7c6da12ee5b-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.084 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:14.085 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:14.085 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.085 DEBUG:boto:Path: /ppfahn-11/.o 2024-03-04T17:39:14.085 DEBUG:boto:Data: 2024-03-04T17:39:14.085 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.085 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.085 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.085 DEBUG:boto:Params: {} 2024-03-04T17:39:14.085 DEBUG:boto:Token: None 2024-03-04T17:39:14.085 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o 2024-03-04T17:39:14.085 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://Zm5EwVO1D+qv6r/ldcvkZl6mY= 2024-03-04T17:39:14.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://Zm5EwVO1D+qv6r/ldcvkZl6mY=', 'Content-Length': '0'} 2024-03-04T17:39:14.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012a239ba666f742c-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:14.088 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:14.088 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:14.089 DEBUG:boto:Method: GET 2024-03-04T17:39:14.089 DEBUG:boto:Path: /ppfahn-11/.o 2024-03-04T17:39:14.089 DEBUG:boto:Data: 2024-03-04T17:39:14.089 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.089 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.089 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.089 DEBUG:boto:Params: {} 2024-03-04T17:39:14.089 DEBUG:boto:Token: None 2024-03-04T17:39:14.089 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o 2024-03-04T17:39:14.089 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ioxbPGFUpM+ZG+0eLs5SruiMNe4= 2024-03-04T17:39:14.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ioxbPGFUpM+ZG+0eLs5SruiMNe4=', 'Content-Length': '0'} 2024-03-04T17:39:14.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4ecb38e2c513d96-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.136 DEBUG:boto:path=/ppfahn-11/.o 2024-03-04T17:39:14.136 DEBUG:boto:auth_path=/ppfahn-11/.o 2024-03-04T17:39:14.136 DEBUG:boto:Method: GET 2024-03-04T17:39:14.136 DEBUG:boto:Path: /ppfahn-11/.o 2024-03-04T17:39:14.136 DEBUG:boto:Data: 2024-03-04T17:39:14.136 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.137 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.137 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.137 DEBUG:boto:Params: {} 2024-03-04T17:39:14.137 DEBUG:boto:Token: None 2024-03-04T17:39:14.137 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o 2024-03-04T17:39:14.137 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ioxbPGFUpM+ZG+0eLs5SruiMNe4= 2024-03-04T17:39:14.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ioxbPGFUpM+ZG+0eLs5SruiMNe4=', 'Content-Length': '0'} 2024-03-04T17:39:14.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000376ff0224e3dcda2-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.141 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.141 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.141 DEBUG:boto:path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:14.141 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:14.141 DEBUG:boto:Method: GET 2024-03-04T17:39:14.141 DEBUG:boto:Path: /ppfahn-11/?prefix=.o 2024-03-04T17:39:14.141 DEBUG:boto:Data: 2024-03-04T17:39:14.141 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.141 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.141 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.142 DEBUG:boto:Params: {} 2024-03-04T17:39:14.142 DEBUG:boto:Token: None 2024-03-04T17:39:14.142 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.142 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cff7eda7fc5d03c-0065e60742-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.146 DEBUG:boto:b'ppfahn-11.o1000false.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.147 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.147 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.147 DEBUG:boto:path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:14.147 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o 2024-03-04T17:39:14.147 DEBUG:boto:Method: GET 2024-03-04T17:39:14.147 DEBUG:boto:Path: /ppfahn-11/?prefix=.o 2024-03-04T17:39:14.147 DEBUG:boto:Data: 2024-03-04T17:39:14.147 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.147 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.147 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.147 DEBUG:boto:Params: {} 2024-03-04T17:39:14.148 DEBUG:boto:Token: None 2024-03-04T17:39:14.148 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.148 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000890609e532d16349-0065e60742-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.153 DEBUG:boto:b'ppfahn-11.o1000false.o2024-03-04T17:36:30.599Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:14.153 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:14.153 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:14.153 DEBUG:boto:path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.153 DEBUG:boto:auth_path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.153 DEBUG:boto:Method: GET 2024-03-04T17:39:14.153 DEBUG:boto:Path: /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.153 DEBUG:boto:Data: 2024-03-04T17:39:14.154 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.154 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.154 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.154 DEBUG:boto:Params: {} 2024-03-04T17:39:14.154 DEBUG:boto:Token: None 2024-03-04T17:39:14.154 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.154 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4VntDCNeqjpTTRuY3zt6gbbVft8= 2024-03-04T17:39:14.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4VntDCNeqjpTTRuY3zt6gbbVft8=', 'Content-Length': '0'} 2024-03-04T17:39:14.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d40e052278b6f01-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.157 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:14.157 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:14.157 DEBUG:boto:path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.157 DEBUG:boto:auth_path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.158 DEBUG:boto:Method: GET 2024-03-04T17:39:14.158 DEBUG:boto:Path: /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.158 DEBUG:boto:Data: 2024-03-04T17:39:14.158 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.158 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.158 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.158 DEBUG:boto:Params: {} 2024-03-04T17:39:14.158 DEBUG:boto:Token: None 2024-03-04T17:39:14.158 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.158 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4VntDCNeqjpTTRuY3zt6gbbVft8= 2024-03-04T17:39:14.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4VntDCNeqjpTTRuY3zt6gbbVft8=', 'Content-Length': '0'} 2024-03-04T17:39:14.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eec0f7a6c28590a9-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.162 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:14.162 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:14.162 DEBUG:boto:path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.162 DEBUG:boto:auth_path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.162 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.162 DEBUG:boto:Path: /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.162 DEBUG:boto:Data: 2024-03-04T17:39:14.162 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.162 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.162 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.162 DEBUG:boto:Params: {} 2024-03-04T17:39:14.163 DEBUG:boto:Token: None 2024-03-04T17:39:14.163 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.163 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jnp7a3B2dIGnJr1JNVCrEIYV1R8= 2024-03-04T17:39:14.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jnp7a3B2dIGnJr1JNVCrEIYV1R8=', 'Content-Length': '0'} 2024-03-04T17:39:14.165 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2a89edaae0b233a-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.166 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:14.166 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:14.166 DEBUG:boto:path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.166 DEBUG:boto:auth_path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.166 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.166 DEBUG:boto:Path: /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.166 DEBUG:boto:Data: 2024-03-04T17:39:14.166 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.166 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.167 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.167 DEBUG:boto:Params: {} 2024-03-04T17:39:14.167 DEBUG:boto:Token: None 2024-03-04T17:39:14.167 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.167 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jnp7a3B2dIGnJr1JNVCrEIYV1R8= 2024-03-04T17:39:14.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jnp7a3B2dIGnJr1JNVCrEIYV1R8=', 'Content-Length': '0'} 2024-03-04T17:39:14.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f6a057e35776f11-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:14.171 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:14.171 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:14.171 DEBUG:boto:path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.171 DEBUG:boto:auth_path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.171 DEBUG:boto:Method: GET 2024-03-04T17:39:14.171 DEBUG:boto:Path: /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.171 DEBUG:boto:Data: 2024-03-04T17:39:14.171 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.172 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.172 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.172 DEBUG:boto:Params: {} 2024-03-04T17:39:14.172 DEBUG:boto:Token: None 2024-03-04T17:39:14.172 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.172 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4VntDCNeqjpTTRuY3zt6gbbVft8= 2024-03-04T17:39:14.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4VntDCNeqjpTTRuY3zt6gbbVft8=', 'Content-Length': '0'} 2024-03-04T17:39:14.175 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009066e05262896f02-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.216 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:14.216 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:14.216 DEBUG:boto:path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.216 DEBUG:boto:auth_path=/ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.217 DEBUG:boto:Method: GET 2024-03-04T17:39:14.217 DEBUG:boto:Path: /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.217 DEBUG:boto:Data: 2024-03-04T17:39:14.217 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.217 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.217 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.217 DEBUG:boto:Params: {} 2024-03-04T17:39:14.217 DEBUG:boto:Token: None 2024-03-04T17:39:14.217 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o.?versionId=null 2024-03-04T17:39:14.217 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4VntDCNeqjpTTRuY3zt6gbbVft8= 2024-03-04T17:39:14.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4VntDCNeqjpTTRuY3zt6gbbVft8=', 'Content-Length': '0'} 2024-03-04T17:39:14.220 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001665976275d58298-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.221 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.221 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.221 DEBUG:boto:path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:14.221 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:14.221 DEBUG:boto:Method: GET 2024-03-04T17:39:14.221 DEBUG:boto:Path: /ppfahn-11/?prefix=.o. 2024-03-04T17:39:14.221 DEBUG:boto:Data: 2024-03-04T17:39:14.221 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.221 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.221 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.221 DEBUG:boto:Params: {} 2024-03-04T17:39:14.222 DEBUG:boto:Token: None 2024-03-04T17:39:14.222 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.222 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002552cfceaa4766c5-0065e60742-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.227 DEBUG:boto:b'ppfahn-11.o.1000false.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.227 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.227 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.227 DEBUG:boto:path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:14.227 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:14.227 DEBUG:boto:Method: GET 2024-03-04T17:39:14.227 DEBUG:boto:Path: /ppfahn-11/?prefix=.o. 2024-03-04T17:39:14.227 DEBUG:boto:Data: 2024-03-04T17:39:14.228 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.228 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.228 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.228 DEBUG:boto:Params: {} 2024-03-04T17:39:14.228 DEBUG:boto:Token: None 2024-03-04T17:39:14.228 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.228 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006907cb6fd95c90b4-0065e60742-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.233 DEBUG:boto:b'ppfahn-11.o.1000false.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.234 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:14.234 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:14.234 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.234 DEBUG:boto:Path: /ppfahn-11/.o. 2024-03-04T17:39:14.234 DEBUG:boto:Data: 2024-03-04T17:39:14.234 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.234 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.234 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.234 DEBUG:boto:Params: {} 2024-03-04T17:39:14.234 DEBUG:boto:Token: None 2024-03-04T17:39:14.234 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o. 2024-03-04T17:39:14.235 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fUuXeU4VW/ZYKweXHQ6Lfd8CfuU= 2024-03-04T17:39:14.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fUuXeU4VW/ZYKweXHQ6Lfd8CfuU=', 'Content-Length': '0'} 2024-03-04T17:39:14.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000556f98ed4d851b62-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.238 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:14.238 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:14.238 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.238 DEBUG:boto:Path: /ppfahn-11/.o. 2024-03-04T17:39:14.238 DEBUG:boto:Data: 2024-03-04T17:39:14.238 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.238 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.238 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.238 DEBUG:boto:Params: {} 2024-03-04T17:39:14.238 DEBUG:boto:Token: None 2024-03-04T17:39:14.239 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o. 2024-03-04T17:39:14.239 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fUuXeU4VW/ZYKweXHQ6Lfd8CfuU= 2024-03-04T17:39:14.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fUuXeU4VW/ZYKweXHQ6Lfd8CfuU=', 'Content-Length': '0'} 2024-03-04T17:39:14.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c2a979e8fe69fea-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:14.242 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:14.242 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:14.242 DEBUG:boto:Method: GET 2024-03-04T17:39:14.242 DEBUG:boto:Path: /ppfahn-11/.o. 2024-03-04T17:39:14.242 DEBUG:boto:Data: 2024-03-04T17:39:14.242 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.242 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.243 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.243 DEBUG:boto:Params: {} 2024-03-04T17:39:14.243 DEBUG:boto:Token: None 2024-03-04T17:39:14.243 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o. 2024-03-04T17:39:14.243 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zBQaL2o4qeKSqsPLnEWld/4SmDw= 2024-03-04T17:39:14.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zBQaL2o4qeKSqsPLnEWld/4SmDw=', 'Content-Length': '0'} 2024-03-04T17:39:14.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ada52411b33b90a-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.288 DEBUG:boto:path=/ppfahn-11/.o. 2024-03-04T17:39:14.288 DEBUG:boto:auth_path=/ppfahn-11/.o. 2024-03-04T17:39:14.288 DEBUG:boto:Method: GET 2024-03-04T17:39:14.288 DEBUG:boto:Path: /ppfahn-11/.o. 2024-03-04T17:39:14.289 DEBUG:boto:Data: 2024-03-04T17:39:14.289 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.289 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.289 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.289 DEBUG:boto:Params: {} 2024-03-04T17:39:14.289 DEBUG:boto:Token: None 2024-03-04T17:39:14.289 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/.o. 2024-03-04T17:39:14.289 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zBQaL2o4qeKSqsPLnEWld/4SmDw= 2024-03-04T17:39:14.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zBQaL2o4qeKSqsPLnEWld/4SmDw=', 'Content-Length': '0'} 2024-03-04T17:39:14.292 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad147c8303a5c597-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.292 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.292 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.292 DEBUG:boto:path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:14.292 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:14.293 DEBUG:boto:Method: GET 2024-03-04T17:39:14.293 DEBUG:boto:Path: /ppfahn-11/?prefix=.o. 2024-03-04T17:39:14.293 DEBUG:boto:Data: 2024-03-04T17:39:14.293 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.293 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.293 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.293 DEBUG:boto:Params: {} 2024-03-04T17:39:14.293 DEBUG:boto:Token: None 2024-03-04T17:39:14.293 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.293 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef8063a8ee2011a7-0065e60742-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.299 DEBUG:boto:b'ppfahn-11.o.1000false.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.299 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.299 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.300 DEBUG:boto:path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:14.300 DEBUG:boto:auth_path=/ppfahn-11/?prefix=.o. 2024-03-04T17:39:14.300 DEBUG:boto:Method: GET 2024-03-04T17:39:14.300 DEBUG:boto:Path: /ppfahn-11/?prefix=.o. 2024-03-04T17:39:14.300 DEBUG:boto:Data: 2024-03-04T17:39:14.300 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.300 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.300 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.300 DEBUG:boto:Params: {} 2024-03-04T17:39:14.300 DEBUG:boto:Token: None 2024-03-04T17:39:14.300 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.301 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3783956861b2f7e-0065e60742-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.348 DEBUG:boto:b'ppfahn-11.o.1000false.o.2024-03-04T17:36:30.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:14.348 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:14.349 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:14.349 DEBUG:boto:path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.349 DEBUG:boto:auth_path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.349 DEBUG:boto:Method: GET 2024-03-04T17:39:14.349 DEBUG:boto:Path: /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.349 DEBUG:boto:Data: 2024-03-04T17:39:14.349 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.349 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.349 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.349 DEBUG:boto:Params: {} 2024-03-04T17:39:14.349 DEBUG:boto:Token: None 2024-03-04T17:39:14.350 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.350 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:X4NrzAe5hu5yogVAb6QdcnATc10= 2024-03-04T17:39:14.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:X4NrzAe5hu5yogVAb6QdcnATc10=', 'Content-Length': '0'} 2024-03-04T17:39:14.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5bc71c4d6b3277d-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.396 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:14.396 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:14.396 DEBUG:boto:path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.396 DEBUG:boto:auth_path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.396 DEBUG:boto:Method: GET 2024-03-04T17:39:14.397 DEBUG:boto:Path: /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.397 DEBUG:boto:Data: 2024-03-04T17:39:14.397 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.397 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.397 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.397 DEBUG:boto:Params: {} 2024-03-04T17:39:14.397 DEBUG:boto:Token: None 2024-03-04T17:39:14.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.397 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:X4NrzAe5hu5yogVAb6QdcnATc10= 2024-03-04T17:39:14.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:X4NrzAe5hu5yogVAb6QdcnATc10=', 'Content-Length': '0'} 2024-03-04T17:39:14.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048e0d54f1e26672f-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.401 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:14.401 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:14.401 DEBUG:boto:path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.401 DEBUG:boto:auth_path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.401 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.401 DEBUG:boto:Path: /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.401 DEBUG:boto:Data: 2024-03-04T17:39:14.402 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.402 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.402 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.402 DEBUG:boto:Params: {} 2024-03-04T17:39:14.402 DEBUG:boto:Token: None 2024-03-04T17:39:14.402 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.402 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HHgmkhZGSlooDaj+Z6jIcegM2o4= 2024-03-04T17:39:14.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HHgmkhZGSlooDaj+Z6jIcegM2o4=', 'Content-Length': '0'} 2024-03-04T17:39:14.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c16107ac0cfc9290-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.405 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:14.405 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:14.405 DEBUG:boto:path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.406 DEBUG:boto:auth_path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.406 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.406 DEBUG:boto:Path: /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.406 DEBUG:boto:Data: 2024-03-04T17:39:14.406 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.406 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.406 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.406 DEBUG:boto:Params: {} 2024-03-04T17:39:14.406 DEBUG:boto:Token: None 2024-03-04T17:39:14.406 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.407 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HHgmkhZGSlooDaj+Z6jIcegM2o4= 2024-03-04T17:39:14.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HHgmkhZGSlooDaj+Z6jIcegM2o4=', 'Content-Length': '0'} 2024-03-04T17:39:14.409 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b42f56444edca93-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:14.410 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:14.410 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:14.410 DEBUG:boto:path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.410 DEBUG:boto:auth_path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.410 DEBUG:boto:Method: GET 2024-03-04T17:39:14.410 DEBUG:boto:Path: /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.410 DEBUG:boto:Data: 2024-03-04T17:39:14.410 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.410 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.411 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.411 DEBUG:boto:Params: {} 2024-03-04T17:39:14.411 DEBUG:boto:Token: None 2024-03-04T17:39:14.411 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.411 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:X4NrzAe5hu5yogVAb6QdcnATc10= 2024-03-04T17:39:14.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:X4NrzAe5hu5yogVAb6QdcnATc10=', 'Content-Length': '0'} 2024-03-04T17:39:14.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bcbfd3c6b8d8726-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.456 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:14.456 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:14.456 DEBUG:boto:path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.456 DEBUG:boto:auth_path=/ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.457 DEBUG:boto:Method: GET 2024-03-04T17:39:14.457 DEBUG:boto:Path: /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.457 DEBUG:boto:Data: 2024-03-04T17:39:14.457 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.457 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.457 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.457 DEBUG:boto:Params: {} 2024-03-04T17:39:14.457 DEBUG:boto:Token: None 2024-03-04T17:39:14.457 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/%3A?versionId=null 2024-03-04T17:39:14.458 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:X4NrzAe5hu5yogVAb6QdcnATc10= 2024-03-04T17:39:14.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:X4NrzAe5hu5yogVAb6QdcnATc10=', 'Content-Length': '0'} 2024-03-04T17:39:14.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e54ab4f78236b35f-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.461 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.461 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.461 DEBUG:boto:path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:14.461 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:14.461 DEBUG:boto:Method: GET 2024-03-04T17:39:14.462 DEBUG:boto:Path: /ppfahn-11/?prefix=%3A 2024-03-04T17:39:14.462 DEBUG:boto:Data: 2024-03-04T17:39:14.462 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.462 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.462 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.462 DEBUG:boto:Params: {} 2024-03-04T17:39:14.462 DEBUG:boto:Token: None 2024-03-04T17:39:14.462 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.463 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bba47478e52663a3-0065e60742-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.468 DEBUG:boto:b'ppfahn-11:1000false:2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.468 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.468 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.469 DEBUG:boto:path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:14.469 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:14.469 DEBUG:boto:Method: GET 2024-03-04T17:39:14.469 DEBUG:boto:Path: /ppfahn-11/?prefix=%3A 2024-03-04T17:39:14.469 DEBUG:boto:Data: 2024-03-04T17:39:14.469 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.469 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.469 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.469 DEBUG:boto:Params: {} 2024-03-04T17:39:14.470 DEBUG:boto:Token: None 2024-03-04T17:39:14.470 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.470 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007de3eaf4851ed18c-0065e60742-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.475 DEBUG:boto:b'ppfahn-11:1000false:2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.475 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:14.475 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:14.475 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.475 DEBUG:boto:Path: /ppfahn-11/%3A 2024-03-04T17:39:14.475 DEBUG:boto:Data: 2024-03-04T17:39:14.476 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.476 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.476 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.476 DEBUG:boto:Params: {} 2024-03-04T17:39:14.476 DEBUG:boto:Token: None 2024-03-04T17:39:14.476 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/%3A 2024-03-04T17:39:14.476 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:26kNA331H6cZgglEIUjhuNnTFIo= 2024-03-04T17:39:14.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:26kNA331H6cZgglEIUjhuNnTFIo=', 'Content-Length': '0'} 2024-03-04T17:39:14.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8cdb6916fe6a96e-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.479 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:14.479 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:14.479 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.480 DEBUG:boto:Path: /ppfahn-11/%3A 2024-03-04T17:39:14.480 DEBUG:boto:Data: 2024-03-04T17:39:14.480 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.480 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.480 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.480 DEBUG:boto:Params: {} 2024-03-04T17:39:14.480 DEBUG:boto:Token: None 2024-03-04T17:39:14.480 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/%3A 2024-03-04T17:39:14.480 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:26kNA331H6cZgglEIUjhuNnTFIo= 2024-03-04T17:39:14.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:26kNA331H6cZgglEIUjhuNnTFIo=', 'Content-Length': '0'} 2024-03-04T17:39:14.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008a88b73401472c5-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:14.484 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:14.484 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:14.484 DEBUG:boto:Method: GET 2024-03-04T17:39:14.484 DEBUG:boto:Path: /ppfahn-11/%3A 2024-03-04T17:39:14.484 DEBUG:boto:Data: 2024-03-04T17:39:14.484 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.484 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.484 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.484 DEBUG:boto:Params: {} 2024-03-04T17:39:14.484 DEBUG:boto:Token: None 2024-03-04T17:39:14.485 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/%3A 2024-03-04T17:39:14.485 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LaqdG13dN4k5J439taibLsvZsu4= 2024-03-04T17:39:14.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LaqdG13dN4k5J439taibLsvZsu4=', 'Content-Length': '0'} 2024-03-04T17:39:14.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c74a200f89d88fe3-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.528 DEBUG:boto:path=/ppfahn-11/%3A 2024-03-04T17:39:14.528 DEBUG:boto:auth_path=/ppfahn-11/%3A 2024-03-04T17:39:14.528 DEBUG:boto:Method: GET 2024-03-04T17:39:14.528 DEBUG:boto:Path: /ppfahn-11/%3A 2024-03-04T17:39:14.528 DEBUG:boto:Data: 2024-03-04T17:39:14.529 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.529 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.529 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.529 DEBUG:boto:Params: {} 2024-03-04T17:39:14.529 DEBUG:boto:Token: None 2024-03-04T17:39:14.529 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/%3A 2024-03-04T17:39:14.529 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LaqdG13dN4k5J439taibLsvZsu4= 2024-03-04T17:39:14.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LaqdG13dN4k5J439taibLsvZsu4=', 'Content-Length': '0'} 2024-03-04T17:39:14.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e797ae25900c410-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.532 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.532 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.533 DEBUG:boto:path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:14.533 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:14.533 DEBUG:boto:Method: GET 2024-03-04T17:39:14.533 DEBUG:boto:Path: /ppfahn-11/?prefix=%3A 2024-03-04T17:39:14.533 DEBUG:boto:Data: 2024-03-04T17:39:14.533 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.533 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.533 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.533 DEBUG:boto:Params: {} 2024-03-04T17:39:14.533 DEBUG:boto:Token: None 2024-03-04T17:39:14.534 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.534 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f67145fb339138d-0065e60742-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.580 DEBUG:boto:b'ppfahn-11:1000false:2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.580 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.581 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.581 DEBUG:boto:path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:14.581 DEBUG:boto:auth_path=/ppfahn-11/?prefix=%3A 2024-03-04T17:39:14.581 DEBUG:boto:Method: GET 2024-03-04T17:39:14.581 DEBUG:boto:Path: /ppfahn-11/?prefix=%3A 2024-03-04T17:39:14.581 DEBUG:boto:Data: 2024-03-04T17:39:14.581 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.581 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.581 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.581 DEBUG:boto:Params: {} 2024-03-04T17:39:14.582 DEBUG:boto:Token: None 2024-03-04T17:39:14.582 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.582 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d1575cde37ee437-0065e60742-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.587 DEBUG:boto:b'ppfahn-11:1000false:2024-03-04T17:36:30.422Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:14.587 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:14.588 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:14.588 DEBUG:boto:path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.588 DEBUG:boto:auth_path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.588 DEBUG:boto:Method: GET 2024-03-04T17:39:14.588 DEBUG:boto:Path: /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.588 DEBUG:boto:Data: 2024-03-04T17:39:14.588 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.588 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.588 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.588 DEBUG:boto:Params: {} 2024-03-04T17:39:14.589 DEBUG:boto:Token: None 2024-03-04T17:39:14.589 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.589 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A/1P4g11Mc45SrRnNON//8wtHwg= 2024-03-04T17:39:14.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A/1P4g11Mc45SrRnNON//8wtHwg=', 'Content-Length': '0'} 2024-03-04T17:39:14.592 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fceda03bb5b2daeb-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.632 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:14.632 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:14.632 DEBUG:boto:path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.633 DEBUG:boto:auth_path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.633 DEBUG:boto:Method: GET 2024-03-04T17:39:14.633 DEBUG:boto:Path: /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.633 DEBUG:boto:Data: 2024-03-04T17:39:14.633 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.633 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.633 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.633 DEBUG:boto:Params: {} 2024-03-04T17:39:14.633 DEBUG:boto:Token: None 2024-03-04T17:39:14.634 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.634 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A/1P4g11Mc45SrRnNON//8wtHwg= 2024-03-04T17:39:14.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A/1P4g11Mc45SrRnNON//8wtHwg=', 'Content-Length': '0'} 2024-03-04T17:39:14.637 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006df951ecdeffae8e-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.637 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:14.637 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:14.637 DEBUG:boto:path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.638 DEBUG:boto:auth_path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.638 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.638 DEBUG:boto:Path: /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.638 DEBUG:boto:Data: 2024-03-04T17:39:14.638 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.638 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.638 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.638 DEBUG:boto:Params: {} 2024-03-04T17:39:14.638 DEBUG:boto:Token: None 2024-03-04T17:39:14.639 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.639 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U4DkuqPQw+ZPlP7S64U0NuaGNrY= 2024-03-04T17:39:14.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U4DkuqPQw+ZPlP7S64U0NuaGNrY=', 'Content-Length': '0'} 2024-03-04T17:39:14.642 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093757019b8521850-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.642 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:14.642 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:14.642 DEBUG:boto:path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.642 DEBUG:boto:auth_path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.642 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.642 DEBUG:boto:Path: /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.643 DEBUG:boto:Data: 2024-03-04T17:39:14.643 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.643 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.643 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.643 DEBUG:boto:Params: {} 2024-03-04T17:39:14.643 DEBUG:boto:Token: None 2024-03-04T17:39:14.643 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.643 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U4DkuqPQw+ZPlP7S64U0NuaGNrY= 2024-03-04T17:39:14.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U4DkuqPQw+ZPlP7S64U0NuaGNrY=', 'Content-Length': '0'} 2024-03-04T17:39:14.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000880d6a4bb147b36b-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:14.647 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:14.647 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:14.647 DEBUG:boto:path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.647 DEBUG:boto:auth_path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.647 DEBUG:boto:Method: GET 2024-03-04T17:39:14.647 DEBUG:boto:Path: /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.647 DEBUG:boto:Data: 2024-03-04T17:39:14.647 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.647 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.648 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.648 DEBUG:boto:Params: {} 2024-03-04T17:39:14.648 DEBUG:boto:Token: None 2024-03-04T17:39:14.648 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.648 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A/1P4g11Mc45SrRnNON//8wtHwg= 2024-03-04T17:39:14.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A/1P4g11Mc45SrRnNON//8wtHwg=', 'Content-Length': '0'} 2024-03-04T17:39:14.651 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc91c2579898040d-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.692 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:14.692 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:14.692 DEBUG:boto:path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.692 DEBUG:boto:auth_path=/ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.693 DEBUG:boto:Method: GET 2024-03-04T17:39:14.693 DEBUG:boto:Path: /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.693 DEBUG:boto:Data: 2024-03-04T17:39:14.693 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.693 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.693 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.693 DEBUG:boto:Params: {} 2024-03-04T17:39:14.693 DEBUG:boto:Token: None 2024-03-04T17:39:14.694 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/_myobj?versionId=null 2024-03-04T17:39:14.694 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A/1P4g11Mc45SrRnNON//8wtHwg= 2024-03-04T17:39:14.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A/1P4g11Mc45SrRnNON//8wtHwg=', 'Content-Length': '0'} 2024-03-04T17:39:14.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000758474a83ffdb2fe-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.697 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.697 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.698 DEBUG:boto:path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:14.698 DEBUG:boto:auth_path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:14.698 DEBUG:boto:Method: GET 2024-03-04T17:39:14.698 DEBUG:boto:Path: /ppfahn-11/?prefix=_myobj 2024-03-04T17:39:14.698 DEBUG:boto:Data: 2024-03-04T17:39:14.698 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.698 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.698 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.698 DEBUG:boto:Params: {} 2024-03-04T17:39:14.699 DEBUG:boto:Token: None 2024-03-04T17:39:14.699 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.699 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c8069bd333b4aa2-0065e60742-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.704 DEBUG:boto:b'ppfahn-11_myobj1000false_myobj2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.704 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.704 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.704 DEBUG:boto:path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:14.704 DEBUG:boto:auth_path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:14.704 DEBUG:boto:Method: GET 2024-03-04T17:39:14.704 DEBUG:boto:Path: /ppfahn-11/?prefix=_myobj 2024-03-04T17:39:14.705 DEBUG:boto:Data: 2024-03-04T17:39:14.705 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.705 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.705 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.705 DEBUG:boto:Params: {} 2024-03-04T17:39:14.705 DEBUG:boto:Token: None 2024-03-04T17:39:14.705 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.705 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071698498475c1a28-0065e60742-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.710 DEBUG:boto:b'ppfahn-11_myobj1000false_myobj2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.710 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:14.710 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:14.710 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.710 DEBUG:boto:Path: /ppfahn-11/_myobj 2024-03-04T17:39:14.710 DEBUG:boto:Data: 2024-03-04T17:39:14.710 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.711 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.711 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.711 DEBUG:boto:Params: {} 2024-03-04T17:39:14.711 DEBUG:boto:Token: None 2024-03-04T17:39:14.711 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/_myobj 2024-03-04T17:39:14.711 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ddq5H2WW9xJ5ESdreUv9yfaXuY4= 2024-03-04T17:39:14.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ddq5H2WW9xJ5ESdreUv9yfaXuY4=', 'Content-Length': '0'} 2024-03-04T17:39:14.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d528ae2beee647da-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.714 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:14.714 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:14.714 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.714 DEBUG:boto:Path: /ppfahn-11/_myobj 2024-03-04T17:39:14.714 DEBUG:boto:Data: 2024-03-04T17:39:14.714 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.715 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.715 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.715 DEBUG:boto:Params: {} 2024-03-04T17:39:14.715 DEBUG:boto:Token: None 2024-03-04T17:39:14.715 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/_myobj 2024-03-04T17:39:14.715 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ddq5H2WW9xJ5ESdreUv9yfaXuY4= 2024-03-04T17:39:14.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ddq5H2WW9xJ5ESdreUv9yfaXuY4=', 'Content-Length': '0'} 2024-03-04T17:39:14.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008195b602916c34af-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:14.718 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:14.719 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:14.719 DEBUG:boto:Method: GET 2024-03-04T17:39:14.719 DEBUG:boto:Path: /ppfahn-11/_myobj 2024-03-04T17:39:14.719 DEBUG:boto:Data: 2024-03-04T17:39:14.719 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.719 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.719 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.719 DEBUG:boto:Params: {} 2024-03-04T17:39:14.719 DEBUG:boto:Token: None 2024-03-04T17:39:14.719 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/_myobj 2024-03-04T17:39:14.720 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tdl20m/fDlPpd7DpjCAzSjo2S+M= 2024-03-04T17:39:14.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tdl20m/fDlPpd7DpjCAzSjo2S+M=', 'Content-Length': '0'} 2024-03-04T17:39:14.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008f6822d3bdef2d9-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.764 DEBUG:boto:path=/ppfahn-11/_myobj 2024-03-04T17:39:14.764 DEBUG:boto:auth_path=/ppfahn-11/_myobj 2024-03-04T17:39:14.764 DEBUG:boto:Method: GET 2024-03-04T17:39:14.765 DEBUG:boto:Path: /ppfahn-11/_myobj 2024-03-04T17:39:14.765 DEBUG:boto:Data: 2024-03-04T17:39:14.765 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.765 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.765 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.765 DEBUG:boto:Params: {} 2024-03-04T17:39:14.765 DEBUG:boto:Token: None 2024-03-04T17:39:14.765 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/_myobj 2024-03-04T17:39:14.765 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tdl20m/fDlPpd7DpjCAzSjo2S+M= 2024-03-04T17:39:14.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tdl20m/fDlPpd7DpjCAzSjo2S+M=', 'Content-Length': '0'} 2024-03-04T17:39:14.768 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f69b42bb3c0b0bca-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.769 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.769 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.769 DEBUG:boto:path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:14.769 DEBUG:boto:auth_path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:14.769 DEBUG:boto:Method: GET 2024-03-04T17:39:14.769 DEBUG:boto:Path: /ppfahn-11/?prefix=_myobj 2024-03-04T17:39:14.769 DEBUG:boto:Data: 2024-03-04T17:39:14.769 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.770 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.770 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.770 DEBUG:boto:Params: {} 2024-03-04T17:39:14.770 DEBUG:boto:Token: None 2024-03-04T17:39:14.770 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.770 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d91f8bd9e7b26a37-0065e60742-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.775 DEBUG:boto:b'ppfahn-11_myobj1000false_myobj2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.775 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.776 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.776 DEBUG:boto:path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:14.776 DEBUG:boto:auth_path=/ppfahn-11/?prefix=_myobj 2024-03-04T17:39:14.776 DEBUG:boto:Method: GET 2024-03-04T17:39:14.776 DEBUG:boto:Path: /ppfahn-11/?prefix=_myobj 2024-03-04T17:39:14.776 DEBUG:boto:Data: 2024-03-04T17:39:14.776 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.776 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.776 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.776 DEBUG:boto:Params: {} 2024-03-04T17:39:14.777 DEBUG:boto:Token: None 2024-03-04T17:39:14.777 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.777 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008472cf7d095f02e-0065e60742-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.781 DEBUG:boto:b'ppfahn-11_myobj1000false_myobj2024-03-04T17:36:30.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:14.782 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:14.782 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:14.782 DEBUG:boto:path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.782 DEBUG:boto:auth_path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.782 DEBUG:boto:Method: GET 2024-03-04T17:39:14.782 DEBUG:boto:Path: /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.782 DEBUG:boto:Data: 2024-03-04T17:39:14.782 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.782 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.782 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.782 DEBUG:boto:Params: {} 2024-03-04T17:39:14.783 DEBUG:boto:Token: None 2024-03-04T17:39:14.783 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.783 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wksHNjWSrpMRx1AVA6P83QltcdA= 2024-03-04T17:39:14.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wksHNjWSrpMRx1AVA6P83QltcdA=', 'Content-Length': '0'} 2024-03-04T17:39:14.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069486cccfe9e0e7e-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.786 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:14.786 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:14.787 DEBUG:boto:path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.787 DEBUG:boto:auth_path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.787 DEBUG:boto:Method: GET 2024-03-04T17:39:14.787 DEBUG:boto:Path: /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.787 DEBUG:boto:Data: 2024-03-04T17:39:14.787 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.787 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.787 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.787 DEBUG:boto:Params: {} 2024-03-04T17:39:14.787 DEBUG:boto:Token: None 2024-03-04T17:39:14.788 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.788 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wksHNjWSrpMRx1AVA6P83QltcdA= 2024-03-04T17:39:14.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wksHNjWSrpMRx1AVA6P83QltcdA=', 'Content-Length': '0'} 2024-03-04T17:39:14.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000653cb67154ea3fbf-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.791 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:14.792 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:14.792 DEBUG:boto:path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.792 DEBUG:boto:auth_path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.792 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.792 DEBUG:boto:Path: /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.792 DEBUG:boto:Data: 2024-03-04T17:39:14.792 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.792 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.792 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.792 DEBUG:boto:Params: {} 2024-03-04T17:39:14.793 DEBUG:boto:Token: None 2024-03-04T17:39:14.793 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.793 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MvFRnRQilLpZ+WTWxujikzynYUI= 2024-03-04T17:39:14.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MvFRnRQilLpZ+WTWxujikzynYUI=', 'Content-Length': '0'} 2024-03-04T17:39:14.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082d6dd1ec544d810-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.796 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:14.796 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:14.796 DEBUG:boto:path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.796 DEBUG:boto:auth_path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.796 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.796 DEBUG:boto:Path: /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.797 DEBUG:boto:Data: 2024-03-04T17:39:14.797 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.797 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.797 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.797 DEBUG:boto:Params: {} 2024-03-04T17:39:14.797 DEBUG:boto:Token: None 2024-03-04T17:39:14.797 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.797 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MvFRnRQilLpZ+WTWxujikzynYUI= 2024-03-04T17:39:14.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MvFRnRQilLpZ+WTWxujikzynYUI=', 'Content-Length': '0'} 2024-03-04T17:39:14.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e79e3974eb30343-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:14.801 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:14.801 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:14.801 DEBUG:boto:path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.801 DEBUG:boto:auth_path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.801 DEBUG:boto:Method: GET 2024-03-04T17:39:14.801 DEBUG:boto:Path: /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.801 DEBUG:boto:Data: 2024-03-04T17:39:14.801 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.801 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.801 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.801 DEBUG:boto:Params: {} 2024-03-04T17:39:14.802 DEBUG:boto:Token: None 2024-03-04T17:39:14.802 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.802 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wksHNjWSrpMRx1AVA6P83QltcdA= 2024-03-04T17:39:14.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wksHNjWSrpMRx1AVA6P83QltcdA=', 'Content-Length': '0'} 2024-03-04T17:39:14.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003572730903f6a98-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.848 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:14.848 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:14.848 DEBUG:boto:path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.848 DEBUG:boto:auth_path=/ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.849 DEBUG:boto:Method: GET 2024-03-04T17:39:14.849 DEBUG:boto:Path: /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.849 DEBUG:boto:Data: 2024-03-04T17:39:14.849 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.849 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.849 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.849 DEBUG:boto:Params: {} 2024-03-04T17:39:14.849 DEBUG:boto:Token: None 2024-03-04T17:39:14.850 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/myobj?versionId=null 2024-03-04T17:39:14.850 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wksHNjWSrpMRx1AVA6P83QltcdA= 2024-03-04T17:39:14.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wksHNjWSrpMRx1AVA6P83QltcdA=', 'Content-Length': '0'} 2024-03-04T17:39:14.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b65edb8941ecd6b-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.853 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.853 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.853 DEBUG:boto:path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:14.853 DEBUG:boto:auth_path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:14.853 DEBUG:boto:Method: GET 2024-03-04T17:39:14.854 DEBUG:boto:Path: /ppfahn-11/?prefix=myobj 2024-03-04T17:39:14.854 DEBUG:boto:Data: 2024-03-04T17:39:14.854 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.854 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.854 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.854 DEBUG:boto:Params: {} 2024-03-04T17:39:14.854 DEBUG:boto:Token: None 2024-03-04T17:39:14.854 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.855 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e63c4d9fe92b0e8c-0065e60742-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.859 DEBUG:boto:b'ppfahn-11myobj1000falsemyobj2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.859 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.860 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.860 DEBUG:boto:path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:14.860 DEBUG:boto:auth_path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:14.860 DEBUG:boto:Method: GET 2024-03-04T17:39:14.860 DEBUG:boto:Path: /ppfahn-11/?prefix=myobj 2024-03-04T17:39:14.860 DEBUG:boto:Data: 2024-03-04T17:39:14.860 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.860 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.860 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.860 DEBUG:boto:Params: {} 2024-03-04T17:39:14.861 DEBUG:boto:Token: None 2024-03-04T17:39:14.861 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.861 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcef8639aef05059-0065e60742-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.908 DEBUG:boto:b'ppfahn-11myobj1000falsemyobj2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.908 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:14.909 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:14.909 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.909 DEBUG:boto:Path: /ppfahn-11/myobj 2024-03-04T17:39:14.909 DEBUG:boto:Data: 2024-03-04T17:39:14.909 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.909 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.909 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.909 DEBUG:boto:Params: {} 2024-03-04T17:39:14.909 DEBUG:boto:Token: None 2024-03-04T17:39:14.910 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/myobj 2024-03-04T17:39:14.910 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CGI6c3dT0m+kOv52cMW7jbzb0Hs= 2024-03-04T17:39:14.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CGI6c3dT0m+kOv52cMW7jbzb0Hs=', 'Content-Length': '0'} 2024-03-04T17:39:14.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059798b318f5df32e-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.913 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:14.913 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:14.913 DEBUG:boto:Method: HEAD 2024-03-04T17:39:14.913 DEBUG:boto:Path: /ppfahn-11/myobj 2024-03-04T17:39:14.913 DEBUG:boto:Data: 2024-03-04T17:39:14.914 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.914 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.914 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.914 DEBUG:boto:Params: {} 2024-03-04T17:39:14.914 DEBUG:boto:Token: None 2024-03-04T17:39:14.914 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/myobj 2024-03-04T17:39:14.914 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CGI6c3dT0m+kOv52cMW7jbzb0Hs= 2024-03-04T17:39:14.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CGI6c3dT0m+kOv52cMW7jbzb0Hs=', 'Content-Length': '0'} 2024-03-04T17:39:14.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057f99f1c2a94f493-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:14.917 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:14.918 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:14.918 DEBUG:boto:Method: GET 2024-03-04T17:39:14.918 DEBUG:boto:Path: /ppfahn-11/myobj 2024-03-04T17:39:14.918 DEBUG:boto:Data: 2024-03-04T17:39:14.918 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.918 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.918 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.918 DEBUG:boto:Params: {} 2024-03-04T17:39:14.918 DEBUG:boto:Token: None 2024-03-04T17:39:14.919 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/myobj 2024-03-04T17:39:14.919 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Xl72gJu0tTa37Y5yE/mD4diFW6g= 2024-03-04T17:39:14.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Xl72gJu0tTa37Y5yE/mD4diFW6g=', 'Content-Length': '0'} 2024-03-04T17:39:14.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-11:d96636ac-054d-40e3-90f3-e3912da784b2.4349.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adb585d499f72c4b-0065e60742-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.964 DEBUG:boto:path=/ppfahn-11/myobj 2024-03-04T17:39:14.964 DEBUG:boto:auth_path=/ppfahn-11/myobj 2024-03-04T17:39:14.964 DEBUG:boto:Method: GET 2024-03-04T17:39:14.964 DEBUG:boto:Path: /ppfahn-11/myobj 2024-03-04T17:39:14.964 DEBUG:boto:Data: 2024-03-04T17:39:14.965 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.965 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.965 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.965 DEBUG:boto:Params: {} 2024-03-04T17:39:14.965 DEBUG:boto:Token: None 2024-03-04T17:39:14.965 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/myobj 2024-03-04T17:39:14.965 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Xl72gJu0tTa37Y5yE/mD4diFW6g= 2024-03-04T17:39:14.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Xl72gJu0tTa37Y5yE/mD4diFW6g=', 'Content-Length': '0'} 2024-03-04T17:39:14.968 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003aafbfd9e5fac282-0065e60742-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.969 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.969 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.969 DEBUG:boto:path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:14.969 DEBUG:boto:auth_path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:14.969 DEBUG:boto:Method: GET 2024-03-04T17:39:14.969 DEBUG:boto:Path: /ppfahn-11/?prefix=myobj 2024-03-04T17:39:14.969 DEBUG:boto:Data: 2024-03-04T17:39:14.969 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.969 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:14.969 DEBUG:boto:Port: 8001 2024-03-04T17:39:14.970 DEBUG:boto:Params: {} 2024-03-04T17:39:14.970 DEBUG:boto:Token: None 2024-03-04T17:39:14.970 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.970 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086cb63ff0af8ba84-0065e60742-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.975 DEBUG:boto:b'ppfahn-11myobj1000falsemyobj2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.975 DEBUG:boto:path=/ppfahn-11/ 2024-03-04T17:39:14.975 DEBUG:boto:auth_path=/ppfahn-11/ 2024-03-04T17:39:14.975 DEBUG:boto:path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:14.975 DEBUG:boto:auth_path=/ppfahn-11/?prefix=myobj 2024-03-04T17:39:14.975 DEBUG:boto:Method: GET 2024-03-04T17:39:14.975 DEBUG:boto:Path: /ppfahn-11/?prefix=myobj 2024-03-04T17:39:14.975 DEBUG:boto:Data: 2024-03-04T17:39:14.976 DEBUG:boto:Headers: {} 2024-03-04T17:39:14.976 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:14.976 DEBUG:boto:Port: 8000 2024-03-04T17:39:14.976 DEBUG:boto:Params: {} 2024-03-04T17:39:14.976 DEBUG:boto:Token: None 2024-03-04T17:39:14.976 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:14 GMT /ppfahn-11/ 2024-03-04T17:39:14.976 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA= 2024-03-04T17:39:14.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XE6MpuKjPl6SUIe036kZeNwJWUA=', 'Content-Length': '0'} 2024-03-04T17:39:14.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c7624fa8f1434ef-0065e60742-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:14.981 DEBUG:boto:b'ppfahn-11myobj1000falsemyobj2024-03-04T17:36:30.342Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:14.981 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-11 zones={test-zone3, test-zone2} 2024-03-04T17:39:14.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:14.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:14.982 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:39:15.032 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:15.033 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:15.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:15.092+0000 7f414df87980 1 bucket sync caught up with source: 2024-03-04T17:39:15.093 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000002.9.6, , 00000000001.9.6, , 00000000001.9.6, 00000000002.26.6, 00000000001.9.6, 00000000001.9.6, 00000000001.14.6, ] 2024-03-04T17:39:15.093 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.9.6, , 00000000001.9.6, , 00000000001.9.6, 00000000002.26.6, 00000000001.9.6, 00000000001.9.6, 00000000001.14.6, ] 2024-03-04T17:39:15.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:15.092+0000 7f414df87980 0 bucket checkpoint complete 2024-03-04T17:39:15.115 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-12 zones={test-zone1, test-zone3} 2024-03-04T17:39:15.115 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.115 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.115 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.115 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:39:15.115 DEBUG:boto:Data: 2024-03-04T17:39:15.115 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.115 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.115 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.116 DEBUG:boto:Params: {} 2024-03-04T17:39:15.116 DEBUG:boto:Token: None 2024-03-04T17:39:15.116 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.116 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:crn8TQ/tXaQhTScCtWuXDuT5Zi0= 2024-03-04T17:39:15.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:crn8TQ/tXaQhTScCtWuXDuT5Zi0=', 'Content-Length': '0'} 2024-03-04T17:39:15.118 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015b4c42dc6a49736-0065e60743-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.118 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.119 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.119 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.119 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:39:15.119 DEBUG:boto:Data: 2024-03-04T17:39:15.119 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.119 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.119 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.119 DEBUG:boto:Params: {} 2024-03-04T17:39:15.119 DEBUG:boto:Token: None 2024-03-04T17:39:15.119 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.120 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:crn8TQ/tXaQhTScCtWuXDuT5Zi0= 2024-03-04T17:39:15.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:crn8TQ/tXaQhTScCtWuXDuT5Zi0=', 'Content-Length': '0'} 2024-03-04T17:39:15.122 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e301a370751f8b01-0065e60743-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.122 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:39:15.122 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.122 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.122 DEBUG:boto:path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:15.122 DEBUG:boto:auth_path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:15.123 DEBUG:boto:Method: GET 2024-03-04T17:39:15.123 DEBUG:boto:Path: /ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:15.123 DEBUG:boto:Data: 2024-03-04T17:39:15.123 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.123 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.123 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.123 DEBUG:boto:Params: {} 2024-03-04T17:39:15.123 DEBUG:boto:Token: None 2024-03-04T17:39:15.123 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/?versions 2024-03-04T17:39:15.123 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gSQlOOZfXPYOgeTRlxwzUuJ5OYk= 2024-03-04T17:39:15.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gSQlOOZfXPYOgeTRlxwzUuJ5OYk=', 'Content-Length': '0'} 2024-03-04T17:39:15.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c9e16d56b629bb0-0065e60743-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.126 DEBUG:boto:b'ppfahn-12999false&nulltrue2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T17:39:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T17:39:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T17:39:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T17:39:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T17:39:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T17:39:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T17:39:15.128 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T17:39:15.128 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:15.128 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:39:15.128 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.128 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.128 DEBUG:boto:path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:15.128 DEBUG:boto:auth_path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:15.128 DEBUG:boto:Method: GET 2024-03-04T17:39:15.128 DEBUG:boto:Path: /ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:15.128 DEBUG:boto:Data: 2024-03-04T17:39:15.129 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.129 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.129 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.129 DEBUG:boto:Params: {} 2024-03-04T17:39:15.129 DEBUG:boto:Token: None 2024-03-04T17:39:15.129 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/?versions 2024-03-04T17:39:15.129 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gSQlOOZfXPYOgeTRlxwzUuJ5OYk= 2024-03-04T17:39:15.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gSQlOOZfXPYOgeTRlxwzUuJ5OYk=', 'Content-Length': '0'} 2024-03-04T17:39:15.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033ccb7afbb2ce14e-0065e60743-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.132 DEBUG:boto:b'ppfahn-12999false&nulltrue2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.132 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T17:39:15.133 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T17:39:15.133 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T17:39:15.133 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T17:39:15.133 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T17:39:15.133 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T17:39:15.133 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T17:39:15.133 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T17:39:15.133 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:15.133 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.134 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.134 DEBUG:boto:path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:15.134 DEBUG:boto:auth_path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:15.134 DEBUG:boto:Method: GET 2024-03-04T17:39:15.134 DEBUG:boto:Path: /ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:15.134 DEBUG:boto:Data: 2024-03-04T17:39:15.134 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.134 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.134 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.134 DEBUG:boto:Params: {} 2024-03-04T17:39:15.135 DEBUG:boto:Token: None 2024-03-04T17:39:15.135 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/?versions 2024-03-04T17:39:15.135 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gSQlOOZfXPYOgeTRlxwzUuJ5OYk= 2024-03-04T17:39:15.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gSQlOOZfXPYOgeTRlxwzUuJ5OYk=', 'Content-Length': '0'} 2024-03-04T17:39:15.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0df67f8f2cc422f-0065e60743-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.138 DEBUG:boto:b'ppfahn-12999false&nulltrue2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.139 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.139 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.139 DEBUG:boto:path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:15.139 DEBUG:boto:auth_path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:15.139 DEBUG:boto:Method: GET 2024-03-04T17:39:15.139 DEBUG:boto:Path: /ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:15.139 DEBUG:boto:Data: 2024-03-04T17:39:15.139 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.139 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.139 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.140 DEBUG:boto:Params: {} 2024-03-04T17:39:15.140 DEBUG:boto:Token: None 2024-03-04T17:39:15.140 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/?versions 2024-03-04T17:39:15.140 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gSQlOOZfXPYOgeTRlxwzUuJ5OYk= 2024-03-04T17:39:15.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gSQlOOZfXPYOgeTRlxwzUuJ5OYk=', 'Content-Length': '0'} 2024-03-04T17:39:15.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094c5838693683c85-0065e60743-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.143 DEBUG:boto:b'ppfahn-12999false&nulltrue2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:15.143 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:15.144 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:15.144 DEBUG:boto:path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.144 DEBUG:boto:auth_path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.144 DEBUG:boto:Method: GET 2024-03-04T17:39:15.144 DEBUG:boto:Path: /ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.144 DEBUG:boto:Data: 2024-03-04T17:39:15.144 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.144 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.144 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.144 DEBUG:boto:Params: {} 2024-03-04T17:39:15.145 DEBUG:boto:Token: None 2024-03-04T17:39:15.145 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.145 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DsYO4rLFYR3gbSdrf2WrxuD/PyI= 2024-03-04T17:39:15.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DsYO4rLFYR3gbSdrf2WrxuD/PyI=', 'Content-Length': '0'} 2024-03-04T17:39:15.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d67a5c4e9cc1688-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.147 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:15.147 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:15.147 DEBUG:boto:path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.147 DEBUG:boto:auth_path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.147 DEBUG:boto:Method: GET 2024-03-04T17:39:15.147 DEBUG:boto:Path: /ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.147 DEBUG:boto:Data: 2024-03-04T17:39:15.148 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.148 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.148 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.148 DEBUG:boto:Params: {} 2024-03-04T17:39:15.148 DEBUG:boto:Token: None 2024-03-04T17:39:15.148 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.148 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DsYO4rLFYR3gbSdrf2WrxuD/PyI= 2024-03-04T17:39:15.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DsYO4rLFYR3gbSdrf2WrxuD/PyI=', 'Content-Length': '0'} 2024-03-04T17:39:15.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be3aaf7e04c704bc-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.151 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:15.151 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:15.151 DEBUG:boto:path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.151 DEBUG:boto:auth_path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.151 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.151 DEBUG:boto:Path: /ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.151 DEBUG:boto:Data: 2024-03-04T17:39:15.151 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.151 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.152 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.152 DEBUG:boto:Params: {} 2024-03-04T17:39:15.152 DEBUG:boto:Token: None 2024-03-04T17:39:15.152 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.152 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2PJApKcDShmKydCkCjMu3CmvzRQ= 2024-03-04T17:39:15.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2PJApKcDShmKydCkCjMu3CmvzRQ=', 'Content-Length': '0'} 2024-03-04T17:39:15.154 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004575c8bb59c504e5-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.155 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:15.155 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:15.155 DEBUG:boto:path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.155 DEBUG:boto:auth_path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.155 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.155 DEBUG:boto:Path: /ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.155 DEBUG:boto:Data: 2024-03-04T17:39:15.155 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.156 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.156 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.156 DEBUG:boto:Params: {} 2024-03-04T17:39:15.156 DEBUG:boto:Token: None 2024-03-04T17:39:15.156 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.156 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2PJApKcDShmKydCkCjMu3CmvzRQ= 2024-03-04T17:39:15.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2PJApKcDShmKydCkCjMu3CmvzRQ=', 'Content-Length': '0'} 2024-03-04T17:39:15.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f569d13d16ac807f-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:15.159 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:15.159 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:15.159 DEBUG:boto:path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.159 DEBUG:boto:auth_path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.159 DEBUG:boto:Method: GET 2024-03-04T17:39:15.159 DEBUG:boto:Path: /ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.159 DEBUG:boto:Data: 2024-03-04T17:39:15.160 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.160 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.160 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.160 DEBUG:boto:Params: {} 2024-03-04T17:39:15.160 DEBUG:boto:Token: None 2024-03-04T17:39:15.160 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.160 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DsYO4rLFYR3gbSdrf2WrxuD/PyI= 2024-03-04T17:39:15.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DsYO4rLFYR3gbSdrf2WrxuD/PyI=', 'Content-Length': '0'} 2024-03-04T17:39:15.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028168d42a0ce8923-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.163 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:15.163 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:15.163 DEBUG:boto:path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.163 DEBUG:boto:auth_path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.163 DEBUG:boto:Method: GET 2024-03-04T17:39:15.163 DEBUG:boto:Path: /ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.163 DEBUG:boto:Data: 2024-03-04T17:39:15.164 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.164 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.164 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.164 DEBUG:boto:Params: {} 2024-03-04T17:39:15.164 DEBUG:boto:Token: None 2024-03-04T17:39:15.164 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/%26?versionId=null 2024-03-04T17:39:15.164 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DsYO4rLFYR3gbSdrf2WrxuD/PyI= 2024-03-04T17:39:15.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DsYO4rLFYR3gbSdrf2WrxuD/PyI=', 'Content-Length': '0'} 2024-03-04T17:39:15.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011138445ec546166-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.167 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.168 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.168 DEBUG:boto:path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:15.168 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:15.168 DEBUG:boto:Method: GET 2024-03-04T17:39:15.168 DEBUG:boto:Path: /ppfahn-12/?prefix=%26 2024-03-04T17:39:15.168 DEBUG:boto:Data: 2024-03-04T17:39:15.168 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.168 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.168 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.168 DEBUG:boto:Params: {} 2024-03-04T17:39:15.168 DEBUG:boto:Token: None 2024-03-04T17:39:15.169 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.169 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000492583a5d7370ff6-0065e60743-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.172 DEBUG:boto:b'ppfahn-12&1000false&2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.173 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.173 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.173 DEBUG:boto:path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:15.173 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:15.173 DEBUG:boto:Method: GET 2024-03-04T17:39:15.173 DEBUG:boto:Path: /ppfahn-12/?prefix=%26 2024-03-04T17:39:15.173 DEBUG:boto:Data: 2024-03-04T17:39:15.173 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.173 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.174 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.174 DEBUG:boto:Params: {} 2024-03-04T17:39:15.174 DEBUG:boto:Token: None 2024-03-04T17:39:15.174 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.174 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6c8fb44eeb05dbf-0065e60743-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.178 DEBUG:boto:b'ppfahn-12&1000false&2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.178 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:15.178 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:15.178 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.178 DEBUG:boto:Path: /ppfahn-12/%26 2024-03-04T17:39:15.179 DEBUG:boto:Data: 2024-03-04T17:39:15.179 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.179 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.179 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.179 DEBUG:boto:Params: {} 2024-03-04T17:39:15.179 DEBUG:boto:Token: None 2024-03-04T17:39:15.179 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/%26 2024-03-04T17:39:15.179 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1s0hANOLZN0m+70i4JEkMcojUjY= 2024-03-04T17:39:15.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1s0hANOLZN0m+70i4JEkMcojUjY=', 'Content-Length': '0'} 2024-03-04T17:39:15.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7aa788ff2fbe6eb-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.182 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:15.182 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:15.182 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.182 DEBUG:boto:Path: /ppfahn-12/%26 2024-03-04T17:39:15.182 DEBUG:boto:Data: 2024-03-04T17:39:15.182 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.183 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.183 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.183 DEBUG:boto:Params: {} 2024-03-04T17:39:15.183 DEBUG:boto:Token: None 2024-03-04T17:39:15.183 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/%26 2024-03-04T17:39:15.183 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1s0hANOLZN0m+70i4JEkMcojUjY= 2024-03-04T17:39:15.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1s0hANOLZN0m+70i4JEkMcojUjY=', 'Content-Length': '0'} 2024-03-04T17:39:15.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000529e62656e23bd2e-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:15.186 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:15.186 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:15.186 DEBUG:boto:Method: GET 2024-03-04T17:39:15.187 DEBUG:boto:Path: /ppfahn-12/%26 2024-03-04T17:39:15.187 DEBUG:boto:Data: 2024-03-04T17:39:15.187 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.187 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.187 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.187 DEBUG:boto:Params: {} 2024-03-04T17:39:15.187 DEBUG:boto:Token: None 2024-03-04T17:39:15.187 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/%26 2024-03-04T17:39:15.187 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Dmcb6GXyoECAVYQuRfj+I2r4ZkU= 2024-03-04T17:39:15.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Dmcb6GXyoECAVYQuRfj+I2r4ZkU=', 'Content-Length': '0'} 2024-03-04T17:39:15.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a42ec6b4653641a0-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.190 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:15.190 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:15.190 DEBUG:boto:Method: GET 2024-03-04T17:39:15.190 DEBUG:boto:Path: /ppfahn-12/%26 2024-03-04T17:39:15.190 DEBUG:boto:Data: 2024-03-04T17:39:15.190 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.190 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.190 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.190 DEBUG:boto:Params: {} 2024-03-04T17:39:15.191 DEBUG:boto:Token: None 2024-03-04T17:39:15.191 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/%26 2024-03-04T17:39:15.191 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Dmcb6GXyoECAVYQuRfj+I2r4ZkU= 2024-03-04T17:39:15.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Dmcb6GXyoECAVYQuRfj+I2r4ZkU=', 'Content-Length': '0'} 2024-03-04T17:39:15.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fe4f729c2aa4218-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.236 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.236 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.236 DEBUG:boto:path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:15.237 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:15.237 DEBUG:boto:Method: GET 2024-03-04T17:39:15.237 DEBUG:boto:Path: /ppfahn-12/?prefix=%26 2024-03-04T17:39:15.237 DEBUG:boto:Data: 2024-03-04T17:39:15.237 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.237 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.237 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.237 DEBUG:boto:Params: {} 2024-03-04T17:39:15.237 DEBUG:boto:Token: None 2024-03-04T17:39:15.237 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.237 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbc6edc99332eec9-0065e60743-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.242 DEBUG:boto:b'ppfahn-12&1000false&2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.242 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.242 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.242 DEBUG:boto:path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:15.242 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:15.242 DEBUG:boto:Method: GET 2024-03-04T17:39:15.242 DEBUG:boto:Path: /ppfahn-12/?prefix=%26 2024-03-04T17:39:15.242 DEBUG:boto:Data: 2024-03-04T17:39:15.243 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.243 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.243 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.243 DEBUG:boto:Params: {} 2024-03-04T17:39:15.243 DEBUG:boto:Token: None 2024-03-04T17:39:15.243 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.243 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087e3263484383f9a-0065e60743-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.248 DEBUG:boto:b'ppfahn-12&1000false&2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:15.249 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:15.249 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:15.249 DEBUG:boto:path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:15.249 DEBUG:boto:auth_path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:15.249 DEBUG:boto:Method: GET 2024-03-04T17:39:15.249 DEBUG:boto:Path: /ppfahn-12/.?versionId=null 2024-03-04T17:39:15.249 DEBUG:boto:Data: 2024-03-04T17:39:15.249 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.249 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.250 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.250 DEBUG:boto:Params: {} 2024-03-04T17:39:15.250 DEBUG:boto:Token: None 2024-03-04T17:39:15.250 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.?versionId=null 2024-03-04T17:39:15.250 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wv/dvfEMXIX9iThtq3anzXOG7cM= 2024-03-04T17:39:15.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wv/dvfEMXIX9iThtq3anzXOG7cM=', 'Content-Length': '0'} 2024-03-04T17:39:15.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000042c87cc7e5d2c8b-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.296 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:15.296 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:15.296 DEBUG:boto:path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:15.297 DEBUG:boto:auth_path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:15.297 DEBUG:boto:Method: GET 2024-03-04T17:39:15.297 DEBUG:boto:Path: /ppfahn-12/.?versionId=null 2024-03-04T17:39:15.297 DEBUG:boto:Data: 2024-03-04T17:39:15.297 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.297 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.297 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.297 DEBUG:boto:Params: {} 2024-03-04T17:39:15.297 DEBUG:boto:Token: None 2024-03-04T17:39:15.297 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.?versionId=null 2024-03-04T17:39:15.297 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wv/dvfEMXIX9iThtq3anzXOG7cM= 2024-03-04T17:39:15.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wv/dvfEMXIX9iThtq3anzXOG7cM=', 'Content-Length': '0'} 2024-03-04T17:39:15.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7e4ce62ddc36d7f-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.301 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:15.301 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:15.301 DEBUG:boto:path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:15.301 DEBUG:boto:auth_path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:15.301 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.301 DEBUG:boto:Path: /ppfahn-12/.?versionId=null 2024-03-04T17:39:15.301 DEBUG:boto:Data: 2024-03-04T17:39:15.301 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.301 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.302 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.302 DEBUG:boto:Params: {} 2024-03-04T17:39:15.302 DEBUG:boto:Token: None 2024-03-04T17:39:15.302 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.?versionId=null 2024-03-04T17:39:15.302 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xwUi/ywJwxSwYy8Iqux6giSwhtw= 2024-03-04T17:39:15.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xwUi/ywJwxSwYy8Iqux6giSwhtw=', 'Content-Length': '0'} 2024-03-04T17:39:15.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000009f2c3ca1a60146-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.305 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:15.306 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:15.306 DEBUG:boto:path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:15.306 DEBUG:boto:auth_path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:15.306 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.306 DEBUG:boto:Path: /ppfahn-12/.?versionId=null 2024-03-04T17:39:15.306 DEBUG:boto:Data: 2024-03-04T17:39:15.306 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.306 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.306 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.306 DEBUG:boto:Params: {} 2024-03-04T17:39:15.306 DEBUG:boto:Token: None 2024-03-04T17:39:15.307 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.?versionId=null 2024-03-04T17:39:15.307 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xwUi/ywJwxSwYy8Iqux6giSwhtw= 2024-03-04T17:39:15.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xwUi/ywJwxSwYy8Iqux6giSwhtw=', 'Content-Length': '0'} 2024-03-04T17:39:15.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000897c596f5fb8414c-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:15.310 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:15.310 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:15.310 DEBUG:boto:path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:15.310 DEBUG:boto:auth_path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:15.310 DEBUG:boto:Method: GET 2024-03-04T17:39:15.310 DEBUG:boto:Path: /ppfahn-12/.?versionId=null 2024-03-04T17:39:15.310 DEBUG:boto:Data: 2024-03-04T17:39:15.310 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.310 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.310 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.311 DEBUG:boto:Params: {} 2024-03-04T17:39:15.311 DEBUG:boto:Token: None 2024-03-04T17:39:15.311 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.?versionId=null 2024-03-04T17:39:15.311 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wv/dvfEMXIX9iThtq3anzXOG7cM= 2024-03-04T17:39:15.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wv/dvfEMXIX9iThtq3anzXOG7cM=', 'Content-Length': '0'} 2024-03-04T17:39:15.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d59154ce9a6b9abe-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.315 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:15.315 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:15.315 DEBUG:boto:path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:15.315 DEBUG:boto:auth_path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:15.315 DEBUG:boto:Method: GET 2024-03-04T17:39:15.315 DEBUG:boto:Path: /ppfahn-12/.?versionId=null 2024-03-04T17:39:15.315 DEBUG:boto:Data: 2024-03-04T17:39:15.315 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.315 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.315 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.316 DEBUG:boto:Params: {} 2024-03-04T17:39:15.316 DEBUG:boto:Token: None 2024-03-04T17:39:15.316 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.?versionId=null 2024-03-04T17:39:15.316 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wv/dvfEMXIX9iThtq3anzXOG7cM= 2024-03-04T17:39:15.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wv/dvfEMXIX9iThtq3anzXOG7cM=', 'Content-Length': '0'} 2024-03-04T17:39:15.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026c2a65a4f64cf11-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.360 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.360 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.360 DEBUG:boto:path=/ppfahn-12/?prefix=. 2024-03-04T17:39:15.360 DEBUG:boto:auth_path=/ppfahn-12/?prefix=. 2024-03-04T17:39:15.360 DEBUG:boto:Method: GET 2024-03-04T17:39:15.360 DEBUG:boto:Path: /ppfahn-12/?prefix=. 2024-03-04T17:39:15.361 DEBUG:boto:Data: 2024-03-04T17:39:15.361 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.361 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.361 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.361 DEBUG:boto:Params: {} 2024-03-04T17:39:15.361 DEBUG:boto:Token: None 2024-03-04T17:39:15.361 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.361 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033e3bebf0a7198bb-0065e60743-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.366 DEBUG:boto:b'ppfahn-12.1000false.2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.366 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.366 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.366 DEBUG:boto:path=/ppfahn-12/?prefix=. 2024-03-04T17:39:15.367 DEBUG:boto:auth_path=/ppfahn-12/?prefix=. 2024-03-04T17:39:15.367 DEBUG:boto:Method: GET 2024-03-04T17:39:15.367 DEBUG:boto:Path: /ppfahn-12/?prefix=. 2024-03-04T17:39:15.367 DEBUG:boto:Data: 2024-03-04T17:39:15.367 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.367 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.367 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.367 DEBUG:boto:Params: {} 2024-03-04T17:39:15.368 DEBUG:boto:Token: None 2024-03-04T17:39:15.368 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.368 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087f6f0352a4ee553-0065e60743-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.372 DEBUG:boto:b'ppfahn-12.1000false.2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.373 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:15.373 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:15.373 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.373 DEBUG:boto:Path: /ppfahn-12/. 2024-03-04T17:39:15.373 DEBUG:boto:Data: 2024-03-04T17:39:15.374 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.374 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.374 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.374 DEBUG:boto:Params: {} 2024-03-04T17:39:15.374 DEBUG:boto:Token: None 2024-03-04T17:39:15.374 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/. 2024-03-04T17:39:15.374 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:w24aWGcvElaTxABvp1OXBYvN/rk= 2024-03-04T17:39:15.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:w24aWGcvElaTxABvp1OXBYvN/rk=', 'Content-Length': '0'} 2024-03-04T17:39:15.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cbcdda89405fd64-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.377 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:15.377 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:15.377 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.378 DEBUG:boto:Path: /ppfahn-12/. 2024-03-04T17:39:15.378 DEBUG:boto:Data: 2024-03-04T17:39:15.378 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.378 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.378 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.378 DEBUG:boto:Params: {} 2024-03-04T17:39:15.378 DEBUG:boto:Token: None 2024-03-04T17:39:15.378 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/. 2024-03-04T17:39:15.378 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:w24aWGcvElaTxABvp1OXBYvN/rk= 2024-03-04T17:39:15.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:w24aWGcvElaTxABvp1OXBYvN/rk=', 'Content-Length': '0'} 2024-03-04T17:39:15.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000242cd05fe395c281-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:15.382 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:15.382 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:15.382 DEBUG:boto:Method: GET 2024-03-04T17:39:15.382 DEBUG:boto:Path: /ppfahn-12/. 2024-03-04T17:39:15.382 DEBUG:boto:Data: 2024-03-04T17:39:15.382 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.382 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.382 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.382 DEBUG:boto:Params: {} 2024-03-04T17:39:15.382 DEBUG:boto:Token: None 2024-03-04T17:39:15.383 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/. 2024-03-04T17:39:15.383 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2B90273l4Joxu+4Eb19O+AyvtJY= 2024-03-04T17:39:15.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2B90273l4Joxu+4Eb19O+AyvtJY=', 'Content-Length': '0'} 2024-03-04T17:39:15.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eca71a37eabc0111-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.432 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:15.432 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:15.432 DEBUG:boto:Method: GET 2024-03-04T17:39:15.432 DEBUG:boto:Path: /ppfahn-12/. 2024-03-04T17:39:15.432 DEBUG:boto:Data: 2024-03-04T17:39:15.432 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.433 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.433 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.433 DEBUG:boto:Params: {} 2024-03-04T17:39:15.433 DEBUG:boto:Token: None 2024-03-04T17:39:15.433 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/. 2024-03-04T17:39:15.433 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2B90273l4Joxu+4Eb19O+AyvtJY= 2024-03-04T17:39:15.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2B90273l4Joxu+4Eb19O+AyvtJY=', 'Content-Length': '0'} 2024-03-04T17:39:15.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b80f4c96089faf58-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.436 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.436 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.436 DEBUG:boto:path=/ppfahn-12/?prefix=. 2024-03-04T17:39:15.437 DEBUG:boto:auth_path=/ppfahn-12/?prefix=. 2024-03-04T17:39:15.437 DEBUG:boto:Method: GET 2024-03-04T17:39:15.437 DEBUG:boto:Path: /ppfahn-12/?prefix=. 2024-03-04T17:39:15.437 DEBUG:boto:Data: 2024-03-04T17:39:15.437 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.437 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.437 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.437 DEBUG:boto:Params: {} 2024-03-04T17:39:15.437 DEBUG:boto:Token: None 2024-03-04T17:39:15.437 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.437 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba35c11355599654-0065e60743-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.442 DEBUG:boto:b'ppfahn-12.1000false.2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.442 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.442 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.443 DEBUG:boto:path=/ppfahn-12/?prefix=. 2024-03-04T17:39:15.443 DEBUG:boto:auth_path=/ppfahn-12/?prefix=. 2024-03-04T17:39:15.443 DEBUG:boto:Method: GET 2024-03-04T17:39:15.443 DEBUG:boto:Path: /ppfahn-12/?prefix=. 2024-03-04T17:39:15.443 DEBUG:boto:Data: 2024-03-04T17:39:15.443 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.443 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.443 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.443 DEBUG:boto:Params: {} 2024-03-04T17:39:15.443 DEBUG:boto:Token: None 2024-03-04T17:39:15.443 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.444 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfa02df9fbd79db9-0065e60743-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.448 DEBUG:boto:b'ppfahn-12.1000false.2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:15.449 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:15.449 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:15.449 DEBUG:boto:path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:15.449 DEBUG:boto:auth_path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:15.449 DEBUG:boto:Method: GET 2024-03-04T17:39:15.449 DEBUG:boto:Path: /ppfahn-12/..?versionId=null 2024-03-04T17:39:15.449 DEBUG:boto:Data: 2024-03-04T17:39:15.450 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.450 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.450 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.450 DEBUG:boto:Params: {} 2024-03-04T17:39:15.450 DEBUG:boto:Token: None 2024-03-04T17:39:15.450 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/..?versionId=null 2024-03-04T17:39:15.450 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oVUrkjs3p9qHjZWtTi9n1uu3U18= 2024-03-04T17:39:15.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oVUrkjs3p9qHjZWtTi9n1uu3U18=', 'Content-Length': '0'} 2024-03-04T17:39:15.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca1367f757d313cf-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.453 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:15.453 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:15.453 DEBUG:boto:path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:15.454 DEBUG:boto:auth_path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:15.454 DEBUG:boto:Method: GET 2024-03-04T17:39:15.454 DEBUG:boto:Path: /ppfahn-12/..?versionId=null 2024-03-04T17:39:15.454 DEBUG:boto:Data: 2024-03-04T17:39:15.454 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.454 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.454 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.454 DEBUG:boto:Params: {} 2024-03-04T17:39:15.454 DEBUG:boto:Token: None 2024-03-04T17:39:15.454 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/..?versionId=null 2024-03-04T17:39:15.455 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oVUrkjs3p9qHjZWtTi9n1uu3U18= 2024-03-04T17:39:15.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oVUrkjs3p9qHjZWtTi9n1uu3U18=', 'Content-Length': '0'} 2024-03-04T17:39:15.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9ac2be0a63c5237-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.500 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:15.500 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:15.500 DEBUG:boto:path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:15.500 DEBUG:boto:auth_path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:15.501 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.501 DEBUG:boto:Path: /ppfahn-12/..?versionId=null 2024-03-04T17:39:15.501 DEBUG:boto:Data: 2024-03-04T17:39:15.501 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.501 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.501 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.501 DEBUG:boto:Params: {} 2024-03-04T17:39:15.501 DEBUG:boto:Token: None 2024-03-04T17:39:15.501 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/..?versionId=null 2024-03-04T17:39:15.501 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:I017GcNYFYaxMQw/pjqu+GmZOTw= 2024-03-04T17:39:15.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:I017GcNYFYaxMQw/pjqu+GmZOTw=', 'Content-Length': '0'} 2024-03-04T17:39:15.504 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000540184f98f7d7962-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.505 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:15.505 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:15.505 DEBUG:boto:path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:15.505 DEBUG:boto:auth_path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:15.505 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.505 DEBUG:boto:Path: /ppfahn-12/..?versionId=null 2024-03-04T17:39:15.505 DEBUG:boto:Data: 2024-03-04T17:39:15.505 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.505 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.505 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.506 DEBUG:boto:Params: {} 2024-03-04T17:39:15.506 DEBUG:boto:Token: None 2024-03-04T17:39:15.506 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/..?versionId=null 2024-03-04T17:39:15.506 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:I017GcNYFYaxMQw/pjqu+GmZOTw= 2024-03-04T17:39:15.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:I017GcNYFYaxMQw/pjqu+GmZOTw=', 'Content-Length': '0'} 2024-03-04T17:39:15.508 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028a163aa54a77a1c-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:15.509 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:15.509 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:15.509 DEBUG:boto:path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:15.509 DEBUG:boto:auth_path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:15.509 DEBUG:boto:Method: GET 2024-03-04T17:39:15.509 DEBUG:boto:Path: /ppfahn-12/..?versionId=null 2024-03-04T17:39:15.510 DEBUG:boto:Data: 2024-03-04T17:39:15.510 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.510 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.510 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.510 DEBUG:boto:Params: {} 2024-03-04T17:39:15.510 DEBUG:boto:Token: None 2024-03-04T17:39:15.510 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/..?versionId=null 2024-03-04T17:39:15.510 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oVUrkjs3p9qHjZWtTi9n1uu3U18= 2024-03-04T17:39:15.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oVUrkjs3p9qHjZWtTi9n1uu3U18=', 'Content-Length': '0'} 2024-03-04T17:39:15.513 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddddbeb71ab47b9c-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.556 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:15.556 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:15.556 DEBUG:boto:path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:15.556 DEBUG:boto:auth_path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:15.556 DEBUG:boto:Method: GET 2024-03-04T17:39:15.557 DEBUG:boto:Path: /ppfahn-12/..?versionId=null 2024-03-04T17:39:15.557 DEBUG:boto:Data: 2024-03-04T17:39:15.557 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.557 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.557 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.557 DEBUG:boto:Params: {} 2024-03-04T17:39:15.557 DEBUG:boto:Token: None 2024-03-04T17:39:15.557 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/..?versionId=null 2024-03-04T17:39:15.557 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oVUrkjs3p9qHjZWtTi9n1uu3U18= 2024-03-04T17:39:15.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oVUrkjs3p9qHjZWtTi9n1uu3U18=', 'Content-Length': '0'} 2024-03-04T17:39:15.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008059bb8684e85793-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.561 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.561 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.561 DEBUG:boto:path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:15.561 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:15.561 DEBUG:boto:Method: GET 2024-03-04T17:39:15.562 DEBUG:boto:Path: /ppfahn-12/?prefix=.. 2024-03-04T17:39:15.562 DEBUG:boto:Data: 2024-03-04T17:39:15.562 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.562 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.562 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.562 DEBUG:boto:Params: {} 2024-03-04T17:39:15.562 DEBUG:boto:Token: None 2024-03-04T17:39:15.562 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.562 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adddc6acbec872e4-0065e60743-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.567 DEBUG:boto:b'ppfahn-12..1000false..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.567 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.568 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.568 DEBUG:boto:path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:15.568 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:15.568 DEBUG:boto:Method: GET 2024-03-04T17:39:15.568 DEBUG:boto:Path: /ppfahn-12/?prefix=.. 2024-03-04T17:39:15.568 DEBUG:boto:Data: 2024-03-04T17:39:15.568 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.568 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.568 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.568 DEBUG:boto:Params: {} 2024-03-04T17:39:15.568 DEBUG:boto:Token: None 2024-03-04T17:39:15.569 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.569 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005629aa623a3d06f3-0065e60743-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.573 DEBUG:boto:b'ppfahn-12..1000false..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.573 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:15.573 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:15.574 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.574 DEBUG:boto:Path: /ppfahn-12/.. 2024-03-04T17:39:15.574 DEBUG:boto:Data: 2024-03-04T17:39:15.574 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.574 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.574 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.574 DEBUG:boto:Params: {} 2024-03-04T17:39:15.574 DEBUG:boto:Token: None 2024-03-04T17:39:15.574 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.. 2024-03-04T17:39:15.574 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tDItoKKH2rrS/cGKUMD/dV3ZsdQ= 2024-03-04T17:39:15.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tDItoKKH2rrS/cGKUMD/dV3ZsdQ=', 'Content-Length': '0'} 2024-03-04T17:39:15.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3290d25cb9b03f2-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.577 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:15.577 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:15.577 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.577 DEBUG:boto:Path: /ppfahn-12/.. 2024-03-04T17:39:15.578 DEBUG:boto:Data: 2024-03-04T17:39:15.578 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.578 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.578 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.578 DEBUG:boto:Params: {} 2024-03-04T17:39:15.578 DEBUG:boto:Token: None 2024-03-04T17:39:15.578 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.. 2024-03-04T17:39:15.578 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tDItoKKH2rrS/cGKUMD/dV3ZsdQ= 2024-03-04T17:39:15.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tDItoKKH2rrS/cGKUMD/dV3ZsdQ=', 'Content-Length': '0'} 2024-03-04T17:39:15.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eee20d863a4741e6-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:15.581 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:15.581 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:15.581 DEBUG:boto:Method: GET 2024-03-04T17:39:15.582 DEBUG:boto:Path: /ppfahn-12/.. 2024-03-04T17:39:15.582 DEBUG:boto:Data: 2024-03-04T17:39:15.582 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.582 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.582 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.582 DEBUG:boto:Params: {} 2024-03-04T17:39:15.582 DEBUG:boto:Token: None 2024-03-04T17:39:15.582 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.. 2024-03-04T17:39:15.582 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsjRt5ndJbhW3ERXQTzBFE9WLSk= 2024-03-04T17:39:15.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsjRt5ndJbhW3ERXQTzBFE9WLSk=', 'Content-Length': '0'} 2024-03-04T17:39:15.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd0fd3c5dff4ec10-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.628 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:15.628 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:15.628 DEBUG:boto:Method: GET 2024-03-04T17:39:15.629 DEBUG:boto:Path: /ppfahn-12/.. 2024-03-04T17:39:15.629 DEBUG:boto:Data: 2024-03-04T17:39:15.629 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.629 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.629 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.629 DEBUG:boto:Params: {} 2024-03-04T17:39:15.629 DEBUG:boto:Token: None 2024-03-04T17:39:15.629 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.. 2024-03-04T17:39:15.629 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsjRt5ndJbhW3ERXQTzBFE9WLSk= 2024-03-04T17:39:15.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsjRt5ndJbhW3ERXQTzBFE9WLSk=', 'Content-Length': '0'} 2024-03-04T17:39:15.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000558074f7f2fe15fc-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.633 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.633 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.633 DEBUG:boto:path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:15.633 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:15.633 DEBUG:boto:Method: GET 2024-03-04T17:39:15.633 DEBUG:boto:Path: /ppfahn-12/?prefix=.. 2024-03-04T17:39:15.633 DEBUG:boto:Data: 2024-03-04T17:39:15.633 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.633 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.634 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.634 DEBUG:boto:Params: {} 2024-03-04T17:39:15.634 DEBUG:boto:Token: None 2024-03-04T17:39:15.634 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.634 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000838a4dc6cbc96158-0065e60743-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.638 DEBUG:boto:b'ppfahn-12..1000false..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.639 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.639 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.639 DEBUG:boto:path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:15.639 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:15.639 DEBUG:boto:Method: GET 2024-03-04T17:39:15.639 DEBUG:boto:Path: /ppfahn-12/?prefix=.. 2024-03-04T17:39:15.639 DEBUG:boto:Data: 2024-03-04T17:39:15.640 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.640 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.640 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.640 DEBUG:boto:Params: {} 2024-03-04T17:39:15.640 DEBUG:boto:Token: None 2024-03-04T17:39:15.640 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.640 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a82aec5293594108-0065e60743-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.645 DEBUG:boto:b'ppfahn-12..1000false..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:15.646 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:15.646 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:15.646 DEBUG:boto:path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:15.646 DEBUG:boto:auth_path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:15.646 DEBUG:boto:Method: GET 2024-03-04T17:39:15.646 DEBUG:boto:Path: /ppfahn-12/...?versionId=null 2024-03-04T17:39:15.646 DEBUG:boto:Data: 2024-03-04T17:39:15.646 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.646 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.646 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.647 DEBUG:boto:Params: {} 2024-03-04T17:39:15.647 DEBUG:boto:Token: None 2024-03-04T17:39:15.647 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/...?versionId=null 2024-03-04T17:39:15.647 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4UEcGwD0bQvUXT42utl1QnhBAg0= 2024-03-04T17:39:15.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4UEcGwD0bQvUXT42utl1QnhBAg0=', 'Content-Length': '0'} 2024-03-04T17:39:15.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082d70a7fdc8a991a-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.650 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:15.650 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:15.650 DEBUG:boto:path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:15.650 DEBUG:boto:auth_path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:15.650 DEBUG:boto:Method: GET 2024-03-04T17:39:15.650 DEBUG:boto:Path: /ppfahn-12/...?versionId=null 2024-03-04T17:39:15.651 DEBUG:boto:Data: 2024-03-04T17:39:15.651 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.651 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.651 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.651 DEBUG:boto:Params: {} 2024-03-04T17:39:15.651 DEBUG:boto:Token: None 2024-03-04T17:39:15.651 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/...?versionId=null 2024-03-04T17:39:15.651 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4UEcGwD0bQvUXT42utl1QnhBAg0= 2024-03-04T17:39:15.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4UEcGwD0bQvUXT42utl1QnhBAg0=', 'Content-Length': '0'} 2024-03-04T17:39:15.654 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006746beeb0e27231b-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.654 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:15.655 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:15.655 DEBUG:boto:path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:15.655 DEBUG:boto:auth_path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:15.655 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.655 DEBUG:boto:Path: /ppfahn-12/...?versionId=null 2024-03-04T17:39:15.655 DEBUG:boto:Data: 2024-03-04T17:39:15.655 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.655 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.655 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.655 DEBUG:boto:Params: {} 2024-03-04T17:39:15.656 DEBUG:boto:Token: None 2024-03-04T17:39:15.656 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/...?versionId=null 2024-03-04T17:39:15.656 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fFeuO2GOy52XYiWN1+j3gZpu5Dk= 2024-03-04T17:39:15.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fFeuO2GOy52XYiWN1+j3gZpu5Dk=', 'Content-Length': '0'} 2024-03-04T17:39:15.659 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062b3ae5cfa2bab5e-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.659 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:15.659 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:15.659 DEBUG:boto:path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:15.659 DEBUG:boto:auth_path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:15.659 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.659 DEBUG:boto:Path: /ppfahn-12/...?versionId=null 2024-03-04T17:39:15.660 DEBUG:boto:Data: 2024-03-04T17:39:15.660 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.660 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.660 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.660 DEBUG:boto:Params: {} 2024-03-04T17:39:15.660 DEBUG:boto:Token: None 2024-03-04T17:39:15.660 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/...?versionId=null 2024-03-04T17:39:15.660 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fFeuO2GOy52XYiWN1+j3gZpu5Dk= 2024-03-04T17:39:15.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fFeuO2GOy52XYiWN1+j3gZpu5Dk=', 'Content-Length': '0'} 2024-03-04T17:39:15.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffce53cd868a7c13-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:15.663 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:15.663 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:15.663 DEBUG:boto:path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:15.664 DEBUG:boto:auth_path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:15.664 DEBUG:boto:Method: GET 2024-03-04T17:39:15.664 DEBUG:boto:Path: /ppfahn-12/...?versionId=null 2024-03-04T17:39:15.664 DEBUG:boto:Data: 2024-03-04T17:39:15.664 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.664 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.664 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.664 DEBUG:boto:Params: {} 2024-03-04T17:39:15.664 DEBUG:boto:Token: None 2024-03-04T17:39:15.665 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/...?versionId=null 2024-03-04T17:39:15.665 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4UEcGwD0bQvUXT42utl1QnhBAg0= 2024-03-04T17:39:15.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4UEcGwD0bQvUXT42utl1QnhBAg0=', 'Content-Length': '0'} 2024-03-04T17:39:15.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f0b55c8462dcd8b-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.708 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:15.708 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:15.708 DEBUG:boto:path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:15.709 DEBUG:boto:auth_path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:15.709 DEBUG:boto:Method: GET 2024-03-04T17:39:15.709 DEBUG:boto:Path: /ppfahn-12/...?versionId=null 2024-03-04T17:39:15.709 DEBUG:boto:Data: 2024-03-04T17:39:15.709 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.709 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.709 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.709 DEBUG:boto:Params: {} 2024-03-04T17:39:15.709 DEBUG:boto:Token: None 2024-03-04T17:39:15.709 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/...?versionId=null 2024-03-04T17:39:15.710 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4UEcGwD0bQvUXT42utl1QnhBAg0= 2024-03-04T17:39:15.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4UEcGwD0bQvUXT42utl1QnhBAg0=', 'Content-Length': '0'} 2024-03-04T17:39:15.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000557beac69ef6cf28-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.713 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.713 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.713 DEBUG:boto:path=/ppfahn-12/?prefix=... 2024-03-04T17:39:15.713 DEBUG:boto:auth_path=/ppfahn-12/?prefix=... 2024-03-04T17:39:15.713 DEBUG:boto:Method: GET 2024-03-04T17:39:15.713 DEBUG:boto:Path: /ppfahn-12/?prefix=... 2024-03-04T17:39:15.714 DEBUG:boto:Data: 2024-03-04T17:39:15.714 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.714 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.714 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.714 DEBUG:boto:Params: {} 2024-03-04T17:39:15.714 DEBUG:boto:Token: None 2024-03-04T17:39:15.714 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.714 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e0021f35d6a1d5e-0065e60743-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.719 DEBUG:boto:b'ppfahn-12...1000false...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.719 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.719 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.720 DEBUG:boto:path=/ppfahn-12/?prefix=... 2024-03-04T17:39:15.720 DEBUG:boto:auth_path=/ppfahn-12/?prefix=... 2024-03-04T17:39:15.720 DEBUG:boto:Method: GET 2024-03-04T17:39:15.720 DEBUG:boto:Path: /ppfahn-12/?prefix=... 2024-03-04T17:39:15.720 DEBUG:boto:Data: 2024-03-04T17:39:15.720 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.720 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.720 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.720 DEBUG:boto:Params: {} 2024-03-04T17:39:15.721 DEBUG:boto:Token: None 2024-03-04T17:39:15.721 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.721 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c94c2a7c29d5010-0065e60743-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.726 DEBUG:boto:b'ppfahn-12...1000false...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.726 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:15.726 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:15.726 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.726 DEBUG:boto:Path: /ppfahn-12/... 2024-03-04T17:39:15.727 DEBUG:boto:Data: 2024-03-04T17:39:15.727 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.727 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.727 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.727 DEBUG:boto:Params: {} 2024-03-04T17:39:15.727 DEBUG:boto:Token: None 2024-03-04T17:39:15.727 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/... 2024-03-04T17:39:15.727 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p9pczFT8FXSh0KHEaAb1dU8fYxc= 2024-03-04T17:39:15.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p9pczFT8FXSh0KHEaAb1dU8fYxc=', 'Content-Length': '0'} 2024-03-04T17:39:15.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccf0129b96ee5626-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.730 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:15.730 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:15.730 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.730 DEBUG:boto:Path: /ppfahn-12/... 2024-03-04T17:39:15.730 DEBUG:boto:Data: 2024-03-04T17:39:15.730 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.731 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.731 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.731 DEBUG:boto:Params: {} 2024-03-04T17:39:15.731 DEBUG:boto:Token: None 2024-03-04T17:39:15.731 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/... 2024-03-04T17:39:15.731 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p9pczFT8FXSh0KHEaAb1dU8fYxc= 2024-03-04T17:39:15.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p9pczFT8FXSh0KHEaAb1dU8fYxc=', 'Content-Length': '0'} 2024-03-04T17:39:15.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d0454ac1a96255f-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:15.734 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:15.734 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:15.734 DEBUG:boto:Method: GET 2024-03-04T17:39:15.734 DEBUG:boto:Path: /ppfahn-12/... 2024-03-04T17:39:15.734 DEBUG:boto:Data: 2024-03-04T17:39:15.735 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.735 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.735 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.735 DEBUG:boto:Params: {} 2024-03-04T17:39:15.735 DEBUG:boto:Token: None 2024-03-04T17:39:15.735 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/... 2024-03-04T17:39:15.735 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:B8xw/Z3oY5vE6gSHiKB665iXKAQ= 2024-03-04T17:39:15.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:B8xw/Z3oY5vE6gSHiKB665iXKAQ=', 'Content-Length': '0'} 2024-03-04T17:39:15.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008788077c4928f467-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.780 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:15.780 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:15.780 DEBUG:boto:Method: GET 2024-03-04T17:39:15.780 DEBUG:boto:Path: /ppfahn-12/... 2024-03-04T17:39:15.781 DEBUG:boto:Data: 2024-03-04T17:39:15.781 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.781 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.781 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.781 DEBUG:boto:Params: {} 2024-03-04T17:39:15.781 DEBUG:boto:Token: None 2024-03-04T17:39:15.781 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/... 2024-03-04T17:39:15.781 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:B8xw/Z3oY5vE6gSHiKB665iXKAQ= 2024-03-04T17:39:15.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:B8xw/Z3oY5vE6gSHiKB665iXKAQ=', 'Content-Length': '0'} 2024-03-04T17:39:15.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000565010b1ebb14895-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.785 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.785 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.785 DEBUG:boto:path=/ppfahn-12/?prefix=... 2024-03-04T17:39:15.785 DEBUG:boto:auth_path=/ppfahn-12/?prefix=... 2024-03-04T17:39:15.785 DEBUG:boto:Method: GET 2024-03-04T17:39:15.785 DEBUG:boto:Path: /ppfahn-12/?prefix=... 2024-03-04T17:39:15.786 DEBUG:boto:Data: 2024-03-04T17:39:15.786 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.786 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.786 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.786 DEBUG:boto:Params: {} 2024-03-04T17:39:15.786 DEBUG:boto:Token: None 2024-03-04T17:39:15.786 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.786 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3d8e90f7bfa193d-0065e60743-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.791 DEBUG:boto:b'ppfahn-12...1000false...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.791 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.792 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.792 DEBUG:boto:path=/ppfahn-12/?prefix=... 2024-03-04T17:39:15.792 DEBUG:boto:auth_path=/ppfahn-12/?prefix=... 2024-03-04T17:39:15.792 DEBUG:boto:Method: GET 2024-03-04T17:39:15.792 DEBUG:boto:Path: /ppfahn-12/?prefix=... 2024-03-04T17:39:15.792 DEBUG:boto:Data: 2024-03-04T17:39:15.792 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.792 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.792 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.792 DEBUG:boto:Params: {} 2024-03-04T17:39:15.793 DEBUG:boto:Token: None 2024-03-04T17:39:15.793 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.793 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000846378c52b900b48-0065e60743-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.840 DEBUG:boto:b'ppfahn-12...1000false...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:15.841 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:15.841 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:15.841 DEBUG:boto:path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.841 DEBUG:boto:auth_path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.841 DEBUG:boto:Method: GET 2024-03-04T17:39:15.841 DEBUG:boto:Path: /ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.841 DEBUG:boto:Data: 2024-03-04T17:39:15.841 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.841 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.841 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.841 DEBUG:boto:Params: {} 2024-03-04T17:39:15.842 DEBUG:boto:Token: None 2024-03-04T17:39:15.842 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.842 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:k194uYfEWyqQ4z9ImRqnxTSpV20= 2024-03-04T17:39:15.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:k194uYfEWyqQ4z9ImRqnxTSpV20=', 'Content-Length': '0'} 2024-03-04T17:39:15.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000716149108e844ccf-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.845 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:15.845 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:15.845 DEBUG:boto:path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.845 DEBUG:boto:auth_path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.845 DEBUG:boto:Method: GET 2024-03-04T17:39:15.846 DEBUG:boto:Path: /ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.846 DEBUG:boto:Data: 2024-03-04T17:39:15.846 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.846 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.846 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.846 DEBUG:boto:Params: {} 2024-03-04T17:39:15.846 DEBUG:boto:Token: None 2024-03-04T17:39:15.846 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.846 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:k194uYfEWyqQ4z9ImRqnxTSpV20= 2024-03-04T17:39:15.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:k194uYfEWyqQ4z9ImRqnxTSpV20=', 'Content-Length': '0'} 2024-03-04T17:39:15.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d67c1147fcb8434e-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.850 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:15.850 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:15.850 DEBUG:boto:path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.850 DEBUG:boto:auth_path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.850 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.850 DEBUG:boto:Path: /ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.850 DEBUG:boto:Data: 2024-03-04T17:39:15.851 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.851 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.851 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.851 DEBUG:boto:Params: {} 2024-03-04T17:39:15.851 DEBUG:boto:Token: None 2024-03-04T17:39:15.851 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.851 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Po0mwXaX4sh4Sdu6b8VrU7509tI= 2024-03-04T17:39:15.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Po0mwXaX4sh4Sdu6b8VrU7509tI=', 'Content-Length': '0'} 2024-03-04T17:39:15.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000634701ea088cfa-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.854 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:15.854 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:15.854 DEBUG:boto:path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.854 DEBUG:boto:auth_path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.855 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.855 DEBUG:boto:Path: /ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.855 DEBUG:boto:Data: 2024-03-04T17:39:15.855 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.855 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.855 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.855 DEBUG:boto:Params: {} 2024-03-04T17:39:15.855 DEBUG:boto:Token: None 2024-03-04T17:39:15.855 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.856 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Po0mwXaX4sh4Sdu6b8VrU7509tI= 2024-03-04T17:39:15.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Po0mwXaX4sh4Sdu6b8VrU7509tI=', 'Content-Length': '0'} 2024-03-04T17:39:15.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f661d7f349cd527-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:15.859 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:15.859 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:15.859 DEBUG:boto:path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.860 DEBUG:boto:auth_path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.860 DEBUG:boto:Method: GET 2024-03-04T17:39:15.860 DEBUG:boto:Path: /ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.860 DEBUG:boto:Data: 2024-03-04T17:39:15.860 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.860 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.860 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.860 DEBUG:boto:Params: {} 2024-03-04T17:39:15.860 DEBUG:boto:Token: None 2024-03-04T17:39:15.861 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.861 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:k194uYfEWyqQ4z9ImRqnxTSpV20= 2024-03-04T17:39:15.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:k194uYfEWyqQ4z9ImRqnxTSpV20=', 'Content-Length': '0'} 2024-03-04T17:39:15.864 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d822654e93cbe260-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.904 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:15.904 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:15.904 DEBUG:boto:path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.905 DEBUG:boto:auth_path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.905 DEBUG:boto:Method: GET 2024-03-04T17:39:15.905 DEBUG:boto:Path: /ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.905 DEBUG:boto:Data: 2024-03-04T17:39:15.905 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.905 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.905 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.905 DEBUG:boto:Params: {} 2024-03-04T17:39:15.905 DEBUG:boto:Token: None 2024-03-04T17:39:15.906 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.o?versionId=null 2024-03-04T17:39:15.906 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:k194uYfEWyqQ4z9ImRqnxTSpV20= 2024-03-04T17:39:15.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:k194uYfEWyqQ4z9ImRqnxTSpV20=', 'Content-Length': '0'} 2024-03-04T17:39:15.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081e96f1e81590b1d-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.909 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.909 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.909 DEBUG:boto:path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:15.909 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:15.909 DEBUG:boto:Method: GET 2024-03-04T17:39:15.909 DEBUG:boto:Path: /ppfahn-12/?prefix=.o 2024-03-04T17:39:15.909 DEBUG:boto:Data: 2024-03-04T17:39:15.910 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.910 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.910 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.910 DEBUG:boto:Params: {} 2024-03-04T17:39:15.910 DEBUG:boto:Token: None 2024-03-04T17:39:15.910 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.910 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c66cc97c06900ddd-0065e60743-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.915 DEBUG:boto:b'ppfahn-12.o1000false.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.915 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.916 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.916 DEBUG:boto:path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:15.916 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:15.916 DEBUG:boto:Method: GET 2024-03-04T17:39:15.916 DEBUG:boto:Path: /ppfahn-12/?prefix=.o 2024-03-04T17:39:15.916 DEBUG:boto:Data: 2024-03-04T17:39:15.916 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.916 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.916 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.916 DEBUG:boto:Params: {} 2024-03-04T17:39:15.916 DEBUG:boto:Token: None 2024-03-04T17:39:15.917 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.917 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008be7937571681b83-0065e60743-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.964 DEBUG:boto:b'ppfahn-12.o1000false.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:15.965 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:15.965 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:15.965 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.965 DEBUG:boto:Path: /ppfahn-12/.o 2024-03-04T17:39:15.965 DEBUG:boto:Data: 2024-03-04T17:39:15.965 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.965 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.965 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.965 DEBUG:boto:Params: {} 2024-03-04T17:39:15.966 DEBUG:boto:Token: None 2024-03-04T17:39:15.966 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.o 2024-03-04T17:39:15.966 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2XJmMOpqfciECBumEJG5hPkJyH0= 2024-03-04T17:39:15.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2XJmMOpqfciECBumEJG5hPkJyH0=', 'Content-Length': '0'} 2024-03-04T17:39:15.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fee31f3ee6a81ccb-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.969 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:15.969 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:15.969 DEBUG:boto:Method: HEAD 2024-03-04T17:39:15.969 DEBUG:boto:Path: /ppfahn-12/.o 2024-03-04T17:39:15.969 DEBUG:boto:Data: 2024-03-04T17:39:15.969 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.969 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.970 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.970 DEBUG:boto:Params: {} 2024-03-04T17:39:15.970 DEBUG:boto:Token: None 2024-03-04T17:39:15.970 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.o 2024-03-04T17:39:15.970 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2XJmMOpqfciECBumEJG5hPkJyH0= 2024-03-04T17:39:15.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2XJmMOpqfciECBumEJG5hPkJyH0=', 'Content-Length': '0'} 2024-03-04T17:39:15.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007da2a4088af5ceb7-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:15.973 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:15.973 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:15.973 DEBUG:boto:Method: GET 2024-03-04T17:39:15.973 DEBUG:boto:Path: /ppfahn-12/.o 2024-03-04T17:39:15.973 DEBUG:boto:Data: 2024-03-04T17:39:15.973 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.973 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.973 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.974 DEBUG:boto:Params: {} 2024-03-04T17:39:15.974 DEBUG:boto:Token: None 2024-03-04T17:39:15.974 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.o 2024-03-04T17:39:15.974 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/MNOakjV3iOixuW1DRmvXdIWqOQ= 2024-03-04T17:39:15.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/MNOakjV3iOixuW1DRmvXdIWqOQ=', 'Content-Length': '0'} 2024-03-04T17:39:15.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcc5c36ec4302080-0065e60743-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.977 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:15.977 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:15.977 DEBUG:boto:Method: GET 2024-03-04T17:39:15.977 DEBUG:boto:Path: /ppfahn-12/.o 2024-03-04T17:39:15.977 DEBUG:boto:Data: 2024-03-04T17:39:15.977 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.977 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:15.977 DEBUG:boto:Port: 8001 2024-03-04T17:39:15.977 DEBUG:boto:Params: {} 2024-03-04T17:39:15.978 DEBUG:boto:Token: None 2024-03-04T17:39:15.978 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/.o 2024-03-04T17:39:15.978 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/MNOakjV3iOixuW1DRmvXdIWqOQ= 2024-03-04T17:39:15.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/MNOakjV3iOixuW1DRmvXdIWqOQ=', 'Content-Length': '0'} 2024-03-04T17:39:15.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090ec6538dc23cdac-0065e60743-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:15.981 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:15.981 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:15.981 DEBUG:boto:path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:15.981 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:15.981 DEBUG:boto:Method: GET 2024-03-04T17:39:15.981 DEBUG:boto:Path: /ppfahn-12/?prefix=.o 2024-03-04T17:39:15.981 DEBUG:boto:Data: 2024-03-04T17:39:15.982 DEBUG:boto:Headers: {} 2024-03-04T17:39:15.982 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:15.982 DEBUG:boto:Port: 8000 2024-03-04T17:39:15.982 DEBUG:boto:Params: {} 2024-03-04T17:39:15.982 DEBUG:boto:Token: None 2024-03-04T17:39:15.982 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:15 GMT /ppfahn-12/ 2024-03-04T17:39:15.982 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI= 2024-03-04T17:39:15.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVcZisJ6RQIClE/O6WUiFilY8OI=', 'Content-Length': '0'} 2024-03-04T17:39:15.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4e48d30de53fbf7-0065e60743-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.028 DEBUG:boto:b'ppfahn-12.o1000false.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.029 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.029 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.029 DEBUG:boto:path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:16.029 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:16.029 DEBUG:boto:Method: GET 2024-03-04T17:39:16.029 DEBUG:boto:Path: /ppfahn-12/?prefix=.o 2024-03-04T17:39:16.029 DEBUG:boto:Data: 2024-03-04T17:39:16.030 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.030 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.030 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.030 DEBUG:boto:Params: {} 2024-03-04T17:39:16.030 DEBUG:boto:Token: None 2024-03-04T17:39:16.030 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.030 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d30cad21fe45dacc-0065e60744-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.035 DEBUG:boto:b'ppfahn-12.o1000false.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:16.035 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:16.035 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:16.036 DEBUG:boto:path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.036 DEBUG:boto:auth_path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.036 DEBUG:boto:Method: GET 2024-03-04T17:39:16.036 DEBUG:boto:Path: /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.036 DEBUG:boto:Data: 2024-03-04T17:39:16.036 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.036 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.036 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.036 DEBUG:boto:Params: {} 2024-03-04T17:39:16.036 DEBUG:boto:Token: None 2024-03-04T17:39:16.036 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.037 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ISqHhYJ6MAHMbllcaqbXzCaXpUg= 2024-03-04T17:39:16.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ISqHhYJ6MAHMbllcaqbXzCaXpUg=', 'Content-Length': '0'} 2024-03-04T17:39:16.039 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e178942de0b1d70f-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.040 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:16.040 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:16.040 DEBUG:boto:path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.040 DEBUG:boto:auth_path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.040 DEBUG:boto:Method: GET 2024-03-04T17:39:16.040 DEBUG:boto:Path: /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.040 DEBUG:boto:Data: 2024-03-04T17:39:16.040 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.041 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.041 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.041 DEBUG:boto:Params: {} 2024-03-04T17:39:16.041 DEBUG:boto:Token: None 2024-03-04T17:39:16.041 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.041 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ISqHhYJ6MAHMbllcaqbXzCaXpUg= 2024-03-04T17:39:16.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ISqHhYJ6MAHMbllcaqbXzCaXpUg=', 'Content-Length': '0'} 2024-03-04T17:39:16.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb7eab438443d985-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.084 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:16.084 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:16.084 DEBUG:boto:path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.084 DEBUG:boto:auth_path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.084 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.084 DEBUG:boto:Path: /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.085 DEBUG:boto:Data: 2024-03-04T17:39:16.085 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.085 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.085 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.085 DEBUG:boto:Params: {} 2024-03-04T17:39:16.085 DEBUG:boto:Token: None 2024-03-04T17:39:16.085 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.085 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ICJXzOnsmwqBrKlxM8oEVG0k3+Q= 2024-03-04T17:39:16.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ICJXzOnsmwqBrKlxM8oEVG0k3+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0a4377df7e0fb55-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.088 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:16.088 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:16.089 DEBUG:boto:path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.089 DEBUG:boto:auth_path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.089 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.089 DEBUG:boto:Path: /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.089 DEBUG:boto:Data: 2024-03-04T17:39:16.089 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.089 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.089 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.089 DEBUG:boto:Params: {} 2024-03-04T17:39:16.089 DEBUG:boto:Token: None 2024-03-04T17:39:16.089 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.090 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ICJXzOnsmwqBrKlxM8oEVG0k3+Q= 2024-03-04T17:39:16.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ICJXzOnsmwqBrKlxM8oEVG0k3+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b42663a538194451-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:16.093 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:16.093 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:16.093 DEBUG:boto:path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.093 DEBUG:boto:auth_path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.093 DEBUG:boto:Method: GET 2024-03-04T17:39:16.093 DEBUG:boto:Path: /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.093 DEBUG:boto:Data: 2024-03-04T17:39:16.093 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.094 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.094 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.094 DEBUG:boto:Params: {} 2024-03-04T17:39:16.094 DEBUG:boto:Token: None 2024-03-04T17:39:16.094 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.094 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ISqHhYJ6MAHMbllcaqbXzCaXpUg= 2024-03-04T17:39:16.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ISqHhYJ6MAHMbllcaqbXzCaXpUg=', 'Content-Length': '0'} 2024-03-04T17:39:16.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bcf5410a1f35e17-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.097 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:16.097 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:16.097 DEBUG:boto:path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.098 DEBUG:boto:auth_path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.098 DEBUG:boto:Method: GET 2024-03-04T17:39:16.098 DEBUG:boto:Path: /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.098 DEBUG:boto:Data: 2024-03-04T17:39:16.098 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.098 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.098 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.098 DEBUG:boto:Params: {} 2024-03-04T17:39:16.098 DEBUG:boto:Token: None 2024-03-04T17:39:16.098 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:16.098 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ISqHhYJ6MAHMbllcaqbXzCaXpUg= 2024-03-04T17:39:16.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ISqHhYJ6MAHMbllcaqbXzCaXpUg=', 'Content-Length': '0'} 2024-03-04T17:39:16.102 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034bff4c379bd846a-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.102 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.102 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.102 DEBUG:boto:path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:16.102 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:16.102 DEBUG:boto:Method: GET 2024-03-04T17:39:16.103 DEBUG:boto:Path: /ppfahn-12/?prefix=.o. 2024-03-04T17:39:16.103 DEBUG:boto:Data: 2024-03-04T17:39:16.103 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.103 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.103 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.103 DEBUG:boto:Params: {} 2024-03-04T17:39:16.103 DEBUG:boto:Token: None 2024-03-04T17:39:16.103 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.103 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006e9c05c9d0138fa-0065e60744-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.148 DEBUG:boto:b'ppfahn-12.o.1000false.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.148 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.148 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.149 DEBUG:boto:path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:16.149 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:16.149 DEBUG:boto:Method: GET 2024-03-04T17:39:16.149 DEBUG:boto:Path: /ppfahn-12/?prefix=.o. 2024-03-04T17:39:16.149 DEBUG:boto:Data: 2024-03-04T17:39:16.149 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.149 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.149 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.149 DEBUG:boto:Params: {} 2024-03-04T17:39:16.149 DEBUG:boto:Token: None 2024-03-04T17:39:16.150 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.150 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004649897bf29e872-0065e60744-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.154 DEBUG:boto:b'ppfahn-12.o.1000false.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.154 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:16.154 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:16.154 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.154 DEBUG:boto:Path: /ppfahn-12/.o. 2024-03-04T17:39:16.155 DEBUG:boto:Data: 2024-03-04T17:39:16.155 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.155 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.155 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.155 DEBUG:boto:Params: {} 2024-03-04T17:39:16.155 DEBUG:boto:Token: None 2024-03-04T17:39:16.155 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/.o. 2024-03-04T17:39:16.155 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wzx3NHhxbqVWWy2t3c84dFX2SIo= 2024-03-04T17:39:16.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wzx3NHhxbqVWWy2t3c84dFX2SIo=', 'Content-Length': '0'} 2024-03-04T17:39:16.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bac8d680c3edf29-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.158 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:16.158 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:16.158 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.158 DEBUG:boto:Path: /ppfahn-12/.o. 2024-03-04T17:39:16.159 DEBUG:boto:Data: 2024-03-04T17:39:16.159 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.159 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.159 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.159 DEBUG:boto:Params: {} 2024-03-04T17:39:16.159 DEBUG:boto:Token: None 2024-03-04T17:39:16.159 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/.o. 2024-03-04T17:39:16.159 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wzx3NHhxbqVWWy2t3c84dFX2SIo= 2024-03-04T17:39:16.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wzx3NHhxbqVWWy2t3c84dFX2SIo=', 'Content-Length': '0'} 2024-03-04T17:39:16.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a0424aa03f7f040-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:16.163 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:16.163 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:16.163 DEBUG:boto:Method: GET 2024-03-04T17:39:16.163 DEBUG:boto:Path: /ppfahn-12/.o. 2024-03-04T17:39:16.163 DEBUG:boto:Data: 2024-03-04T17:39:16.163 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.163 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.163 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.163 DEBUG:boto:Params: {} 2024-03-04T17:39:16.164 DEBUG:boto:Token: None 2024-03-04T17:39:16.164 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/.o. 2024-03-04T17:39:16.164 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aFmoV6huUMtLiSnpvNBvVs36MJE= 2024-03-04T17:39:16.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aFmoV6huUMtLiSnpvNBvVs36MJE=', 'Content-Length': '0'} 2024-03-04T17:39:16.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005609c877606a77e4-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.167 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:16.167 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:16.167 DEBUG:boto:Method: GET 2024-03-04T17:39:16.167 DEBUG:boto:Path: /ppfahn-12/.o. 2024-03-04T17:39:16.168 DEBUG:boto:Data: 2024-03-04T17:39:16.168 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.168 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.168 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.168 DEBUG:boto:Params: {} 2024-03-04T17:39:16.168 DEBUG:boto:Token: None 2024-03-04T17:39:16.168 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/.o. 2024-03-04T17:39:16.168 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aFmoV6huUMtLiSnpvNBvVs36MJE= 2024-03-04T17:39:16.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aFmoV6huUMtLiSnpvNBvVs36MJE=', 'Content-Length': '0'} 2024-03-04T17:39:16.171 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042c2de82043222a7-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.212 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.212 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.212 DEBUG:boto:path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:16.213 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:16.213 DEBUG:boto:Method: GET 2024-03-04T17:39:16.213 DEBUG:boto:Path: /ppfahn-12/?prefix=.o. 2024-03-04T17:39:16.213 DEBUG:boto:Data: 2024-03-04T17:39:16.213 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.213 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.213 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.213 DEBUG:boto:Params: {} 2024-03-04T17:39:16.213 DEBUG:boto:Token: None 2024-03-04T17:39:16.213 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.214 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077fe98fcf7e37375-0065e60744-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.218 DEBUG:boto:b'ppfahn-12.o.1000false.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.218 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.218 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.218 DEBUG:boto:path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:16.218 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:16.219 DEBUG:boto:Method: GET 2024-03-04T17:39:16.219 DEBUG:boto:Path: /ppfahn-12/?prefix=.o. 2024-03-04T17:39:16.219 DEBUG:boto:Data: 2024-03-04T17:39:16.219 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.219 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.219 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.219 DEBUG:boto:Params: {} 2024-03-04T17:39:16.219 DEBUG:boto:Token: None 2024-03-04T17:39:16.219 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.219 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb39b6ff27567197-0065e60744-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.224 DEBUG:boto:b'ppfahn-12.o.1000false.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:16.224 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:16.224 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:16.224 DEBUG:boto:path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.225 DEBUG:boto:auth_path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.225 DEBUG:boto:Method: GET 2024-03-04T17:39:16.225 DEBUG:boto:Path: /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.225 DEBUG:boto:Data: 2024-03-04T17:39:16.225 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.225 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.225 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.225 DEBUG:boto:Params: {} 2024-03-04T17:39:16.225 DEBUG:boto:Token: None 2024-03-04T17:39:16.225 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.226 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dXzIaUdYrlen5ruVa7g/UE+6Dz8= 2024-03-04T17:39:16.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dXzIaUdYrlen5ruVa7g/UE+6Dz8=', 'Content-Length': '0'} 2024-03-04T17:39:16.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057adc24ac2a7eb36-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.229 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:16.229 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:16.229 DEBUG:boto:path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.229 DEBUG:boto:auth_path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.229 DEBUG:boto:Method: GET 2024-03-04T17:39:16.229 DEBUG:boto:Path: /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.229 DEBUG:boto:Data: 2024-03-04T17:39:16.230 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.230 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.230 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.230 DEBUG:boto:Params: {} 2024-03-04T17:39:16.230 DEBUG:boto:Token: None 2024-03-04T17:39:16.230 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.230 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dXzIaUdYrlen5ruVa7g/UE+6Dz8= 2024-03-04T17:39:16.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dXzIaUdYrlen5ruVa7g/UE+6Dz8=', 'Content-Length': '0'} 2024-03-04T17:39:16.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa1c748ebbe1585e-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.233 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:16.233 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:16.234 DEBUG:boto:path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.234 DEBUG:boto:auth_path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.234 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.234 DEBUG:boto:Path: /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.234 DEBUG:boto:Data: 2024-03-04T17:39:16.234 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.234 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.234 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.234 DEBUG:boto:Params: {} 2024-03-04T17:39:16.234 DEBUG:boto:Token: None 2024-03-04T17:39:16.235 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.235 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V4d7CnqoEeMXxSsqfLJFCvJfsdc= 2024-03-04T17:39:16.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V4d7CnqoEeMXxSsqfLJFCvJfsdc=', 'Content-Length': '0'} 2024-03-04T17:39:16.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000796f755923467d2d-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.237 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:16.238 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:16.238 DEBUG:boto:path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.238 DEBUG:boto:auth_path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.238 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.238 DEBUG:boto:Path: /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.238 DEBUG:boto:Data: 2024-03-04T17:39:16.238 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.238 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.238 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.238 DEBUG:boto:Params: {} 2024-03-04T17:39:16.239 DEBUG:boto:Token: None 2024-03-04T17:39:16.239 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.239 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V4d7CnqoEeMXxSsqfLJFCvJfsdc= 2024-03-04T17:39:16.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V4d7CnqoEeMXxSsqfLJFCvJfsdc=', 'Content-Length': '0'} 2024-03-04T17:39:16.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b24509f75c114c17-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:16.242 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:16.242 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:16.242 DEBUG:boto:path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.242 DEBUG:boto:auth_path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.242 DEBUG:boto:Method: GET 2024-03-04T17:39:16.242 DEBUG:boto:Path: /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.242 DEBUG:boto:Data: 2024-03-04T17:39:16.242 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.242 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.242 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.243 DEBUG:boto:Params: {} 2024-03-04T17:39:16.243 DEBUG:boto:Token: None 2024-03-04T17:39:16.243 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.243 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dXzIaUdYrlen5ruVa7g/UE+6Dz8= 2024-03-04T17:39:16.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dXzIaUdYrlen5ruVa7g/UE+6Dz8=', 'Content-Length': '0'} 2024-03-04T17:39:16.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c3f360fc2f091b8-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.288 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:16.288 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:16.288 DEBUG:boto:path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.288 DEBUG:boto:auth_path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.288 DEBUG:boto:Method: GET 2024-03-04T17:39:16.289 DEBUG:boto:Path: /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.289 DEBUG:boto:Data: 2024-03-04T17:39:16.289 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.289 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.289 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.289 DEBUG:boto:Params: {} 2024-03-04T17:39:16.289 DEBUG:boto:Token: None 2024-03-04T17:39:16.289 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:16.289 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dXzIaUdYrlen5ruVa7g/UE+6Dz8= 2024-03-04T17:39:16.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dXzIaUdYrlen5ruVa7g/UE+6Dz8=', 'Content-Length': '0'} 2024-03-04T17:39:16.292 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d6f67e1a45e242d-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.293 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.293 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.293 DEBUG:boto:path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:16.293 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:16.293 DEBUG:boto:Method: GET 2024-03-04T17:39:16.293 DEBUG:boto:Path: /ppfahn-12/?prefix=%3A 2024-03-04T17:39:16.293 DEBUG:boto:Data: 2024-03-04T17:39:16.293 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.293 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.293 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.293 DEBUG:boto:Params: {} 2024-03-04T17:39:16.294 DEBUG:boto:Token: None 2024-03-04T17:39:16.294 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.294 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc2bdd1656d440cd-0065e60744-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.298 DEBUG:boto:b'ppfahn-12:1000false:2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.299 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.299 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.299 DEBUG:boto:path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:16.299 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:16.299 DEBUG:boto:Method: GET 2024-03-04T17:39:16.299 DEBUG:boto:Path: /ppfahn-12/?prefix=%3A 2024-03-04T17:39:16.299 DEBUG:boto:Data: 2024-03-04T17:39:16.299 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.299 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.299 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.300 DEBUG:boto:Params: {} 2024-03-04T17:39:16.300 DEBUG:boto:Token: None 2024-03-04T17:39:16.300 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.300 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b07fba46616559d-0065e60744-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.348 DEBUG:boto:b'ppfahn-12:1000false:2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.348 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:16.348 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:16.349 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.349 DEBUG:boto:Path: /ppfahn-12/%3A 2024-03-04T17:39:16.349 DEBUG:boto:Data: 2024-03-04T17:39:16.349 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.349 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.349 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.349 DEBUG:boto:Params: {} 2024-03-04T17:39:16.349 DEBUG:boto:Token: None 2024-03-04T17:39:16.349 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/%3A 2024-03-04T17:39:16.350 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V4zIde0x7t9X9Vy8BkYShwL5ihQ= 2024-03-04T17:39:16.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V4zIde0x7t9X9Vy8BkYShwL5ihQ=', 'Content-Length': '0'} 2024-03-04T17:39:16.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d8803788317f9dc-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.353 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:16.353 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:16.353 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.353 DEBUG:boto:Path: /ppfahn-12/%3A 2024-03-04T17:39:16.353 DEBUG:boto:Data: 2024-03-04T17:39:16.353 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.353 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.353 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.353 DEBUG:boto:Params: {} 2024-03-04T17:39:16.353 DEBUG:boto:Token: None 2024-03-04T17:39:16.354 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/%3A 2024-03-04T17:39:16.354 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V4zIde0x7t9X9Vy8BkYShwL5ihQ= 2024-03-04T17:39:16.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V4zIde0x7t9X9Vy8BkYShwL5ihQ=', 'Content-Length': '0'} 2024-03-04T17:39:16.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdc90d1bbdd5edf4-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:16.357 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:16.357 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:16.357 DEBUG:boto:Method: GET 2024-03-04T17:39:16.357 DEBUG:boto:Path: /ppfahn-12/%3A 2024-03-04T17:39:16.357 DEBUG:boto:Data: 2024-03-04T17:39:16.357 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.357 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.357 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.357 DEBUG:boto:Params: {} 2024-03-04T17:39:16.358 DEBUG:boto:Token: None 2024-03-04T17:39:16.358 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/%3A 2024-03-04T17:39:16.358 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aLZjLiDgZs4huHYWvwU5CdGBe3U= 2024-03-04T17:39:16.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aLZjLiDgZs4huHYWvwU5CdGBe3U=', 'Content-Length': '0'} 2024-03-04T17:39:16.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d1481e2c665c8a1-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.361 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:16.361 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:16.362 DEBUG:boto:Method: GET 2024-03-04T17:39:16.362 DEBUG:boto:Path: /ppfahn-12/%3A 2024-03-04T17:39:16.362 DEBUG:boto:Data: 2024-03-04T17:39:16.362 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.362 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.362 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.362 DEBUG:boto:Params: {} 2024-03-04T17:39:16.362 DEBUG:boto:Token: None 2024-03-04T17:39:16.362 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/%3A 2024-03-04T17:39:16.362 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aLZjLiDgZs4huHYWvwU5CdGBe3U= 2024-03-04T17:39:16.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aLZjLiDgZs4huHYWvwU5CdGBe3U=', 'Content-Length': '0'} 2024-03-04T17:39:16.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011993cec8078f590-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.365 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.366 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.366 DEBUG:boto:path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:16.366 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:16.366 DEBUG:boto:Method: GET 2024-03-04T17:39:16.366 DEBUG:boto:Path: /ppfahn-12/?prefix=%3A 2024-03-04T17:39:16.366 DEBUG:boto:Data: 2024-03-04T17:39:16.366 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.366 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.366 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.366 DEBUG:boto:Params: {} 2024-03-04T17:39:16.366 DEBUG:boto:Token: None 2024-03-04T17:39:16.367 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.367 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d2e129c02d9a1dc-0065e60744-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.412 DEBUG:boto:b'ppfahn-12:1000false:2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.413 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.413 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.413 DEBUG:boto:path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:16.413 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:16.413 DEBUG:boto:Method: GET 2024-03-04T17:39:16.413 DEBUG:boto:Path: /ppfahn-12/?prefix=%3A 2024-03-04T17:39:16.413 DEBUG:boto:Data: 2024-03-04T17:39:16.413 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.413 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.413 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.413 DEBUG:boto:Params: {} 2024-03-04T17:39:16.414 DEBUG:boto:Token: None 2024-03-04T17:39:16.414 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.414 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adf7574f5b2fab1d-0065e60744-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.419 DEBUG:boto:b'ppfahn-12:1000false:2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:16.419 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:16.419 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:16.419 DEBUG:boto:path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.419 DEBUG:boto:auth_path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.420 DEBUG:boto:Method: GET 2024-03-04T17:39:16.420 DEBUG:boto:Path: /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.420 DEBUG:boto:Data: 2024-03-04T17:39:16.420 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.420 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.420 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.420 DEBUG:boto:Params: {} 2024-03-04T17:39:16.420 DEBUG:boto:Token: None 2024-03-04T17:39:16.420 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.421 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TUpXwbr4d4kAMSJegMCqai0jIBo= 2024-03-04T17:39:16.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TUpXwbr4d4kAMSJegMCqai0jIBo=', 'Content-Length': '0'} 2024-03-04T17:39:16.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cec8a29c7137b66-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.424 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:16.424 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:16.424 DEBUG:boto:path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.424 DEBUG:boto:auth_path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.424 DEBUG:boto:Method: GET 2024-03-04T17:39:16.424 DEBUG:boto:Path: /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.424 DEBUG:boto:Data: 2024-03-04T17:39:16.424 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.425 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.425 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.425 DEBUG:boto:Params: {} 2024-03-04T17:39:16.425 DEBUG:boto:Token: None 2024-03-04T17:39:16.425 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.425 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TUpXwbr4d4kAMSJegMCqai0jIBo= 2024-03-04T17:39:16.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TUpXwbr4d4kAMSJegMCqai0jIBo=', 'Content-Length': '0'} 2024-03-04T17:39:16.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d667235cc0ffb99-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.472 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:16.472 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:16.472 DEBUG:boto:path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.472 DEBUG:boto:auth_path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.472 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.473 DEBUG:boto:Path: /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.473 DEBUG:boto:Data: 2024-03-04T17:39:16.473 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.473 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.473 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.473 DEBUG:boto:Params: {} 2024-03-04T17:39:16.473 DEBUG:boto:Token: None 2024-03-04T17:39:16.473 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.473 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nGT0n+9E/XkGGIRGxVILWhAEYLo= 2024-03-04T17:39:16.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nGT0n+9E/XkGGIRGxVILWhAEYLo=', 'Content-Length': '0'} 2024-03-04T17:39:16.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007753d85065ef7b69-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.476 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:16.476 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:16.477 DEBUG:boto:path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.477 DEBUG:boto:auth_path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.477 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.477 DEBUG:boto:Path: /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.477 DEBUG:boto:Data: 2024-03-04T17:39:16.477 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.477 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.477 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.477 DEBUG:boto:Params: {} 2024-03-04T17:39:16.477 DEBUG:boto:Token: None 2024-03-04T17:39:16.477 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.478 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nGT0n+9E/XkGGIRGxVILWhAEYLo= 2024-03-04T17:39:16.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nGT0n+9E/XkGGIRGxVILWhAEYLo=', 'Content-Length': '0'} 2024-03-04T17:39:16.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba8a193096d227b2-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:16.481 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:16.481 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:16.481 DEBUG:boto:path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.481 DEBUG:boto:auth_path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.481 DEBUG:boto:Method: GET 2024-03-04T17:39:16.481 DEBUG:boto:Path: /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.481 DEBUG:boto:Data: 2024-03-04T17:39:16.481 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.481 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.481 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.481 DEBUG:boto:Params: {} 2024-03-04T17:39:16.482 DEBUG:boto:Token: None 2024-03-04T17:39:16.482 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.482 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TUpXwbr4d4kAMSJegMCqai0jIBo= 2024-03-04T17:39:16.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TUpXwbr4d4kAMSJegMCqai0jIBo=', 'Content-Length': '0'} 2024-03-04T17:39:16.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004705c94c7dafa50f-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.485 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:16.485 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:16.485 DEBUG:boto:path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.485 DEBUG:boto:auth_path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.485 DEBUG:boto:Method: GET 2024-03-04T17:39:16.485 DEBUG:boto:Path: /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.485 DEBUG:boto:Data: 2024-03-04T17:39:16.485 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.486 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.486 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.486 DEBUG:boto:Params: {} 2024-03-04T17:39:16.486 DEBUG:boto:Token: None 2024-03-04T17:39:16.486 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:16.486 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TUpXwbr4d4kAMSJegMCqai0jIBo= 2024-03-04T17:39:16.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TUpXwbr4d4kAMSJegMCqai0jIBo=', 'Content-Length': '0'} 2024-03-04T17:39:16.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000382cd881afc1d87f-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.489 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.489 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.489 DEBUG:boto:path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:16.489 DEBUG:boto:auth_path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:16.490 DEBUG:boto:Method: GET 2024-03-04T17:39:16.490 DEBUG:boto:Path: /ppfahn-12/?prefix=_myobj 2024-03-04T17:39:16.490 DEBUG:boto:Data: 2024-03-04T17:39:16.490 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.490 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.490 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.490 DEBUG:boto:Params: {} 2024-03-04T17:39:16.490 DEBUG:boto:Token: None 2024-03-04T17:39:16.490 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.490 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055a781f5f9edd190-0065e60744-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.536 DEBUG:boto:b'ppfahn-12_myobj1000false_myobj2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.536 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.536 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.537 DEBUG:boto:path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:16.537 DEBUG:boto:auth_path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:16.537 DEBUG:boto:Method: GET 2024-03-04T17:39:16.537 DEBUG:boto:Path: /ppfahn-12/?prefix=_myobj 2024-03-04T17:39:16.537 DEBUG:boto:Data: 2024-03-04T17:39:16.537 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.537 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.537 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.537 DEBUG:boto:Params: {} 2024-03-04T17:39:16.537 DEBUG:boto:Token: None 2024-03-04T17:39:16.537 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.538 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d35968d5d8a5a33-0065e60744-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.542 DEBUG:boto:b'ppfahn-12_myobj1000false_myobj2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.542 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:16.542 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:16.542 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.543 DEBUG:boto:Path: /ppfahn-12/_myobj 2024-03-04T17:39:16.543 DEBUG:boto:Data: 2024-03-04T17:39:16.543 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.543 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.543 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.543 DEBUG:boto:Params: {} 2024-03-04T17:39:16.543 DEBUG:boto:Token: None 2024-03-04T17:39:16.543 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/_myobj 2024-03-04T17:39:16.543 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Znuz6qhwrLugZpyI7JfiqbnymS0= 2024-03-04T17:39:16.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Znuz6qhwrLugZpyI7JfiqbnymS0=', 'Content-Length': '0'} 2024-03-04T17:39:16.546 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a82cb122fb92d6cf-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.546 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:16.546 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:16.546 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.547 DEBUG:boto:Path: /ppfahn-12/_myobj 2024-03-04T17:39:16.547 DEBUG:boto:Data: 2024-03-04T17:39:16.547 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.547 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.547 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.547 DEBUG:boto:Params: {} 2024-03-04T17:39:16.547 DEBUG:boto:Token: None 2024-03-04T17:39:16.547 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/_myobj 2024-03-04T17:39:16.547 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Znuz6qhwrLugZpyI7JfiqbnymS0= 2024-03-04T17:39:16.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Znuz6qhwrLugZpyI7JfiqbnymS0=', 'Content-Length': '0'} 2024-03-04T17:39:16.550 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edc96e82cfcac0e9-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:16.550 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:16.550 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:16.551 DEBUG:boto:Method: GET 2024-03-04T17:39:16.551 DEBUG:boto:Path: /ppfahn-12/_myobj 2024-03-04T17:39:16.551 DEBUG:boto:Data: 2024-03-04T17:39:16.551 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.551 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.551 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.551 DEBUG:boto:Params: {} 2024-03-04T17:39:16.551 DEBUG:boto:Token: None 2024-03-04T17:39:16.551 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/_myobj 2024-03-04T17:39:16.551 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tz8lKJCkPiuA/4KOjulEEnfycfA= 2024-03-04T17:39:16.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tz8lKJCkPiuA/4KOjulEEnfycfA=', 'Content-Length': '0'} 2024-03-04T17:39:16.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7c1b8ff3fae08a0-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.555 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:16.555 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:16.555 DEBUG:boto:Method: GET 2024-03-04T17:39:16.555 DEBUG:boto:Path: /ppfahn-12/_myobj 2024-03-04T17:39:16.555 DEBUG:boto:Data: 2024-03-04T17:39:16.555 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.555 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.555 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.556 DEBUG:boto:Params: {} 2024-03-04T17:39:16.556 DEBUG:boto:Token: None 2024-03-04T17:39:16.556 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/_myobj 2024-03-04T17:39:16.556 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tz8lKJCkPiuA/4KOjulEEnfycfA= 2024-03-04T17:39:16.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tz8lKJCkPiuA/4KOjulEEnfycfA=', 'Content-Length': '0'} 2024-03-04T17:39:16.559 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5ee605cb23a8bc0-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.600 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.600 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.601 DEBUG:boto:path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:16.601 DEBUG:boto:auth_path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:16.601 DEBUG:boto:Method: GET 2024-03-04T17:39:16.601 DEBUG:boto:Path: /ppfahn-12/?prefix=_myobj 2024-03-04T17:39:16.601 DEBUG:boto:Data: 2024-03-04T17:39:16.601 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.601 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.601 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.601 DEBUG:boto:Params: {} 2024-03-04T17:39:16.601 DEBUG:boto:Token: None 2024-03-04T17:39:16.601 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.602 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064b5531be5f2c7fd-0065e60744-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.606 DEBUG:boto:b'ppfahn-12_myobj1000false_myobj2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.606 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.606 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.606 DEBUG:boto:path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:16.606 DEBUG:boto:auth_path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:16.606 DEBUG:boto:Method: GET 2024-03-04T17:39:16.607 DEBUG:boto:Path: /ppfahn-12/?prefix=_myobj 2024-03-04T17:39:16.607 DEBUG:boto:Data: 2024-03-04T17:39:16.607 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.607 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.607 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.607 DEBUG:boto:Params: {} 2024-03-04T17:39:16.607 DEBUG:boto:Token: None 2024-03-04T17:39:16.607 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.607 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a27011084d21cbe-0065e60744-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.612 DEBUG:boto:b'ppfahn-12_myobj1000false_myobj2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:16.612 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:16.613 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:16.613 DEBUG:boto:path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.613 DEBUG:boto:auth_path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.613 DEBUG:boto:Method: GET 2024-03-04T17:39:16.613 DEBUG:boto:Path: /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.613 DEBUG:boto:Data: 2024-03-04T17:39:16.613 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.613 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.613 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.613 DEBUG:boto:Params: {} 2024-03-04T17:39:16.613 DEBUG:boto:Token: None 2024-03-04T17:39:16.614 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.614 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dK7SfLsHxernITMYmOvrhQYNgg8= 2024-03-04T17:39:16.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dK7SfLsHxernITMYmOvrhQYNgg8=', 'Content-Length': '0'} 2024-03-04T17:39:16.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2aaf576c109efeb-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.660 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:16.660 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:16.660 DEBUG:boto:path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.660 DEBUG:boto:auth_path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.660 DEBUG:boto:Method: GET 2024-03-04T17:39:16.661 DEBUG:boto:Path: /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.661 DEBUG:boto:Data: 2024-03-04T17:39:16.661 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.661 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.661 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.661 DEBUG:boto:Params: {} 2024-03-04T17:39:16.661 DEBUG:boto:Token: None 2024-03-04T17:39:16.661 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.662 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dK7SfLsHxernITMYmOvrhQYNgg8= 2024-03-04T17:39:16.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dK7SfLsHxernITMYmOvrhQYNgg8=', 'Content-Length': '0'} 2024-03-04T17:39:16.664 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000531113393fc756ec-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.665 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:16.665 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:16.665 DEBUG:boto:path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.665 DEBUG:boto:auth_path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.665 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.665 DEBUG:boto:Path: /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.665 DEBUG:boto:Data: 2024-03-04T17:39:16.666 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.666 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.666 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.666 DEBUG:boto:Params: {} 2024-03-04T17:39:16.666 DEBUG:boto:Token: None 2024-03-04T17:39:16.666 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.666 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:R4fNH3AI7rmxcgogfMOUvGRUD80= 2024-03-04T17:39:16.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:R4fNH3AI7rmxcgogfMOUvGRUD80=', 'Content-Length': '0'} 2024-03-04T17:39:16.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d96ab0ef7165233b-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.669 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:16.670 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:16.670 DEBUG:boto:path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.670 DEBUG:boto:auth_path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.670 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.670 DEBUG:boto:Path: /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.670 DEBUG:boto:Data: 2024-03-04T17:39:16.670 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.670 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.670 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.670 DEBUG:boto:Params: {} 2024-03-04T17:39:16.670 DEBUG:boto:Token: None 2024-03-04T17:39:16.671 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.671 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:R4fNH3AI7rmxcgogfMOUvGRUD80= 2024-03-04T17:39:16.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:R4fNH3AI7rmxcgogfMOUvGRUD80=', 'Content-Length': '0'} 2024-03-04T17:39:16.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004845cd951c61fccc-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:16.674 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:16.674 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:16.674 DEBUG:boto:path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.674 DEBUG:boto:auth_path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.674 DEBUG:boto:Method: GET 2024-03-04T17:39:16.674 DEBUG:boto:Path: /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.674 DEBUG:boto:Data: 2024-03-04T17:39:16.674 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.674 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.675 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.675 DEBUG:boto:Params: {} 2024-03-04T17:39:16.675 DEBUG:boto:Token: None 2024-03-04T17:39:16.675 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.675 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dK7SfLsHxernITMYmOvrhQYNgg8= 2024-03-04T17:39:16.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dK7SfLsHxernITMYmOvrhQYNgg8=', 'Content-Length': '0'} 2024-03-04T17:39:16.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024649598fda131da-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.678 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:16.678 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:16.679 DEBUG:boto:path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.679 DEBUG:boto:auth_path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.679 DEBUG:boto:Method: GET 2024-03-04T17:39:16.679 DEBUG:boto:Path: /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.679 DEBUG:boto:Data: 2024-03-04T17:39:16.679 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.679 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.679 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.679 DEBUG:boto:Params: {} 2024-03-04T17:39:16.679 DEBUG:boto:Token: None 2024-03-04T17:39:16.679 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:16.680 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dK7SfLsHxernITMYmOvrhQYNgg8= 2024-03-04T17:39:16.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dK7SfLsHxernITMYmOvrhQYNgg8=', 'Content-Length': '0'} 2024-03-04T17:39:16.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003269c744b38c5e4e-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.724 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.724 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.724 DEBUG:boto:path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:16.724 DEBUG:boto:auth_path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:16.724 DEBUG:boto:Method: GET 2024-03-04T17:39:16.724 DEBUG:boto:Path: /ppfahn-12/?prefix=myobj 2024-03-04T17:39:16.725 DEBUG:boto:Data: 2024-03-04T17:39:16.725 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.725 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.725 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.725 DEBUG:boto:Params: {} 2024-03-04T17:39:16.725 DEBUG:boto:Token: None 2024-03-04T17:39:16.725 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.725 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050b12e24fdd59f0f-0065e60744-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.730 DEBUG:boto:b'ppfahn-12myobj1000falsemyobj2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.730 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.730 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.730 DEBUG:boto:path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:16.730 DEBUG:boto:auth_path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:16.730 DEBUG:boto:Method: GET 2024-03-04T17:39:16.730 DEBUG:boto:Path: /ppfahn-12/?prefix=myobj 2024-03-04T17:39:16.730 DEBUG:boto:Data: 2024-03-04T17:39:16.731 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.731 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.731 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.731 DEBUG:boto:Params: {} 2024-03-04T17:39:16.731 DEBUG:boto:Token: None 2024-03-04T17:39:16.731 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.731 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d038b19f3364bfd-0065e60744-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.735 DEBUG:boto:b'ppfahn-12myobj1000falsemyobj2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.735 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:16.736 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:16.736 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.736 DEBUG:boto:Path: /ppfahn-12/myobj 2024-03-04T17:39:16.736 DEBUG:boto:Data: 2024-03-04T17:39:16.736 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.736 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.736 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.736 DEBUG:boto:Params: {} 2024-03-04T17:39:16.736 DEBUG:boto:Token: None 2024-03-04T17:39:16.737 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/myobj 2024-03-04T17:39:16.737 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VxXBqAMqZiUDNCS/u3YNivdqkOw= 2024-03-04T17:39:16.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VxXBqAMqZiUDNCS/u3YNivdqkOw=', 'Content-Length': '0'} 2024-03-04T17:39:16.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009470917f03006f83-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.740 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:16.740 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:16.740 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.740 DEBUG:boto:Path: /ppfahn-12/myobj 2024-03-04T17:39:16.740 DEBUG:boto:Data: 2024-03-04T17:39:16.740 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.740 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.740 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.740 DEBUG:boto:Params: {} 2024-03-04T17:39:16.741 DEBUG:boto:Token: None 2024-03-04T17:39:16.741 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/myobj 2024-03-04T17:39:16.741 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VxXBqAMqZiUDNCS/u3YNivdqkOw= 2024-03-04T17:39:16.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VxXBqAMqZiUDNCS/u3YNivdqkOw=', 'Content-Length': '0'} 2024-03-04T17:39:16.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c373a40fac64b3f3-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:16.744 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:16.744 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:16.744 DEBUG:boto:Method: GET 2024-03-04T17:39:16.744 DEBUG:boto:Path: /ppfahn-12/myobj 2024-03-04T17:39:16.744 DEBUG:boto:Data: 2024-03-04T17:39:16.744 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.744 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.745 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.745 DEBUG:boto:Params: {} 2024-03-04T17:39:16.745 DEBUG:boto:Token: None 2024-03-04T17:39:16.745 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/myobj 2024-03-04T17:39:16.745 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AYYjII38I8C01lHxpG7YDPZphLY= 2024-03-04T17:39:16.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AYYjII38I8C01lHxpG7YDPZphLY=', 'Content-Length': '0'} 2024-03-04T17:39:16.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068721dd796439a2b-0065e60744-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.788 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:16.788 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:16.788 DEBUG:boto:Method: GET 2024-03-04T17:39:16.789 DEBUG:boto:Path: /ppfahn-12/myobj 2024-03-04T17:39:16.789 DEBUG:boto:Data: 2024-03-04T17:39:16.789 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.789 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.789 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.789 DEBUG:boto:Params: {} 2024-03-04T17:39:16.789 DEBUG:boto:Token: None 2024-03-04T17:39:16.789 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/myobj 2024-03-04T17:39:16.789 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AYYjII38I8C01lHxpG7YDPZphLY= 2024-03-04T17:39:16.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AYYjII38I8C01lHxpG7YDPZphLY=', 'Content-Length': '0'} 2024-03-04T17:39:16.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e31f2b989a555eaf-0065e60744-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.792 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.792 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.792 DEBUG:boto:path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:16.793 DEBUG:boto:auth_path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:16.793 DEBUG:boto:Method: GET 2024-03-04T17:39:16.793 DEBUG:boto:Path: /ppfahn-12/?prefix=myobj 2024-03-04T17:39:16.793 DEBUG:boto:Data: 2024-03-04T17:39:16.793 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.793 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:16.793 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.793 DEBUG:boto:Params: {} 2024-03-04T17:39:16.793 DEBUG:boto:Token: None 2024-03-04T17:39:16.793 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.793 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e075477ee5a4053-0065e60744-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.798 DEBUG:boto:b'ppfahn-12myobj1000falsemyobj2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.798 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.798 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.799 DEBUG:boto:path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:16.799 DEBUG:boto:auth_path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:16.799 DEBUG:boto:Method: GET 2024-03-04T17:39:16.799 DEBUG:boto:Path: /ppfahn-12/?prefix=myobj 2024-03-04T17:39:16.799 DEBUG:boto:Data: 2024-03-04T17:39:16.799 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.799 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.799 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.799 DEBUG:boto:Params: {} 2024-03-04T17:39:16.799 DEBUG:boto:Token: None 2024-03-04T17:39:16.800 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.800 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q= 2024-03-04T17:39:16.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MkmYDf16EZrFsH0mbdtrlLDA2+Q=', 'Content-Length': '0'} 2024-03-04T17:39:16.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001df6125694843288-0065e60744-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.848 DEBUG:boto:b'ppfahn-12myobj1000falsemyobj2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:16.848 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-12 zones={test-zone1, test-zone3} 2024-03-04T17:39:16.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:16.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:16.849 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:39:16.900 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:16.900 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:16.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:16.959+0000 7fb236a87980 1 bucket sync caught up with source: 2024-03-04T17:39:16.965 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000002.9.6, , 00000000001.9.6, , 00000000001.9.6, 00000000002.26.6, 00000000001.9.6, 00000000001.9.6, 00000000001.14.6, ] 2024-03-04T17:39:16.965 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000002.9.6, , 00000000001.9.6, , 00000000001.9.6, 00000000002.26.6, 00000000001.9.6, 00000000001.9.6, 00000000001.14.6, ] 2024-03-04T17:39:16.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:16.959+0000 7fb236a87980 0 bucket checkpoint complete 2024-03-04T17:39:16.987 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-12 zones={test-zone2, test-zone3} 2024-03-04T17:39:16.987 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.987 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.987 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.988 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:39:16.988 DEBUG:boto:Data: 2024-03-04T17:39:16.988 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.988 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:16.988 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.988 DEBUG:boto:Params: {} 2024-03-04T17:39:16.988 DEBUG:boto:Token: None 2024-03-04T17:39:16.988 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.989 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WXbHpkt1EqSKH5AwUZwlqvpmVmY= 2024-03-04T17:39:16.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WXbHpkt1EqSKH5AwUZwlqvpmVmY=', 'Content-Length': '0'} 2024-03-04T17:39:16.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edeb8457cad10f9e-0065e60744-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.991 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.991 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.991 DEBUG:boto:Method: HEAD 2024-03-04T17:39:16.992 DEBUG:boto:Path: /ppfahn-12/ 2024-03-04T17:39:16.992 DEBUG:boto:Data: 2024-03-04T17:39:16.992 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.992 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:16.992 DEBUG:boto:Port: 8001 2024-03-04T17:39:16.992 DEBUG:boto:Params: {} 2024-03-04T17:39:16.992 DEBUG:boto:Token: None 2024-03-04T17:39:16.992 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/ 2024-03-04T17:39:16.992 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WXbHpkt1EqSKH5AwUZwlqvpmVmY= 2024-03-04T17:39:16.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WXbHpkt1EqSKH5AwUZwlqvpmVmY=', 'Content-Length': '0'} 2024-03-04T17:39:16.996 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000026289ce9d00c549-0065e60744-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:16.996 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:39:16.996 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:16.997 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:16.997 DEBUG:boto:path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:16.997 DEBUG:boto:auth_path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:16.997 DEBUG:boto:Method: GET 2024-03-04T17:39:16.997 DEBUG:boto:Path: /ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:16.997 DEBUG:boto:Data: 2024-03-04T17:39:16.997 DEBUG:boto:Headers: {} 2024-03-04T17:39:16.997 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:16.997 DEBUG:boto:Port: 8000 2024-03-04T17:39:16.997 DEBUG:boto:Params: {} 2024-03-04T17:39:16.997 DEBUG:boto:Token: None 2024-03-04T17:39:16.998 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:16 GMT /ppfahn-12/?versions 2024-03-04T17:39:16.998 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PY43P+K8I60PiwL/r3AHdmlbass= 2024-03-04T17:39:16.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PY43P+K8I60PiwL/r3AHdmlbass=', 'Content-Length': '0'} 2024-03-04T17:39:17.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cba2e7a5b9a407b4-0065e60744-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.000 DEBUG:boto:b'ppfahn-12999false&nulltrue2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.001 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T17:39:17.001 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T17:39:17.001 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T17:39:17.001 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T17:39:17.001 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T17:39:17.001 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T17:39:17.001 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T17:39:17.001 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T17:39:17.001 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:17.002 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:39:17.002 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.002 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.002 DEBUG:boto:path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:17.002 DEBUG:boto:auth_path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:17.002 DEBUG:boto:Method: GET 2024-03-04T17:39:17.002 DEBUG:boto:Path: /ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:17.002 DEBUG:boto:Data: 2024-03-04T17:39:17.002 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.002 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.003 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.003 DEBUG:boto:Params: {} 2024-03-04T17:39:17.003 DEBUG:boto:Token: None 2024-03-04T17:39:17.003 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/?versions 2024-03-04T17:39:17.003 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PQGAJpq/fB6/INqzt3In0YMHges= 2024-03-04T17:39:17.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PQGAJpq/fB6/INqzt3In0YMHges=', 'Content-Length': '0'} 2024-03-04T17:39:17.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e602524b2ccbe8be-0065e60745-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.009 DEBUG:boto:b'ppfahn-12999false&nulltrue2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.010 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T17:39:17.010 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T17:39:17.010 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T17:39:17.010 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T17:39:17.010 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T17:39:17.010 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T17:39:17.010 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T17:39:17.010 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T17:39:17.010 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:17.010 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.011 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.011 DEBUG:boto:path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:17.011 DEBUG:boto:auth_path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:17.011 DEBUG:boto:Method: GET 2024-03-04T17:39:17.011 DEBUG:boto:Path: /ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:17.011 DEBUG:boto:Data: 2024-03-04T17:39:17.011 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.011 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.011 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.011 DEBUG:boto:Params: {} 2024-03-04T17:39:17.011 DEBUG:boto:Token: None 2024-03-04T17:39:17.012 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/?versions 2024-03-04T17:39:17.012 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PQGAJpq/fB6/INqzt3In0YMHges= 2024-03-04T17:39:17.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PQGAJpq/fB6/INqzt3In0YMHges=', 'Content-Length': '0'} 2024-03-04T17:39:17.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d71fea9d844bd72d-0065e60745-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.014 DEBUG:boto:b'ppfahn-12999false&nulltrue2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.015 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.015 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.015 DEBUG:boto:path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:17.015 DEBUG:boto:auth_path=/ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:17.015 DEBUG:boto:Method: GET 2024-03-04T17:39:17.016 DEBUG:boto:Path: /ppfahn-12/?versions&max-keys=999 2024-03-04T17:39:17.016 DEBUG:boto:Data: 2024-03-04T17:39:17.016 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.016 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.016 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.016 DEBUG:boto:Params: {} 2024-03-04T17:39:17.016 DEBUG:boto:Token: None 2024-03-04T17:39:17.016 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/?versions 2024-03-04T17:39:17.016 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PQGAJpq/fB6/INqzt3In0YMHges= 2024-03-04T17:39:17.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PQGAJpq/fB6/INqzt3In0YMHges=', 'Content-Length': '0'} 2024-03-04T17:39:17.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002db6d397c520627a-0065e60745-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.021 DEBUG:boto:b'ppfahn-12999false&nulltrue2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:17.022 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:17.022 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:17.022 DEBUG:boto:path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.022 DEBUG:boto:auth_path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.022 DEBUG:boto:Method: GET 2024-03-04T17:39:17.022 DEBUG:boto:Path: /ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.022 DEBUG:boto:Data: 2024-03-04T17:39:17.022 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.023 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.023 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.023 DEBUG:boto:Params: {} 2024-03-04T17:39:17.023 DEBUG:boto:Token: None 2024-03-04T17:39:17.023 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.023 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9R//bYlt/PrvuJPOhWz8eW+TM+8= 2024-03-04T17:39:17.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9R//bYlt/PrvuJPOhWz8eW+TM+8=', 'Content-Length': '0'} 2024-03-04T17:39:17.025 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000033dd230926dd87e-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.068 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:17.068 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:17.068 DEBUG:boto:path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.068 DEBUG:boto:auth_path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.068 DEBUG:boto:Method: GET 2024-03-04T17:39:17.069 DEBUG:boto:Path: /ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.069 DEBUG:boto:Data: 2024-03-04T17:39:17.069 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.069 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.069 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.069 DEBUG:boto:Params: {} 2024-03-04T17:39:17.069 DEBUG:boto:Token: None 2024-03-04T17:39:17.069 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.069 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9R//bYlt/PrvuJPOhWz8eW+TM+8= 2024-03-04T17:39:17.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9R//bYlt/PrvuJPOhWz8eW+TM+8=', 'Content-Length': '0'} 2024-03-04T17:39:17.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae8f74f25e52a5d7-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.073 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:17.073 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:17.073 DEBUG:boto:path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.073 DEBUG:boto:auth_path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.073 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.073 DEBUG:boto:Path: /ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.073 DEBUG:boto:Data: 2024-03-04T17:39:17.074 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.074 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.074 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.074 DEBUG:boto:Params: {} 2024-03-04T17:39:17.074 DEBUG:boto:Token: None 2024-03-04T17:39:17.074 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.074 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6xLUKSd6ZFdDAh7UGBob/pTZbBw= 2024-03-04T17:39:17.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6xLUKSd6ZFdDAh7UGBob/pTZbBw=', 'Content-Length': '0'} 2024-03-04T17:39:17.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b85c75671ad06f11-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.077 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:17.077 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:17.077 DEBUG:boto:path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.077 DEBUG:boto:auth_path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.078 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.078 DEBUG:boto:Path: /ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.078 DEBUG:boto:Data: 2024-03-04T17:39:17.078 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.078 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.078 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.078 DEBUG:boto:Params: {} 2024-03-04T17:39:17.078 DEBUG:boto:Token: None 2024-03-04T17:39:17.078 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.078 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6xLUKSd6ZFdDAh7UGBob/pTZbBw= 2024-03-04T17:39:17.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6xLUKSd6ZFdDAh7UGBob/pTZbBw=', 'Content-Length': '0'} 2024-03-04T17:39:17.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030bedc7ec2016f02-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:17.082 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:17.082 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:17.082 DEBUG:boto:path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.082 DEBUG:boto:auth_path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.082 DEBUG:boto:Method: GET 2024-03-04T17:39:17.082 DEBUG:boto:Path: /ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.082 DEBUG:boto:Data: 2024-03-04T17:39:17.082 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.082 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.083 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.083 DEBUG:boto:Params: {} 2024-03-04T17:39:17.083 DEBUG:boto:Token: None 2024-03-04T17:39:17.083 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.083 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9R//bYlt/PrvuJPOhWz8eW+TM+8= 2024-03-04T17:39:17.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9R//bYlt/PrvuJPOhWz8eW+TM+8=', 'Content-Length': '0'} 2024-03-04T17:39:17.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9f21aa0a39ee176-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.086 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:17.086 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:17.086 DEBUG:boto:path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.086 DEBUG:boto:auth_path=/ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.087 DEBUG:boto:Method: GET 2024-03-04T17:39:17.087 DEBUG:boto:Path: /ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.087 DEBUG:boto:Data: 2024-03-04T17:39:17.087 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.087 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.087 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.087 DEBUG:boto:Params: {} 2024-03-04T17:39:17.087 DEBUG:boto:Token: None 2024-03-04T17:39:17.087 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/%26?versionId=null 2024-03-04T17:39:17.087 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9R//bYlt/PrvuJPOhWz8eW+TM+8= 2024-03-04T17:39:17.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9R//bYlt/PrvuJPOhWz8eW+TM+8=', 'Content-Length': '0'} 2024-03-04T17:39:17.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ffc54127df77ff5-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.132 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.132 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.132 DEBUG:boto:path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:17.132 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:17.133 DEBUG:boto:Method: GET 2024-03-04T17:39:17.133 DEBUG:boto:Path: /ppfahn-12/?prefix=%26 2024-03-04T17:39:17.133 DEBUG:boto:Data: 2024-03-04T17:39:17.133 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.133 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.133 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.133 DEBUG:boto:Params: {} 2024-03-04T17:39:17.133 DEBUG:boto:Token: None 2024-03-04T17:39:17.133 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.133 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084a7426f6f5a9657-0065e60745-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.138 DEBUG:boto:b'ppfahn-12&1000false&2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.138 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.138 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.138 DEBUG:boto:path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:17.138 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:17.138 DEBUG:boto:Method: GET 2024-03-04T17:39:17.139 DEBUG:boto:Path: /ppfahn-12/?prefix=%26 2024-03-04T17:39:17.139 DEBUG:boto:Data: 2024-03-04T17:39:17.139 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.139 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.139 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.139 DEBUG:boto:Params: {} 2024-03-04T17:39:17.139 DEBUG:boto:Token: None 2024-03-04T17:39:17.139 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.139 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc3714d49797d2d2-0065e60745-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.143 DEBUG:boto:b'ppfahn-12&1000false&2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.144 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:17.144 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:17.144 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.144 DEBUG:boto:Path: /ppfahn-12/%26 2024-03-04T17:39:17.144 DEBUG:boto:Data: 2024-03-04T17:39:17.144 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.144 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.144 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.145 DEBUG:boto:Params: {} 2024-03-04T17:39:17.145 DEBUG:boto:Token: None 2024-03-04T17:39:17.145 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/%26 2024-03-04T17:39:17.145 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MTwraH+kI8ahx+B3oAW3ie/JhYY= 2024-03-04T17:39:17.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MTwraH+kI8ahx+B3oAW3ie/JhYY=', 'Content-Length': '0'} 2024-03-04T17:39:17.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098a149f8277bf36f-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.148 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:17.148 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:17.148 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.148 DEBUG:boto:Path: /ppfahn-12/%26 2024-03-04T17:39:17.148 DEBUG:boto:Data: 2024-03-04T17:39:17.148 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.148 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.149 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.149 DEBUG:boto:Params: {} 2024-03-04T17:39:17.149 DEBUG:boto:Token: None 2024-03-04T17:39:17.149 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/%26 2024-03-04T17:39:17.149 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MTwraH+kI8ahx+B3oAW3ie/JhYY= 2024-03-04T17:39:17.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MTwraH+kI8ahx+B3oAW3ie/JhYY=', 'Content-Length': '0'} 2024-03-04T17:39:17.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003dce790d05a16773-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T17:39:17.152 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:17.152 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:17.152 DEBUG:boto:Method: GET 2024-03-04T17:39:17.153 DEBUG:boto:Path: /ppfahn-12/%26 2024-03-04T17:39:17.153 DEBUG:boto:Data: 2024-03-04T17:39:17.153 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.153 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.153 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.153 DEBUG:boto:Params: {} 2024-03-04T17:39:17.153 DEBUG:boto:Token: None 2024-03-04T17:39:17.153 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/%26 2024-03-04T17:39:17.153 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Z001NuggPl2nXajEu/6H0sdtq1c= 2024-03-04T17:39:17.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Z001NuggPl2nXajEu/6H0sdtq1c=', 'Content-Length': '0'} 2024-03-04T17:39:17.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050ea78455f53ae2f-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.200 DEBUG:boto:path=/ppfahn-12/%26 2024-03-04T17:39:17.200 DEBUG:boto:auth_path=/ppfahn-12/%26 2024-03-04T17:39:17.200 DEBUG:boto:Method: GET 2024-03-04T17:39:17.200 DEBUG:boto:Path: /ppfahn-12/%26 2024-03-04T17:39:17.200 DEBUG:boto:Data: 2024-03-04T17:39:17.201 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.201 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.201 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.201 DEBUG:boto:Params: {} 2024-03-04T17:39:17.201 DEBUG:boto:Token: None 2024-03-04T17:39:17.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/%26 2024-03-04T17:39:17.201 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Z001NuggPl2nXajEu/6H0sdtq1c= 2024-03-04T17:39:17.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Z001NuggPl2nXajEu/6H0sdtq1c=', 'Content-Length': '0'} 2024-03-04T17:39:17.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063878cf0a24f7d5a-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.205 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.205 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.205 DEBUG:boto:path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:17.205 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:17.205 DEBUG:boto:Method: GET 2024-03-04T17:39:17.205 DEBUG:boto:Path: /ppfahn-12/?prefix=%26 2024-03-04T17:39:17.205 DEBUG:boto:Data: 2024-03-04T17:39:17.205 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.205 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.205 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.205 DEBUG:boto:Params: {} 2024-03-04T17:39:17.206 DEBUG:boto:Token: None 2024-03-04T17:39:17.206 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.206 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024c5d06edad77474-0065e60745-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.210 DEBUG:boto:b'ppfahn-12&1000false&2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.211 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.211 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.211 DEBUG:boto:path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:17.211 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%26 2024-03-04T17:39:17.211 DEBUG:boto:Method: GET 2024-03-04T17:39:17.211 DEBUG:boto:Path: /ppfahn-12/?prefix=%26 2024-03-04T17:39:17.211 DEBUG:boto:Data: 2024-03-04T17:39:17.211 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.211 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.211 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.211 DEBUG:boto:Params: {} 2024-03-04T17:39:17.212 DEBUG:boto:Token: None 2024-03-04T17:39:17.212 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.212 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cae63a3c765d2b99-0065e60745-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.260 DEBUG:boto:b'ppfahn-12&1000false&2024-03-04T17:36:30.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:17.260 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:17.261 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:17.261 DEBUG:boto:path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:17.261 DEBUG:boto:auth_path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:17.261 DEBUG:boto:Method: GET 2024-03-04T17:39:17.261 DEBUG:boto:Path: /ppfahn-12/.?versionId=null 2024-03-04T17:39:17.261 DEBUG:boto:Data: 2024-03-04T17:39:17.261 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.261 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.261 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.261 DEBUG:boto:Params: {} 2024-03-04T17:39:17.261 DEBUG:boto:Token: None 2024-03-04T17:39:17.262 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.?versionId=null 2024-03-04T17:39:17.262 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BRtJEoF2hB9jtnZeQuKpQAK2Jk8= 2024-03-04T17:39:17.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BRtJEoF2hB9jtnZeQuKpQAK2Jk8=', 'Content-Length': '0'} 2024-03-04T17:39:17.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006650bb90e3556af-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.265 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:17.265 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:17.265 DEBUG:boto:path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:17.265 DEBUG:boto:auth_path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:17.265 DEBUG:boto:Method: GET 2024-03-04T17:39:17.265 DEBUG:boto:Path: /ppfahn-12/.?versionId=null 2024-03-04T17:39:17.265 DEBUG:boto:Data: 2024-03-04T17:39:17.265 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.266 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.266 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.266 DEBUG:boto:Params: {} 2024-03-04T17:39:17.266 DEBUG:boto:Token: None 2024-03-04T17:39:17.266 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.?versionId=null 2024-03-04T17:39:17.266 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BRtJEoF2hB9jtnZeQuKpQAK2Jk8= 2024-03-04T17:39:17.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BRtJEoF2hB9jtnZeQuKpQAK2Jk8=', 'Content-Length': '0'} 2024-03-04T17:39:17.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c217eeae5a115fdf-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.269 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:17.269 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:17.270 DEBUG:boto:path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:17.270 DEBUG:boto:auth_path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:17.270 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.270 DEBUG:boto:Path: /ppfahn-12/.?versionId=null 2024-03-04T17:39:17.270 DEBUG:boto:Data: 2024-03-04T17:39:17.270 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.270 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.270 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.270 DEBUG:boto:Params: {} 2024-03-04T17:39:17.270 DEBUG:boto:Token: None 2024-03-04T17:39:17.270 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.?versionId=null 2024-03-04T17:39:17.271 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sxxy65KTL5qBm7hXLCPAb7x0SXM= 2024-03-04T17:39:17.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sxxy65KTL5qBm7hXLCPAb7x0SXM=', 'Content-Length': '0'} 2024-03-04T17:39:17.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d0dfbe5cdf24468-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.274 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:17.274 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:17.274 DEBUG:boto:path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:17.274 DEBUG:boto:auth_path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:17.274 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.274 DEBUG:boto:Path: /ppfahn-12/.?versionId=null 2024-03-04T17:39:17.274 DEBUG:boto:Data: 2024-03-04T17:39:17.275 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.275 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.275 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.275 DEBUG:boto:Params: {} 2024-03-04T17:39:17.275 DEBUG:boto:Token: None 2024-03-04T17:39:17.275 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.?versionId=null 2024-03-04T17:39:17.275 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sxxy65KTL5qBm7hXLCPAb7x0SXM= 2024-03-04T17:39:17.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sxxy65KTL5qBm7hXLCPAb7x0SXM=', 'Content-Length': '0'} 2024-03-04T17:39:17.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e74316472dce84df-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:17.278 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:17.278 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:17.278 DEBUG:boto:path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:17.278 DEBUG:boto:auth_path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:17.278 DEBUG:boto:Method: GET 2024-03-04T17:39:17.279 DEBUG:boto:Path: /ppfahn-12/.?versionId=null 2024-03-04T17:39:17.279 DEBUG:boto:Data: 2024-03-04T17:39:17.279 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.279 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.279 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.279 DEBUG:boto:Params: {} 2024-03-04T17:39:17.279 DEBUG:boto:Token: None 2024-03-04T17:39:17.279 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.?versionId=null 2024-03-04T17:39:17.280 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BRtJEoF2hB9jtnZeQuKpQAK2Jk8= 2024-03-04T17:39:17.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BRtJEoF2hB9jtnZeQuKpQAK2Jk8=', 'Content-Length': '0'} 2024-03-04T17:39:17.283 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6913b9df780d44f-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.324 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:17.324 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:17.324 DEBUG:boto:path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:17.324 DEBUG:boto:auth_path=/ppfahn-12/.?versionId=null 2024-03-04T17:39:17.324 DEBUG:boto:Method: GET 2024-03-04T17:39:17.324 DEBUG:boto:Path: /ppfahn-12/.?versionId=null 2024-03-04T17:39:17.325 DEBUG:boto:Data: 2024-03-04T17:39:17.325 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.325 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.325 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.325 DEBUG:boto:Params: {} 2024-03-04T17:39:17.325 DEBUG:boto:Token: None 2024-03-04T17:39:17.325 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.?versionId=null 2024-03-04T17:39:17.325 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BRtJEoF2hB9jtnZeQuKpQAK2Jk8= 2024-03-04T17:39:17.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BRtJEoF2hB9jtnZeQuKpQAK2Jk8=', 'Content-Length': '0'} 2024-03-04T17:39:17.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e1e675d03b0417d-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.329 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.329 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.329 DEBUG:boto:path=/ppfahn-12/?prefix=. 2024-03-04T17:39:17.329 DEBUG:boto:auth_path=/ppfahn-12/?prefix=. 2024-03-04T17:39:17.329 DEBUG:boto:Method: GET 2024-03-04T17:39:17.329 DEBUG:boto:Path: /ppfahn-12/?prefix=. 2024-03-04T17:39:17.329 DEBUG:boto:Data: 2024-03-04T17:39:17.330 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.330 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.330 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.330 DEBUG:boto:Params: {} 2024-03-04T17:39:17.330 DEBUG:boto:Token: None 2024-03-04T17:39:17.330 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.330 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000643d6ac567655b01-0065e60745-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.335 DEBUG:boto:b'ppfahn-12.1000false.2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.336 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.336 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.336 DEBUG:boto:path=/ppfahn-12/?prefix=. 2024-03-04T17:39:17.336 DEBUG:boto:auth_path=/ppfahn-12/?prefix=. 2024-03-04T17:39:17.336 DEBUG:boto:Method: GET 2024-03-04T17:39:17.336 DEBUG:boto:Path: /ppfahn-12/?prefix=. 2024-03-04T17:39:17.336 DEBUG:boto:Data: 2024-03-04T17:39:17.336 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.336 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.336 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.336 DEBUG:boto:Params: {} 2024-03-04T17:39:17.337 DEBUG:boto:Token: None 2024-03-04T17:39:17.337 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.337 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000383b8af539528871-0065e60745-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.341 DEBUG:boto:b'ppfahn-12.1000false.2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.342 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:17.342 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:17.342 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.342 DEBUG:boto:Path: /ppfahn-12/. 2024-03-04T17:39:17.342 DEBUG:boto:Data: 2024-03-04T17:39:17.342 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.342 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.342 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.342 DEBUG:boto:Params: {} 2024-03-04T17:39:17.343 DEBUG:boto:Token: None 2024-03-04T17:39:17.343 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/. 2024-03-04T17:39:17.343 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XnduULALxCVeDavQU0MIaxrHr24= 2024-03-04T17:39:17.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XnduULALxCVeDavQU0MIaxrHr24=', 'Content-Length': '0'} 2024-03-04T17:39:17.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed66035353347648-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.346 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:17.346 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:17.346 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.346 DEBUG:boto:Path: /ppfahn-12/. 2024-03-04T17:39:17.346 DEBUG:boto:Data: 2024-03-04T17:39:17.346 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.346 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.346 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.346 DEBUG:boto:Params: {} 2024-03-04T17:39:17.346 DEBUG:boto:Token: None 2024-03-04T17:39:17.347 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/. 2024-03-04T17:39:17.347 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XnduULALxCVeDavQU0MIaxrHr24= 2024-03-04T17:39:17.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XnduULALxCVeDavQU0MIaxrHr24=', 'Content-Length': '0'} 2024-03-04T17:39:17.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008b73d6a4965493a-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T17:39:17.350 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:17.350 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:17.350 DEBUG:boto:Method: GET 2024-03-04T17:39:17.350 DEBUG:boto:Path: /ppfahn-12/. 2024-03-04T17:39:17.350 DEBUG:boto:Data: 2024-03-04T17:39:17.350 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.350 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.350 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.350 DEBUG:boto:Params: {} 2024-03-04T17:39:17.351 DEBUG:boto:Token: None 2024-03-04T17:39:17.351 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/. 2024-03-04T17:39:17.351 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E3JeGYpneNvfmfGYmw1EAcB+U3Q= 2024-03-04T17:39:17.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E3JeGYpneNvfmfGYmw1EAcB+U3Q=', 'Content-Length': '0'} 2024-03-04T17:39:17.354 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000566a122a41f65f05-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.396 DEBUG:boto:path=/ppfahn-12/. 2024-03-04T17:39:17.396 DEBUG:boto:auth_path=/ppfahn-12/. 2024-03-04T17:39:17.396 DEBUG:boto:Method: GET 2024-03-04T17:39:17.396 DEBUG:boto:Path: /ppfahn-12/. 2024-03-04T17:39:17.396 DEBUG:boto:Data: 2024-03-04T17:39:17.397 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.397 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.397 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.397 DEBUG:boto:Params: {} 2024-03-04T17:39:17.397 DEBUG:boto:Token: None 2024-03-04T17:39:17.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/. 2024-03-04T17:39:17.397 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E3JeGYpneNvfmfGYmw1EAcB+U3Q= 2024-03-04T17:39:17.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E3JeGYpneNvfmfGYmw1EAcB+U3Q=', 'Content-Length': '0'} 2024-03-04T17:39:17.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dc9a685ac81c979-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.400 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.400 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.400 DEBUG:boto:path=/ppfahn-12/?prefix=. 2024-03-04T17:39:17.400 DEBUG:boto:auth_path=/ppfahn-12/?prefix=. 2024-03-04T17:39:17.401 DEBUG:boto:Method: GET 2024-03-04T17:39:17.401 DEBUG:boto:Path: /ppfahn-12/?prefix=. 2024-03-04T17:39:17.401 DEBUG:boto:Data: 2024-03-04T17:39:17.401 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.401 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.401 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.401 DEBUG:boto:Params: {} 2024-03-04T17:39:17.401 DEBUG:boto:Token: None 2024-03-04T17:39:17.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.402 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dfd4fa64afda73a-0065e60745-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.407 DEBUG:boto:b'ppfahn-12.1000false.2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.408 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.408 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.408 DEBUG:boto:path=/ppfahn-12/?prefix=. 2024-03-04T17:39:17.408 DEBUG:boto:auth_path=/ppfahn-12/?prefix=. 2024-03-04T17:39:17.408 DEBUG:boto:Method: GET 2024-03-04T17:39:17.408 DEBUG:boto:Path: /ppfahn-12/?prefix=. 2024-03-04T17:39:17.408 DEBUG:boto:Data: 2024-03-04T17:39:17.409 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.409 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.409 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.409 DEBUG:boto:Params: {} 2024-03-04T17:39:17.409 DEBUG:boto:Token: None 2024-03-04T17:39:17.409 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.409 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dbf3e7e3644954b-0065e60745-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.415 DEBUG:boto:b'ppfahn-12.1000false.2024-03-04T17:36:30.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:17.416 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:17.416 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:17.416 DEBUG:boto:path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:17.416 DEBUG:boto:auth_path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:17.416 DEBUG:boto:Method: GET 2024-03-04T17:39:17.416 DEBUG:boto:Path: /ppfahn-12/..?versionId=null 2024-03-04T17:39:17.416 DEBUG:boto:Data: 2024-03-04T17:39:17.416 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.416 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.417 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.417 DEBUG:boto:Params: {} 2024-03-04T17:39:17.417 DEBUG:boto:Token: None 2024-03-04T17:39:17.417 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/..?versionId=null 2024-03-04T17:39:17.417 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+yTjGWLzXOdViDm3ZjMtLkgAluY= 2024-03-04T17:39:17.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+yTjGWLzXOdViDm3ZjMtLkgAluY=', 'Content-Length': '0'} 2024-03-04T17:39:17.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d284399883b51801-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.420 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:17.420 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:17.420 DEBUG:boto:path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:17.421 DEBUG:boto:auth_path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:17.421 DEBUG:boto:Method: GET 2024-03-04T17:39:17.421 DEBUG:boto:Path: /ppfahn-12/..?versionId=null 2024-03-04T17:39:17.421 DEBUG:boto:Data: 2024-03-04T17:39:17.421 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.421 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.421 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.421 DEBUG:boto:Params: {} 2024-03-04T17:39:17.421 DEBUG:boto:Token: None 2024-03-04T17:39:17.421 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/..?versionId=null 2024-03-04T17:39:17.422 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+yTjGWLzXOdViDm3ZjMtLkgAluY= 2024-03-04T17:39:17.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+yTjGWLzXOdViDm3ZjMtLkgAluY=', 'Content-Length': '0'} 2024-03-04T17:39:17.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032ae63fd6394570f-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.425 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:17.425 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:17.425 DEBUG:boto:path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:17.425 DEBUG:boto:auth_path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:17.425 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.425 DEBUG:boto:Path: /ppfahn-12/..?versionId=null 2024-03-04T17:39:17.425 DEBUG:boto:Data: 2024-03-04T17:39:17.425 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.425 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.425 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.426 DEBUG:boto:Params: {} 2024-03-04T17:39:17.426 DEBUG:boto:Token: None 2024-03-04T17:39:17.426 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/..?versionId=null 2024-03-04T17:39:17.426 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:02SMQkdn4U8u7JEbMul+l6+XWbU= 2024-03-04T17:39:17.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:02SMQkdn4U8u7JEbMul+l6+XWbU=', 'Content-Length': '0'} 2024-03-04T17:39:17.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab4bad7f3daae09a-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.429 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:17.429 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:17.429 DEBUG:boto:path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:17.429 DEBUG:boto:auth_path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:17.429 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.429 DEBUG:boto:Path: /ppfahn-12/..?versionId=null 2024-03-04T17:39:17.429 DEBUG:boto:Data: 2024-03-04T17:39:17.429 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.430 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.430 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.430 DEBUG:boto:Params: {} 2024-03-04T17:39:17.430 DEBUG:boto:Token: None 2024-03-04T17:39:17.430 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/..?versionId=null 2024-03-04T17:39:17.430 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:02SMQkdn4U8u7JEbMul+l6+XWbU= 2024-03-04T17:39:17.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:02SMQkdn4U8u7JEbMul+l6+XWbU=', 'Content-Length': '0'} 2024-03-04T17:39:17.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5d0d2f8aa55d2c9-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:17.433 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:17.433 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:17.433 DEBUG:boto:path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:17.433 DEBUG:boto:auth_path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:17.433 DEBUG:boto:Method: GET 2024-03-04T17:39:17.433 DEBUG:boto:Path: /ppfahn-12/..?versionId=null 2024-03-04T17:39:17.433 DEBUG:boto:Data: 2024-03-04T17:39:17.434 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.434 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.434 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.434 DEBUG:boto:Params: {} 2024-03-04T17:39:17.434 DEBUG:boto:Token: None 2024-03-04T17:39:17.434 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/..?versionId=null 2024-03-04T17:39:17.434 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+yTjGWLzXOdViDm3ZjMtLkgAluY= 2024-03-04T17:39:17.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+yTjGWLzXOdViDm3ZjMtLkgAluY=', 'Content-Length': '0'} 2024-03-04T17:39:17.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a7c81a71e57f6cd-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.480 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:17.480 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:17.480 DEBUG:boto:path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:17.480 DEBUG:boto:auth_path=/ppfahn-12/..?versionId=null 2024-03-04T17:39:17.481 DEBUG:boto:Method: GET 2024-03-04T17:39:17.481 DEBUG:boto:Path: /ppfahn-12/..?versionId=null 2024-03-04T17:39:17.481 DEBUG:boto:Data: 2024-03-04T17:39:17.481 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.481 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.481 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.481 DEBUG:boto:Params: {} 2024-03-04T17:39:17.481 DEBUG:boto:Token: None 2024-03-04T17:39:17.481 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/..?versionId=null 2024-03-04T17:39:17.481 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+yTjGWLzXOdViDm3ZjMtLkgAluY= 2024-03-04T17:39:17.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+yTjGWLzXOdViDm3ZjMtLkgAluY=', 'Content-Length': '0'} 2024-03-04T17:39:17.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000321e95c6237c53f0-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.485 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.485 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.485 DEBUG:boto:path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:17.485 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:17.485 DEBUG:boto:Method: GET 2024-03-04T17:39:17.485 DEBUG:boto:Path: /ppfahn-12/?prefix=.. 2024-03-04T17:39:17.485 DEBUG:boto:Data: 2024-03-04T17:39:17.485 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.485 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.486 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.486 DEBUG:boto:Params: {} 2024-03-04T17:39:17.486 DEBUG:boto:Token: None 2024-03-04T17:39:17.486 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.486 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b6ae3fb1f920429-0065e60745-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.491 DEBUG:boto:b'ppfahn-12..1000false..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.491 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.491 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.492 DEBUG:boto:path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:17.492 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:17.492 DEBUG:boto:Method: GET 2024-03-04T17:39:17.492 DEBUG:boto:Path: /ppfahn-12/?prefix=.. 2024-03-04T17:39:17.492 DEBUG:boto:Data: 2024-03-04T17:39:17.492 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.492 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.492 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.492 DEBUG:boto:Params: {} 2024-03-04T17:39:17.492 DEBUG:boto:Token: None 2024-03-04T17:39:17.492 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.493 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da6691599e1b6f5-0065e60745-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.497 DEBUG:boto:b'ppfahn-12..1000false..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.497 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:17.498 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:17.498 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.498 DEBUG:boto:Path: /ppfahn-12/.. 2024-03-04T17:39:17.498 DEBUG:boto:Data: 2024-03-04T17:39:17.498 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.498 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.498 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.498 DEBUG:boto:Params: {} 2024-03-04T17:39:17.498 DEBUG:boto:Token: None 2024-03-04T17:39:17.498 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.. 2024-03-04T17:39:17.498 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eKb9Jtqs2FGDIS9EZLKE3VGJBH0= 2024-03-04T17:39:17.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eKb9Jtqs2FGDIS9EZLKE3VGJBH0=', 'Content-Length': '0'} 2024-03-04T17:39:17.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036de12fdffea32ff-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.501 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:17.502 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:17.502 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.502 DEBUG:boto:Path: /ppfahn-12/.. 2024-03-04T17:39:17.502 DEBUG:boto:Data: 2024-03-04T17:39:17.502 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.502 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.502 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.502 DEBUG:boto:Params: {} 2024-03-04T17:39:17.502 DEBUG:boto:Token: None 2024-03-04T17:39:17.502 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.. 2024-03-04T17:39:17.502 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eKb9Jtqs2FGDIS9EZLKE3VGJBH0= 2024-03-04T17:39:17.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eKb9Jtqs2FGDIS9EZLKE3VGJBH0=', 'Content-Length': '0'} 2024-03-04T17:39:17.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000077d418842bdd2a4-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T17:39:17.505 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:17.506 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:17.506 DEBUG:boto:Method: GET 2024-03-04T17:39:17.506 DEBUG:boto:Path: /ppfahn-12/.. 2024-03-04T17:39:17.506 DEBUG:boto:Data: 2024-03-04T17:39:17.506 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.506 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.506 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.506 DEBUG:boto:Params: {} 2024-03-04T17:39:17.506 DEBUG:boto:Token: None 2024-03-04T17:39:17.506 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.. 2024-03-04T17:39:17.506 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:O8VUQrCfVfBfgSILI5LTeQpxoLo= 2024-03-04T17:39:17.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:O8VUQrCfVfBfgSILI5LTeQpxoLo=', 'Content-Length': '0'} 2024-03-04T17:39:17.509 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026f2102c203fdbf6-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.552 DEBUG:boto:path=/ppfahn-12/.. 2024-03-04T17:39:17.552 DEBUG:boto:auth_path=/ppfahn-12/.. 2024-03-04T17:39:17.552 DEBUG:boto:Method: GET 2024-03-04T17:39:17.552 DEBUG:boto:Path: /ppfahn-12/.. 2024-03-04T17:39:17.553 DEBUG:boto:Data: 2024-03-04T17:39:17.553 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.553 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.553 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.553 DEBUG:boto:Params: {} 2024-03-04T17:39:17.553 DEBUG:boto:Token: None 2024-03-04T17:39:17.553 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.. 2024-03-04T17:39:17.553 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:O8VUQrCfVfBfgSILI5LTeQpxoLo= 2024-03-04T17:39:17.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:O8VUQrCfVfBfgSILI5LTeQpxoLo=', 'Content-Length': '0'} 2024-03-04T17:39:17.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c43f8329ba7d8048-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.556 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.556 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.557 DEBUG:boto:path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:17.557 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:17.557 DEBUG:boto:Method: GET 2024-03-04T17:39:17.557 DEBUG:boto:Path: /ppfahn-12/?prefix=.. 2024-03-04T17:39:17.557 DEBUG:boto:Data: 2024-03-04T17:39:17.557 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.557 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.557 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.557 DEBUG:boto:Params: {} 2024-03-04T17:39:17.557 DEBUG:boto:Token: None 2024-03-04T17:39:17.558 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.558 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031451d959c1ec38c-0065e60745-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.562 DEBUG:boto:b'ppfahn-12..1000false..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.563 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.563 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.563 DEBUG:boto:path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:17.563 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.. 2024-03-04T17:39:17.563 DEBUG:boto:Method: GET 2024-03-04T17:39:17.563 DEBUG:boto:Path: /ppfahn-12/?prefix=.. 2024-03-04T17:39:17.563 DEBUG:boto:Data: 2024-03-04T17:39:17.563 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.563 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.563 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.563 DEBUG:boto:Params: {} 2024-03-04T17:39:17.564 DEBUG:boto:Token: None 2024-03-04T17:39:17.564 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.564 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004910f0fdca38b351-0065e60745-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.616 DEBUG:boto:b'ppfahn-12..1000false..2024-03-04T17:36:30.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:17.616 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:17.617 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:17.617 DEBUG:boto:path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:17.617 DEBUG:boto:auth_path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:17.617 DEBUG:boto:Method: GET 2024-03-04T17:39:17.617 DEBUG:boto:Path: /ppfahn-12/...?versionId=null 2024-03-04T17:39:17.617 DEBUG:boto:Data: 2024-03-04T17:39:17.617 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.617 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.617 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.618 DEBUG:boto:Params: {} 2024-03-04T17:39:17.618 DEBUG:boto:Token: None 2024-03-04T17:39:17.618 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/...?versionId=null 2024-03-04T17:39:17.618 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:N5/bg4AZRpYHxVDgFkimZccMVII= 2024-03-04T17:39:17.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:N5/bg4AZRpYHxVDgFkimZccMVII=', 'Content-Length': '0'} 2024-03-04T17:39:17.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7e5ac502f93ca97-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.621 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:17.621 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:17.621 DEBUG:boto:path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:17.622 DEBUG:boto:auth_path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:17.622 DEBUG:boto:Method: GET 2024-03-04T17:39:17.622 DEBUG:boto:Path: /ppfahn-12/...?versionId=null 2024-03-04T17:39:17.622 DEBUG:boto:Data: 2024-03-04T17:39:17.622 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.622 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.622 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.622 DEBUG:boto:Params: {} 2024-03-04T17:39:17.622 DEBUG:boto:Token: None 2024-03-04T17:39:17.622 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/...?versionId=null 2024-03-04T17:39:17.623 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:N5/bg4AZRpYHxVDgFkimZccMVII= 2024-03-04T17:39:17.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:N5/bg4AZRpYHxVDgFkimZccMVII=', 'Content-Length': '0'} 2024-03-04T17:39:17.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043f75bfc3993d727-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.626 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:17.626 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:17.626 DEBUG:boto:path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:17.626 DEBUG:boto:auth_path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:17.626 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.626 DEBUG:boto:Path: /ppfahn-12/...?versionId=null 2024-03-04T17:39:17.626 DEBUG:boto:Data: 2024-03-04T17:39:17.626 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.626 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.627 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.627 DEBUG:boto:Params: {} 2024-03-04T17:39:17.627 DEBUG:boto:Token: None 2024-03-04T17:39:17.627 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/...?versionId=null 2024-03-04T17:39:17.627 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:detVGAO0bjlHjP6of6BNfQGkjJA= 2024-03-04T17:39:17.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:detVGAO0bjlHjP6of6BNfQGkjJA=', 'Content-Length': '0'} 2024-03-04T17:39:17.630 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006faada25b505fa1f-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.630 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:17.630 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:17.630 DEBUG:boto:path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:17.630 DEBUG:boto:auth_path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:17.630 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.631 DEBUG:boto:Path: /ppfahn-12/...?versionId=null 2024-03-04T17:39:17.631 DEBUG:boto:Data: 2024-03-04T17:39:17.631 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.631 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.631 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.631 DEBUG:boto:Params: {} 2024-03-04T17:39:17.631 DEBUG:boto:Token: None 2024-03-04T17:39:17.631 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/...?versionId=null 2024-03-04T17:39:17.631 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:detVGAO0bjlHjP6of6BNfQGkjJA= 2024-03-04T17:39:17.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:detVGAO0bjlHjP6of6BNfQGkjJA=', 'Content-Length': '0'} 2024-03-04T17:39:17.634 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096632121b29e5fa1-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:17.634 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:17.635 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:17.635 DEBUG:boto:path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:17.635 DEBUG:boto:auth_path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:17.635 DEBUG:boto:Method: GET 2024-03-04T17:39:17.635 DEBUG:boto:Path: /ppfahn-12/...?versionId=null 2024-03-04T17:39:17.635 DEBUG:boto:Data: 2024-03-04T17:39:17.635 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.635 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.635 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.635 DEBUG:boto:Params: {} 2024-03-04T17:39:17.635 DEBUG:boto:Token: None 2024-03-04T17:39:17.636 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/...?versionId=null 2024-03-04T17:39:17.636 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:N5/bg4AZRpYHxVDgFkimZccMVII= 2024-03-04T17:39:17.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:N5/bg4AZRpYHxVDgFkimZccMVII=', 'Content-Length': '0'} 2024-03-04T17:39:17.638 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b21e96be23fd90d0-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.684 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:17.684 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:17.684 DEBUG:boto:path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:17.684 DEBUG:boto:auth_path=/ppfahn-12/...?versionId=null 2024-03-04T17:39:17.685 DEBUG:boto:Method: GET 2024-03-04T17:39:17.685 DEBUG:boto:Path: /ppfahn-12/...?versionId=null 2024-03-04T17:39:17.685 DEBUG:boto:Data: 2024-03-04T17:39:17.685 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.685 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.685 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.685 DEBUG:boto:Params: {} 2024-03-04T17:39:17.685 DEBUG:boto:Token: None 2024-03-04T17:39:17.685 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/...?versionId=null 2024-03-04T17:39:17.686 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:N5/bg4AZRpYHxVDgFkimZccMVII= 2024-03-04T17:39:17.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:N5/bg4AZRpYHxVDgFkimZccMVII=', 'Content-Length': '0'} 2024-03-04T17:39:17.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051091b9115be84a1-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.689 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.689 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.689 DEBUG:boto:path=/ppfahn-12/?prefix=... 2024-03-04T17:39:17.689 DEBUG:boto:auth_path=/ppfahn-12/?prefix=... 2024-03-04T17:39:17.690 DEBUG:boto:Method: GET 2024-03-04T17:39:17.690 DEBUG:boto:Path: /ppfahn-12/?prefix=... 2024-03-04T17:39:17.690 DEBUG:boto:Data: 2024-03-04T17:39:17.690 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.690 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.690 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.690 DEBUG:boto:Params: {} 2024-03-04T17:39:17.690 DEBUG:boto:Token: None 2024-03-04T17:39:17.690 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.690 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e97ffcf3fd89884-0065e60745-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.695 DEBUG:boto:b'ppfahn-12...1000false...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.696 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.696 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.696 DEBUG:boto:path=/ppfahn-12/?prefix=... 2024-03-04T17:39:17.696 DEBUG:boto:auth_path=/ppfahn-12/?prefix=... 2024-03-04T17:39:17.696 DEBUG:boto:Method: GET 2024-03-04T17:39:17.696 DEBUG:boto:Path: /ppfahn-12/?prefix=... 2024-03-04T17:39:17.696 DEBUG:boto:Data: 2024-03-04T17:39:17.697 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.697 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.697 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.697 DEBUG:boto:Params: {} 2024-03-04T17:39:17.697 DEBUG:boto:Token: None 2024-03-04T17:39:17.697 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.697 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f826d7ba9ba29c15-0065e60745-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.744 DEBUG:boto:b'ppfahn-12...1000false...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.744 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:17.744 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:17.745 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.745 DEBUG:boto:Path: /ppfahn-12/... 2024-03-04T17:39:17.745 DEBUG:boto:Data: 2024-03-04T17:39:17.745 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.745 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.745 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.745 DEBUG:boto:Params: {} 2024-03-04T17:39:17.745 DEBUG:boto:Token: None 2024-03-04T17:39:17.745 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/... 2024-03-04T17:39:17.746 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ynwK0n8sYeqSftkmJqJQ/t1QlJ0= 2024-03-04T17:39:17.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ynwK0n8sYeqSftkmJqJQ/t1QlJ0=', 'Content-Length': '0'} 2024-03-04T17:39:17.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035fede710780da58-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.749 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:17.749 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:17.749 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.749 DEBUG:boto:Path: /ppfahn-12/... 2024-03-04T17:39:17.749 DEBUG:boto:Data: 2024-03-04T17:39:17.749 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.749 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.750 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.750 DEBUG:boto:Params: {} 2024-03-04T17:39:17.750 DEBUG:boto:Token: None 2024-03-04T17:39:17.750 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/... 2024-03-04T17:39:17.750 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ynwK0n8sYeqSftkmJqJQ/t1QlJ0= 2024-03-04T17:39:17.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ynwK0n8sYeqSftkmJqJQ/t1QlJ0=', 'Content-Length': '0'} 2024-03-04T17:39:17.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d630e9f571dfada8-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T17:39:17.753 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:17.754 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:17.754 DEBUG:boto:Method: GET 2024-03-04T17:39:17.754 DEBUG:boto:Path: /ppfahn-12/... 2024-03-04T17:39:17.754 DEBUG:boto:Data: 2024-03-04T17:39:17.754 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.754 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.754 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.754 DEBUG:boto:Params: {} 2024-03-04T17:39:17.755 DEBUG:boto:Token: None 2024-03-04T17:39:17.755 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/... 2024-03-04T17:39:17.755 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2hXSTM4P2VpPyud4+KLl8zV0vwQ= 2024-03-04T17:39:17.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2hXSTM4P2VpPyud4+KLl8zV0vwQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.758 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064ae6df4dcb195b0-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.758 DEBUG:boto:path=/ppfahn-12/... 2024-03-04T17:39:17.758 DEBUG:boto:auth_path=/ppfahn-12/... 2024-03-04T17:39:17.758 DEBUG:boto:Method: GET 2024-03-04T17:39:17.758 DEBUG:boto:Path: /ppfahn-12/... 2024-03-04T17:39:17.758 DEBUG:boto:Data: 2024-03-04T17:39:17.759 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.759 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.759 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.759 DEBUG:boto:Params: {} 2024-03-04T17:39:17.759 DEBUG:boto:Token: None 2024-03-04T17:39:17.759 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/... 2024-03-04T17:39:17.759 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2hXSTM4P2VpPyud4+KLl8zV0vwQ= 2024-03-04T17:39:17.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2hXSTM4P2VpPyud4+KLl8zV0vwQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cf6de49131beb0c-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.763 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.763 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.763 DEBUG:boto:path=/ppfahn-12/?prefix=... 2024-03-04T17:39:17.763 DEBUG:boto:auth_path=/ppfahn-12/?prefix=... 2024-03-04T17:39:17.763 DEBUG:boto:Method: GET 2024-03-04T17:39:17.763 DEBUG:boto:Path: /ppfahn-12/?prefix=... 2024-03-04T17:39:17.763 DEBUG:boto:Data: 2024-03-04T17:39:17.763 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.763 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.763 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.763 DEBUG:boto:Params: {} 2024-03-04T17:39:17.763 DEBUG:boto:Token: None 2024-03-04T17:39:17.764 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.764 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6eb94a02d2a84d6-0065e60745-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.812 DEBUG:boto:b'ppfahn-12...1000false...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.812 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.813 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.813 DEBUG:boto:path=/ppfahn-12/?prefix=... 2024-03-04T17:39:17.813 DEBUG:boto:auth_path=/ppfahn-12/?prefix=... 2024-03-04T17:39:17.813 DEBUG:boto:Method: GET 2024-03-04T17:39:17.813 DEBUG:boto:Path: /ppfahn-12/?prefix=... 2024-03-04T17:39:17.813 DEBUG:boto:Data: 2024-03-04T17:39:17.813 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.813 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.813 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.813 DEBUG:boto:Params: {} 2024-03-04T17:39:17.813 DEBUG:boto:Token: None 2024-03-04T17:39:17.814 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.814 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cb525d67ebb1931-0065e60745-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.819 DEBUG:boto:b'ppfahn-12...1000false...2024-03-04T17:36:30.869Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:17.819 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:17.819 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:17.820 DEBUG:boto:path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.820 DEBUG:boto:auth_path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.820 DEBUG:boto:Method: GET 2024-03-04T17:39:17.820 DEBUG:boto:Path: /ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.820 DEBUG:boto:Data: 2024-03-04T17:39:17.820 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.820 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.820 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.820 DEBUG:boto:Params: {} 2024-03-04T17:39:17.821 DEBUG:boto:Token: None 2024-03-04T17:39:17.821 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.821 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:od2oLdekWtiDFwhayHREDnjMjpY= 2024-03-04T17:39:17.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:od2oLdekWtiDFwhayHREDnjMjpY=', 'Content-Length': '0'} 2024-03-04T17:39:17.824 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000270e0f7dfbfe9fc7-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.824 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:17.824 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:17.824 DEBUG:boto:path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.824 DEBUG:boto:auth_path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.825 DEBUG:boto:Method: GET 2024-03-04T17:39:17.825 DEBUG:boto:Path: /ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.825 DEBUG:boto:Data: 2024-03-04T17:39:17.825 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.825 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.825 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.825 DEBUG:boto:Params: {} 2024-03-04T17:39:17.825 DEBUG:boto:Token: None 2024-03-04T17:39:17.825 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.825 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:od2oLdekWtiDFwhayHREDnjMjpY= 2024-03-04T17:39:17.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:od2oLdekWtiDFwhayHREDnjMjpY=', 'Content-Length': '0'} 2024-03-04T17:39:17.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b57ba564bb784cb-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.872 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:17.872 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:17.872 DEBUG:boto:path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.872 DEBUG:boto:auth_path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.873 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.873 DEBUG:boto:Path: /ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.873 DEBUG:boto:Data: 2024-03-04T17:39:17.873 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.873 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.873 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.873 DEBUG:boto:Params: {} 2024-03-04T17:39:17.873 DEBUG:boto:Token: None 2024-03-04T17:39:17.873 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.874 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9IY7itSoRXvrSTUCMuow/79H1qQ= 2024-03-04T17:39:17.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9IY7itSoRXvrSTUCMuow/79H1qQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1228fafa387be3c-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.877 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:17.877 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:17.877 DEBUG:boto:path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.877 DEBUG:boto:auth_path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.877 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.877 DEBUG:boto:Path: /ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.877 DEBUG:boto:Data: 2024-03-04T17:39:17.878 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.878 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.878 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.878 DEBUG:boto:Params: {} 2024-03-04T17:39:17.878 DEBUG:boto:Token: None 2024-03-04T17:39:17.878 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.878 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9IY7itSoRXvrSTUCMuow/79H1qQ= 2024-03-04T17:39:17.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9IY7itSoRXvrSTUCMuow/79H1qQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000011bdcc614076605-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:17.881 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:17.881 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:17.881 DEBUG:boto:path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.882 DEBUG:boto:auth_path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.882 DEBUG:boto:Method: GET 2024-03-04T17:39:17.882 DEBUG:boto:Path: /ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.882 DEBUG:boto:Data: 2024-03-04T17:39:17.882 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.882 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.882 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.882 DEBUG:boto:Params: {} 2024-03-04T17:39:17.882 DEBUG:boto:Token: None 2024-03-04T17:39:17.882 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.882 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:od2oLdekWtiDFwhayHREDnjMjpY= 2024-03-04T17:39:17.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:od2oLdekWtiDFwhayHREDnjMjpY=', 'Content-Length': '0'} 2024-03-04T17:39:17.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074a90e3fbb2a9e68-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.886 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:17.886 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:17.886 DEBUG:boto:path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.886 DEBUG:boto:auth_path=/ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.886 DEBUG:boto:Method: GET 2024-03-04T17:39:17.886 DEBUG:boto:Path: /ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.886 DEBUG:boto:Data: 2024-03-04T17:39:17.886 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.886 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.886 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.887 DEBUG:boto:Params: {} 2024-03-04T17:39:17.887 DEBUG:boto:Token: None 2024-03-04T17:39:17.887 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.o?versionId=null 2024-03-04T17:39:17.887 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:od2oLdekWtiDFwhayHREDnjMjpY= 2024-03-04T17:39:17.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:od2oLdekWtiDFwhayHREDnjMjpY=', 'Content-Length': '0'} 2024-03-04T17:39:17.890 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005dedb0c1c829c505-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.890 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.890 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.891 DEBUG:boto:path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:17.891 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:17.891 DEBUG:boto:Method: GET 2024-03-04T17:39:17.891 DEBUG:boto:Path: /ppfahn-12/?prefix=.o 2024-03-04T17:39:17.891 DEBUG:boto:Data: 2024-03-04T17:39:17.891 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.891 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.891 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.891 DEBUG:boto:Params: {} 2024-03-04T17:39:17.891 DEBUG:boto:Token: None 2024-03-04T17:39:17.892 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.892 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4701ceae73817bc-0065e60745-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.940 DEBUG:boto:b'ppfahn-12.o1000false.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.941 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:17.941 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:17.941 DEBUG:boto:path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:17.941 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:17.941 DEBUG:boto:Method: GET 2024-03-04T17:39:17.941 DEBUG:boto:Path: /ppfahn-12/?prefix=.o 2024-03-04T17:39:17.941 DEBUG:boto:Data: 2024-03-04T17:39:17.941 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.941 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.941 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.941 DEBUG:boto:Params: {} 2024-03-04T17:39:17.942 DEBUG:boto:Token: None 2024-03-04T17:39:17.942 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/ 2024-03-04T17:39:17.942 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ= 2024-03-04T17:39:17.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zy3jsvS6jU6qs5u9oBb8sjbb+IQ=', 'Content-Length': '0'} 2024-03-04T17:39:17.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cc3a7d20f70b082-0065e60745-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.947 DEBUG:boto:b'ppfahn-12.o1000false.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:17.947 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:17.947 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:17.947 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.947 DEBUG:boto:Path: /ppfahn-12/.o 2024-03-04T17:39:17.947 DEBUG:boto:Data: 2024-03-04T17:39:17.947 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.948 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.948 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.948 DEBUG:boto:Params: {} 2024-03-04T17:39:17.948 DEBUG:boto:Token: None 2024-03-04T17:39:17.948 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.o 2024-03-04T17:39:17.948 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uRpnyNXt0Y2JTNLvIvEqB89XX1A= 2024-03-04T17:39:17.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uRpnyNXt0Y2JTNLvIvEqB89XX1A=', 'Content-Length': '0'} 2024-03-04T17:39:17.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fb1e6e35294e313-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.951 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:17.951 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:17.951 DEBUG:boto:Method: HEAD 2024-03-04T17:39:17.951 DEBUG:boto:Path: /ppfahn-12/.o 2024-03-04T17:39:17.952 DEBUG:boto:Data: 2024-03-04T17:39:17.952 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.952 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.952 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.952 DEBUG:boto:Params: {} 2024-03-04T17:39:17.952 DEBUG:boto:Token: None 2024-03-04T17:39:17.952 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.o 2024-03-04T17:39:17.952 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uRpnyNXt0Y2JTNLvIvEqB89XX1A= 2024-03-04T17:39:17.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uRpnyNXt0Y2JTNLvIvEqB89XX1A=', 'Content-Length': '0'} 2024-03-04T17:39:17.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a7142ce79ef54bd-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T17:39:17.955 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:17.955 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:17.956 DEBUG:boto:Method: GET 2024-03-04T17:39:17.956 DEBUG:boto:Path: /ppfahn-12/.o 2024-03-04T17:39:17.956 DEBUG:boto:Data: 2024-03-04T17:39:17.956 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.956 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:17.956 DEBUG:boto:Port: 8000 2024-03-04T17:39:17.956 DEBUG:boto:Params: {} 2024-03-04T17:39:17.956 DEBUG:boto:Token: None 2024-03-04T17:39:17.956 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.o 2024-03-04T17:39:17.957 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8fHkVZQPtzL0oUixV9bNWcHQNJk= 2024-03-04T17:39:17.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8fHkVZQPtzL0oUixV9bNWcHQNJk=', 'Content-Length': '0'} 2024-03-04T17:39:17.959 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002eceebc9ea7939b3-0065e60745-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:17.960 DEBUG:boto:path=/ppfahn-12/.o 2024-03-04T17:39:17.960 DEBUG:boto:auth_path=/ppfahn-12/.o 2024-03-04T17:39:17.960 DEBUG:boto:Method: GET 2024-03-04T17:39:17.960 DEBUG:boto:Path: /ppfahn-12/.o 2024-03-04T17:39:17.960 DEBUG:boto:Data: 2024-03-04T17:39:17.960 DEBUG:boto:Headers: {} 2024-03-04T17:39:17.960 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:17.960 DEBUG:boto:Port: 8001 2024-03-04T17:39:17.960 DEBUG:boto:Params: {} 2024-03-04T17:39:17.960 DEBUG:boto:Token: None 2024-03-04T17:39:17.961 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:17 GMT /ppfahn-12/.o 2024-03-04T17:39:17.961 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8fHkVZQPtzL0oUixV9bNWcHQNJk= 2024-03-04T17:39:17.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8fHkVZQPtzL0oUixV9bNWcHQNJk=', 'Content-Length': '0'} 2024-03-04T17:39:17.964 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ac34fdaee52e5e0-0065e60745-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.008 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.008 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.008 DEBUG:boto:path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:18.008 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:18.009 DEBUG:boto:Method: GET 2024-03-04T17:39:18.009 DEBUG:boto:Path: /ppfahn-12/?prefix=.o 2024-03-04T17:39:18.009 DEBUG:boto:Data: 2024-03-04T17:39:18.009 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.009 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.009 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.009 DEBUG:boto:Params: {} 2024-03-04T17:39:18.009 DEBUG:boto:Token: None 2024-03-04T17:39:18.009 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.010 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ddc26b1e970f7ca-0065e60746-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.014 DEBUG:boto:b'ppfahn-12.o1000false.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.014 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.014 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.015 DEBUG:boto:path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:18.015 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o 2024-03-04T17:39:18.015 DEBUG:boto:Method: GET 2024-03-04T17:39:18.015 DEBUG:boto:Path: /ppfahn-12/?prefix=.o 2024-03-04T17:39:18.015 DEBUG:boto:Data: 2024-03-04T17:39:18.015 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.015 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.015 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.015 DEBUG:boto:Params: {} 2024-03-04T17:39:18.015 DEBUG:boto:Token: None 2024-03-04T17:39:18.015 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.016 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094379502c953ff71-0065e60746-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.021 DEBUG:boto:b'ppfahn-12.o1000false.o2024-03-04T17:36:30.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:18.021 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:18.021 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:18.022 DEBUG:boto:path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.022 DEBUG:boto:auth_path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.022 DEBUG:boto:Method: GET 2024-03-04T17:39:18.022 DEBUG:boto:Path: /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.022 DEBUG:boto:Data: 2024-03-04T17:39:18.022 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.022 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.022 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.022 DEBUG:boto:Params: {} 2024-03-04T17:39:18.022 DEBUG:boto:Token: None 2024-03-04T17:39:18.023 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.023 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fqsbdSCiaCb9d9DKagzcDiSi42o= 2024-03-04T17:39:18.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fqsbdSCiaCb9d9DKagzcDiSi42o=', 'Content-Length': '0'} 2024-03-04T17:39:18.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014dc79a959fd3e8b-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.026 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:18.026 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:18.026 DEBUG:boto:path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.026 DEBUG:boto:auth_path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.026 DEBUG:boto:Method: GET 2024-03-04T17:39:18.026 DEBUG:boto:Path: /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.026 DEBUG:boto:Data: 2024-03-04T17:39:18.027 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.027 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.027 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.027 DEBUG:boto:Params: {} 2024-03-04T17:39:18.027 DEBUG:boto:Token: None 2024-03-04T17:39:18.027 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.027 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fqsbdSCiaCb9d9DKagzcDiSi42o= 2024-03-04T17:39:18.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fqsbdSCiaCb9d9DKagzcDiSi42o=', 'Content-Length': '0'} 2024-03-04T17:39:18.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000357686981b236a12-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.030 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:18.030 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:18.031 DEBUG:boto:path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.031 DEBUG:boto:auth_path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.031 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.031 DEBUG:boto:Path: /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.031 DEBUG:boto:Data: 2024-03-04T17:39:18.031 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.031 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.031 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.031 DEBUG:boto:Params: {} 2024-03-04T17:39:18.031 DEBUG:boto:Token: None 2024-03-04T17:39:18.031 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.032 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RAOkgQWA2YKKi8VzycET427uBSY= 2024-03-04T17:39:18.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RAOkgQWA2YKKi8VzycET427uBSY=', 'Content-Length': '0'} 2024-03-04T17:39:18.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad12860581f24cce-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.035 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:18.035 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:18.035 DEBUG:boto:path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.035 DEBUG:boto:auth_path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.035 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.035 DEBUG:boto:Path: /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.035 DEBUG:boto:Data: 2024-03-04T17:39:18.035 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.035 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.035 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.036 DEBUG:boto:Params: {} 2024-03-04T17:39:18.036 DEBUG:boto:Token: None 2024-03-04T17:39:18.036 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.036 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RAOkgQWA2YKKi8VzycET427uBSY= 2024-03-04T17:39:18.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RAOkgQWA2YKKi8VzycET427uBSY=', 'Content-Length': '0'} 2024-03-04T17:39:18.039 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000994d9d5b87f7b7bd-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:18.039 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:18.039 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:18.040 DEBUG:boto:path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.040 DEBUG:boto:auth_path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.040 DEBUG:boto:Method: GET 2024-03-04T17:39:18.040 DEBUG:boto:Path: /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.040 DEBUG:boto:Data: 2024-03-04T17:39:18.040 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.040 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.040 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.040 DEBUG:boto:Params: {} 2024-03-04T17:39:18.040 DEBUG:boto:Token: None 2024-03-04T17:39:18.041 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.041 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fqsbdSCiaCb9d9DKagzcDiSi42o= 2024-03-04T17:39:18.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fqsbdSCiaCb9d9DKagzcDiSi42o=', 'Content-Length': '0'} 2024-03-04T17:39:18.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ac753dc7d89d96b-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.088 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:18.088 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:18.088 DEBUG:boto:path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.088 DEBUG:boto:auth_path=/ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.089 DEBUG:boto:Method: GET 2024-03-04T17:39:18.089 DEBUG:boto:Path: /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.089 DEBUG:boto:Data: 2024-03-04T17:39:18.089 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.089 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.089 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.089 DEBUG:boto:Params: {} 2024-03-04T17:39:18.089 DEBUG:boto:Token: None 2024-03-04T17:39:18.089 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/.o.?versionId=null 2024-03-04T17:39:18.089 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fqsbdSCiaCb9d9DKagzcDiSi42o= 2024-03-04T17:39:18.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fqsbdSCiaCb9d9DKagzcDiSi42o=', 'Content-Length': '0'} 2024-03-04T17:39:18.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1a4bf682795560f-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.093 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.093 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.093 DEBUG:boto:path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:18.093 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:18.093 DEBUG:boto:Method: GET 2024-03-04T17:39:18.093 DEBUG:boto:Path: /ppfahn-12/?prefix=.o. 2024-03-04T17:39:18.093 DEBUG:boto:Data: 2024-03-04T17:39:18.093 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.093 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.094 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.094 DEBUG:boto:Params: {} 2024-03-04T17:39:18.094 DEBUG:boto:Token: None 2024-03-04T17:39:18.094 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.094 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5c5cb05621399cb-0065e60746-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.099 DEBUG:boto:b'ppfahn-12.o.1000false.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.099 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.099 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.099 DEBUG:boto:path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:18.099 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:18.099 DEBUG:boto:Method: GET 2024-03-04T17:39:18.100 DEBUG:boto:Path: /ppfahn-12/?prefix=.o. 2024-03-04T17:39:18.100 DEBUG:boto:Data: 2024-03-04T17:39:18.100 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.100 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.100 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.100 DEBUG:boto:Params: {} 2024-03-04T17:39:18.100 DEBUG:boto:Token: None 2024-03-04T17:39:18.100 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.100 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018143724ce103a0b-0065e60746-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.148 DEBUG:boto:b'ppfahn-12.o.1000false.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.148 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:18.148 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:18.148 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.149 DEBUG:boto:Path: /ppfahn-12/.o. 2024-03-04T17:39:18.149 DEBUG:boto:Data: 2024-03-04T17:39:18.149 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.149 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.149 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.149 DEBUG:boto:Params: {} 2024-03-04T17:39:18.149 DEBUG:boto:Token: None 2024-03-04T17:39:18.149 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/.o. 2024-03-04T17:39:18.149 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m+PSEhYMXSLde3jPTkf343yFlNc= 2024-03-04T17:39:18.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m+PSEhYMXSLde3jPTkf343yFlNc=', 'Content-Length': '0'} 2024-03-04T17:39:18.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087fbc2c5a373b49d-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.152 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:18.152 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:18.152 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.152 DEBUG:boto:Path: /ppfahn-12/.o. 2024-03-04T17:39:18.153 DEBUG:boto:Data: 2024-03-04T17:39:18.153 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.153 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.153 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.153 DEBUG:boto:Params: {} 2024-03-04T17:39:18.153 DEBUG:boto:Token: None 2024-03-04T17:39:18.153 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/.o. 2024-03-04T17:39:18.153 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m+PSEhYMXSLde3jPTkf343yFlNc= 2024-03-04T17:39:18.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m+PSEhYMXSLde3jPTkf343yFlNc=', 'Content-Length': '0'} 2024-03-04T17:39:18.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000250ecdaac9865c17-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T17:39:18.156 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:18.156 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:18.157 DEBUG:boto:Method: GET 2024-03-04T17:39:18.157 DEBUG:boto:Path: /ppfahn-12/.o. 2024-03-04T17:39:18.157 DEBUG:boto:Data: 2024-03-04T17:39:18.157 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.157 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.157 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.157 DEBUG:boto:Params: {} 2024-03-04T17:39:18.157 DEBUG:boto:Token: None 2024-03-04T17:39:18.157 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/.o. 2024-03-04T17:39:18.157 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:10UPndEa17Xo8wgL8hnQT+ld+4U= 2024-03-04T17:39:18.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:10UPndEa17Xo8wgL8hnQT+ld+4U=', 'Content-Length': '0'} 2024-03-04T17:39:18.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b8b44815a50c1e5-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.161 DEBUG:boto:path=/ppfahn-12/.o. 2024-03-04T17:39:18.161 DEBUG:boto:auth_path=/ppfahn-12/.o. 2024-03-04T17:39:18.161 DEBUG:boto:Method: GET 2024-03-04T17:39:18.161 DEBUG:boto:Path: /ppfahn-12/.o. 2024-03-04T17:39:18.161 DEBUG:boto:Data: 2024-03-04T17:39:18.161 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.161 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.161 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.162 DEBUG:boto:Params: {} 2024-03-04T17:39:18.162 DEBUG:boto:Token: None 2024-03-04T17:39:18.162 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/.o. 2024-03-04T17:39:18.162 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:10UPndEa17Xo8wgL8hnQT+ld+4U= 2024-03-04T17:39:18.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:10UPndEa17Xo8wgL8hnQT+ld+4U=', 'Content-Length': '0'} 2024-03-04T17:39:18.165 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009756e7d8b7c57681-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.165 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.165 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.165 DEBUG:boto:path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:18.165 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:18.165 DEBUG:boto:Method: GET 2024-03-04T17:39:18.166 DEBUG:boto:Path: /ppfahn-12/?prefix=.o. 2024-03-04T17:39:18.166 DEBUG:boto:Data: 2024-03-04T17:39:18.166 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.166 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.166 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.166 DEBUG:boto:Params: {} 2024-03-04T17:39:18.166 DEBUG:boto:Token: None 2024-03-04T17:39:18.166 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.166 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049a587e57e650afa-0065e60746-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.212 DEBUG:boto:b'ppfahn-12.o.1000false.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.212 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.213 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.213 DEBUG:boto:path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:18.213 DEBUG:boto:auth_path=/ppfahn-12/?prefix=.o. 2024-03-04T17:39:18.213 DEBUG:boto:Method: GET 2024-03-04T17:39:18.213 DEBUG:boto:Path: /ppfahn-12/?prefix=.o. 2024-03-04T17:39:18.213 DEBUG:boto:Data: 2024-03-04T17:39:18.213 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.213 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.213 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.213 DEBUG:boto:Params: {} 2024-03-04T17:39:18.214 DEBUG:boto:Token: None 2024-03-04T17:39:18.214 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.214 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065a8a246adcaac65-0065e60746-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.219 DEBUG:boto:b'ppfahn-12.o.1000false.o.2024-03-04T17:36:30.940Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:18.219 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:18.220 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:18.220 DEBUG:boto:path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.220 DEBUG:boto:auth_path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.220 DEBUG:boto:Method: GET 2024-03-04T17:39:18.220 DEBUG:boto:Path: /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.220 DEBUG:boto:Data: 2024-03-04T17:39:18.220 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.220 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.220 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.220 DEBUG:boto:Params: {} 2024-03-04T17:39:18.220 DEBUG:boto:Token: None 2024-03-04T17:39:18.221 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.221 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9plTfwTEKnZ/04p/emOdQvwXaPA= 2024-03-04T17:39:18.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9plTfwTEKnZ/04p/emOdQvwXaPA=', 'Content-Length': '0'} 2024-03-04T17:39:18.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e548c006814e92b3-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.224 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:18.224 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:18.224 DEBUG:boto:path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.224 DEBUG:boto:auth_path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.224 DEBUG:boto:Method: GET 2024-03-04T17:39:18.224 DEBUG:boto:Path: /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.224 DEBUG:boto:Data: 2024-03-04T17:39:18.224 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.224 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.225 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.225 DEBUG:boto:Params: {} 2024-03-04T17:39:18.225 DEBUG:boto:Token: None 2024-03-04T17:39:18.225 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.225 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9plTfwTEKnZ/04p/emOdQvwXaPA= 2024-03-04T17:39:18.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9plTfwTEKnZ/04p/emOdQvwXaPA=', 'Content-Length': '0'} 2024-03-04T17:39:18.228 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026773a282139f250-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.272 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:18.272 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:18.272 DEBUG:boto:path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.272 DEBUG:boto:auth_path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.273 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.273 DEBUG:boto:Path: /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.273 DEBUG:boto:Data: 2024-03-04T17:39:18.273 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.273 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.273 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.273 DEBUG:boto:Params: {} 2024-03-04T17:39:18.273 DEBUG:boto:Token: None 2024-03-04T17:39:18.273 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.274 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/guA6mfIJJDQS+1XJjRlyYs7jOc= 2024-03-04T17:39:18.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/guA6mfIJJDQS+1XJjRlyYs7jOc=', 'Content-Length': '0'} 2024-03-04T17:39:18.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6276017cd91dd90-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.277 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:18.277 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:18.277 DEBUG:boto:path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.277 DEBUG:boto:auth_path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.277 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.277 DEBUG:boto:Path: /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.277 DEBUG:boto:Data: 2024-03-04T17:39:18.278 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.278 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.278 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.278 DEBUG:boto:Params: {} 2024-03-04T17:39:18.278 DEBUG:boto:Token: None 2024-03-04T17:39:18.278 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.278 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/guA6mfIJJDQS+1XJjRlyYs7jOc= 2024-03-04T17:39:18.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/guA6mfIJJDQS+1XJjRlyYs7jOc=', 'Content-Length': '0'} 2024-03-04T17:39:18.281 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb5a605ec773fa5c-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:18.282 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:18.282 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:18.282 DEBUG:boto:path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.282 DEBUG:boto:auth_path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.282 DEBUG:boto:Method: GET 2024-03-04T17:39:18.282 DEBUG:boto:Path: /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.282 DEBUG:boto:Data: 2024-03-04T17:39:18.282 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.283 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.283 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.283 DEBUG:boto:Params: {} 2024-03-04T17:39:18.283 DEBUG:boto:Token: None 2024-03-04T17:39:18.283 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.283 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9plTfwTEKnZ/04p/emOdQvwXaPA= 2024-03-04T17:39:18.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9plTfwTEKnZ/04p/emOdQvwXaPA=', 'Content-Length': '0'} 2024-03-04T17:39:18.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bb2f3f95493f327-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.287 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:18.287 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:18.287 DEBUG:boto:path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.287 DEBUG:boto:auth_path=/ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.287 DEBUG:boto:Method: GET 2024-03-04T17:39:18.287 DEBUG:boto:Path: /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.287 DEBUG:boto:Data: 2024-03-04T17:39:18.287 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.288 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.288 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.288 DEBUG:boto:Params: {} 2024-03-04T17:39:18.288 DEBUG:boto:Token: None 2024-03-04T17:39:18.288 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/%3A?versionId=null 2024-03-04T17:39:18.288 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9plTfwTEKnZ/04p/emOdQvwXaPA= 2024-03-04T17:39:18.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9plTfwTEKnZ/04p/emOdQvwXaPA=', 'Content-Length': '0'} 2024-03-04T17:39:18.291 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fc6a922b7fae3e5-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.292 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.292 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.292 DEBUG:boto:path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:18.292 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:18.292 DEBUG:boto:Method: GET 2024-03-04T17:39:18.292 DEBUG:boto:Path: /ppfahn-12/?prefix=%3A 2024-03-04T17:39:18.292 DEBUG:boto:Data: 2024-03-04T17:39:18.293 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.293 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.293 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.293 DEBUG:boto:Params: {} 2024-03-04T17:39:18.293 DEBUG:boto:Token: None 2024-03-04T17:39:18.293 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.293 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9d2a958839008e0-0065e60746-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.340 DEBUG:boto:b'ppfahn-12:1000false:2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.341 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.341 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.341 DEBUG:boto:path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:18.341 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:18.341 DEBUG:boto:Method: GET 2024-03-04T17:39:18.341 DEBUG:boto:Path: /ppfahn-12/?prefix=%3A 2024-03-04T17:39:18.341 DEBUG:boto:Data: 2024-03-04T17:39:18.341 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.341 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.341 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.342 DEBUG:boto:Params: {} 2024-03-04T17:39:18.342 DEBUG:boto:Token: None 2024-03-04T17:39:18.342 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.342 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5ec41755bcc8233-0065e60746-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.347 DEBUG:boto:b'ppfahn-12:1000false:2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.348 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:18.348 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:18.348 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.348 DEBUG:boto:Path: /ppfahn-12/%3A 2024-03-04T17:39:18.348 DEBUG:boto:Data: 2024-03-04T17:39:18.348 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.348 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.349 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.349 DEBUG:boto:Params: {} 2024-03-04T17:39:18.349 DEBUG:boto:Token: None 2024-03-04T17:39:18.349 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/%3A 2024-03-04T17:39:18.349 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7c2aI6jO9XmAis6qxPXOH1c07sY= 2024-03-04T17:39:18.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7c2aI6jO9XmAis6qxPXOH1c07sY=', 'Content-Length': '0'} 2024-03-04T17:39:18.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006578b282183dcfe7-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.352 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:18.352 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:18.352 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.353 DEBUG:boto:Path: /ppfahn-12/%3A 2024-03-04T17:39:18.353 DEBUG:boto:Data: 2024-03-04T17:39:18.353 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.353 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.353 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.353 DEBUG:boto:Params: {} 2024-03-04T17:39:18.353 DEBUG:boto:Token: None 2024-03-04T17:39:18.353 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/%3A 2024-03-04T17:39:18.353 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7c2aI6jO9XmAis6qxPXOH1c07sY= 2024-03-04T17:39:18.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7c2aI6jO9XmAis6qxPXOH1c07sY=', 'Content-Length': '0'} 2024-03-04T17:39:18.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f80807a64e403a2e-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T17:39:18.357 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:18.357 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:18.357 DEBUG:boto:Method: GET 2024-03-04T17:39:18.357 DEBUG:boto:Path: /ppfahn-12/%3A 2024-03-04T17:39:18.357 DEBUG:boto:Data: 2024-03-04T17:39:18.357 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.357 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.358 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.358 DEBUG:boto:Params: {} 2024-03-04T17:39:18.358 DEBUG:boto:Token: None 2024-03-04T17:39:18.358 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/%3A 2024-03-04T17:39:18.358 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:j9NaGDf0ugumaxgtO/+rrnSWIOo= 2024-03-04T17:39:18.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:j9NaGDf0ugumaxgtO/+rrnSWIOo=', 'Content-Length': '0'} 2024-03-04T17:39:18.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070b7224f4b805557-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.361 DEBUG:boto:path=/ppfahn-12/%3A 2024-03-04T17:39:18.362 DEBUG:boto:auth_path=/ppfahn-12/%3A 2024-03-04T17:39:18.362 DEBUG:boto:Method: GET 2024-03-04T17:39:18.362 DEBUG:boto:Path: /ppfahn-12/%3A 2024-03-04T17:39:18.362 DEBUG:boto:Data: 2024-03-04T17:39:18.362 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.362 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.362 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.362 DEBUG:boto:Params: {} 2024-03-04T17:39:18.362 DEBUG:boto:Token: None 2024-03-04T17:39:18.362 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/%3A 2024-03-04T17:39:18.363 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:j9NaGDf0ugumaxgtO/+rrnSWIOo= 2024-03-04T17:39:18.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:j9NaGDf0ugumaxgtO/+rrnSWIOo=', 'Content-Length': '0'} 2024-03-04T17:39:18.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002208dc7cdf6417b1-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.408 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.408 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.408 DEBUG:boto:path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:18.408 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:18.409 DEBUG:boto:Method: GET 2024-03-04T17:39:18.409 DEBUG:boto:Path: /ppfahn-12/?prefix=%3A 2024-03-04T17:39:18.409 DEBUG:boto:Data: 2024-03-04T17:39:18.409 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.409 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.409 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.409 DEBUG:boto:Params: {} 2024-03-04T17:39:18.409 DEBUG:boto:Token: None 2024-03-04T17:39:18.410 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.410 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079abd1e91816ad65-0065e60746-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.415 DEBUG:boto:b'ppfahn-12:1000false:2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.415 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.416 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.416 DEBUG:boto:path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:18.416 DEBUG:boto:auth_path=/ppfahn-12/?prefix=%3A 2024-03-04T17:39:18.416 DEBUG:boto:Method: GET 2024-03-04T17:39:18.416 DEBUG:boto:Path: /ppfahn-12/?prefix=%3A 2024-03-04T17:39:18.416 DEBUG:boto:Data: 2024-03-04T17:39:18.416 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.416 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.416 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.416 DEBUG:boto:Params: {} 2024-03-04T17:39:18.417 DEBUG:boto:Token: None 2024-03-04T17:39:18.417 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.417 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046e7055c3686efea-0065e60746-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.421 DEBUG:boto:b'ppfahn-12:1000false:2024-03-04T17:36:30.728Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:18.422 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:18.422 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:18.422 DEBUG:boto:path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.422 DEBUG:boto:auth_path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.422 DEBUG:boto:Method: GET 2024-03-04T17:39:18.422 DEBUG:boto:Path: /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.422 DEBUG:boto:Data: 2024-03-04T17:39:18.422 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.422 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.422 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.423 DEBUG:boto:Params: {} 2024-03-04T17:39:18.423 DEBUG:boto:Token: None 2024-03-04T17:39:18.423 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.423 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MOMViCFaDlb2JJ7rEdrn/6eLJ7s= 2024-03-04T17:39:18.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MOMViCFaDlb2JJ7rEdrn/6eLJ7s=', 'Content-Length': '0'} 2024-03-04T17:39:18.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065db1e6e91eb0829-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.426 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:18.426 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:18.426 DEBUG:boto:path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.426 DEBUG:boto:auth_path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.427 DEBUG:boto:Method: GET 2024-03-04T17:39:18.427 DEBUG:boto:Path: /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.427 DEBUG:boto:Data: 2024-03-04T17:39:18.427 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.427 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.427 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.427 DEBUG:boto:Params: {} 2024-03-04T17:39:18.427 DEBUG:boto:Token: None 2024-03-04T17:39:18.427 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.428 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MOMViCFaDlb2JJ7rEdrn/6eLJ7s= 2024-03-04T17:39:18.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MOMViCFaDlb2JJ7rEdrn/6eLJ7s=', 'Content-Length': '0'} 2024-03-04T17:39:18.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b52a30f3754187e7-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.431 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:18.431 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:18.431 DEBUG:boto:path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.431 DEBUG:boto:auth_path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.431 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.431 DEBUG:boto:Path: /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.432 DEBUG:boto:Data: 2024-03-04T17:39:18.432 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.432 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.432 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.432 DEBUG:boto:Params: {} 2024-03-04T17:39:18.432 DEBUG:boto:Token: None 2024-03-04T17:39:18.432 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.432 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v9nFFg4KmifFoq5SuH3gxX4hO3A= 2024-03-04T17:39:18.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v9nFFg4KmifFoq5SuH3gxX4hO3A=', 'Content-Length': '0'} 2024-03-04T17:39:18.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a1975fdf2be6806-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.435 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:18.435 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:18.436 DEBUG:boto:path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.436 DEBUG:boto:auth_path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.436 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.436 DEBUG:boto:Path: /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.436 DEBUG:boto:Data: 2024-03-04T17:39:18.436 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.436 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.436 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.436 DEBUG:boto:Params: {} 2024-03-04T17:39:18.436 DEBUG:boto:Token: None 2024-03-04T17:39:18.437 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.437 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v9nFFg4KmifFoq5SuH3gxX4hO3A= 2024-03-04T17:39:18.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v9nFFg4KmifFoq5SuH3gxX4hO3A=', 'Content-Length': '0'} 2024-03-04T17:39:18.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e101c122f05a3467-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:18.440 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:18.440 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:18.440 DEBUG:boto:path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.440 DEBUG:boto:auth_path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.440 DEBUG:boto:Method: GET 2024-03-04T17:39:18.441 DEBUG:boto:Path: /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.441 DEBUG:boto:Data: 2024-03-04T17:39:18.441 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.441 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.441 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.441 DEBUG:boto:Params: {} 2024-03-04T17:39:18.441 DEBUG:boto:Token: None 2024-03-04T17:39:18.441 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.441 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MOMViCFaDlb2JJ7rEdrn/6eLJ7s= 2024-03-04T17:39:18.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MOMViCFaDlb2JJ7rEdrn/6eLJ7s=', 'Content-Length': '0'} 2024-03-04T17:39:18.444 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c850d60d44edff1-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.488 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:18.488 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:18.488 DEBUG:boto:path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.488 DEBUG:boto:auth_path=/ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.488 DEBUG:boto:Method: GET 2024-03-04T17:39:18.489 DEBUG:boto:Path: /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.489 DEBUG:boto:Data: 2024-03-04T17:39:18.489 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.489 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.489 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.489 DEBUG:boto:Params: {} 2024-03-04T17:39:18.489 DEBUG:boto:Token: None 2024-03-04T17:39:18.489 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/_myobj?versionId=null 2024-03-04T17:39:18.489 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MOMViCFaDlb2JJ7rEdrn/6eLJ7s= 2024-03-04T17:39:18.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MOMViCFaDlb2JJ7rEdrn/6eLJ7s=', 'Content-Length': '0'} 2024-03-04T17:39:18.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccaaad88002623bc-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.493 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.493 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.493 DEBUG:boto:path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:18.493 DEBUG:boto:auth_path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:18.493 DEBUG:boto:Method: GET 2024-03-04T17:39:18.493 DEBUG:boto:Path: /ppfahn-12/?prefix=_myobj 2024-03-04T17:39:18.493 DEBUG:boto:Data: 2024-03-04T17:39:18.493 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.494 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.494 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.494 DEBUG:boto:Params: {} 2024-03-04T17:39:18.494 DEBUG:boto:Token: None 2024-03-04T17:39:18.494 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.494 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004feecd1d4dfebb9d-0065e60746-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.499 DEBUG:boto:b'ppfahn-12_myobj1000false_myobj2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.500 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.500 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.500 DEBUG:boto:path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:18.500 DEBUG:boto:auth_path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:18.500 DEBUG:boto:Method: GET 2024-03-04T17:39:18.500 DEBUG:boto:Path: /ppfahn-12/?prefix=_myobj 2024-03-04T17:39:18.500 DEBUG:boto:Data: 2024-03-04T17:39:18.500 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.500 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.501 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.501 DEBUG:boto:Params: {} 2024-03-04T17:39:18.501 DEBUG:boto:Token: None 2024-03-04T17:39:18.501 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.501 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b4c390e653d5487-0065e60746-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.548 DEBUG:boto:b'ppfahn-12_myobj1000false_myobj2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.548 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:18.548 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:18.549 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.549 DEBUG:boto:Path: /ppfahn-12/_myobj 2024-03-04T17:39:18.549 DEBUG:boto:Data: 2024-03-04T17:39:18.549 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.549 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.549 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.549 DEBUG:boto:Params: {} 2024-03-04T17:39:18.549 DEBUG:boto:Token: None 2024-03-04T17:39:18.549 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/_myobj 2024-03-04T17:39:18.549 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TVIZhE1gAZevJmLh15fHrdlodQw= 2024-03-04T17:39:18.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TVIZhE1gAZevJmLh15fHrdlodQw=', 'Content-Length': '0'} 2024-03-04T17:39:18.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000918be3f3a817dfb8-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.553 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:18.553 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:18.553 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.553 DEBUG:boto:Path: /ppfahn-12/_myobj 2024-03-04T17:39:18.553 DEBUG:boto:Data: 2024-03-04T17:39:18.553 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.553 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.553 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.553 DEBUG:boto:Params: {} 2024-03-04T17:39:18.554 DEBUG:boto:Token: None 2024-03-04T17:39:18.554 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/_myobj 2024-03-04T17:39:18.554 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TVIZhE1gAZevJmLh15fHrdlodQw= 2024-03-04T17:39:18.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TVIZhE1gAZevJmLh15fHrdlodQw=', 'Content-Length': '0'} 2024-03-04T17:39:18.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c42d74f3f800043f-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T17:39:18.557 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:18.557 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:18.557 DEBUG:boto:Method: GET 2024-03-04T17:39:18.558 DEBUG:boto:Path: /ppfahn-12/_myobj 2024-03-04T17:39:18.558 DEBUG:boto:Data: 2024-03-04T17:39:18.558 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.558 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.558 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.558 DEBUG:boto:Params: {} 2024-03-04T17:39:18.558 DEBUG:boto:Token: None 2024-03-04T17:39:18.558 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/_myobj 2024-03-04T17:39:18.558 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:el6EYvSE4ndoASoeWf/SLY9DSYk= 2024-03-04T17:39:18.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:el6EYvSE4ndoASoeWf/SLY9DSYk=', 'Content-Length': '0'} 2024-03-04T17:39:18.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9ee8d04912180d4-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.562 DEBUG:boto:path=/ppfahn-12/_myobj 2024-03-04T17:39:18.562 DEBUG:boto:auth_path=/ppfahn-12/_myobj 2024-03-04T17:39:18.562 DEBUG:boto:Method: GET 2024-03-04T17:39:18.562 DEBUG:boto:Path: /ppfahn-12/_myobj 2024-03-04T17:39:18.562 DEBUG:boto:Data: 2024-03-04T17:39:18.562 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.562 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.562 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.562 DEBUG:boto:Params: {} 2024-03-04T17:39:18.562 DEBUG:boto:Token: None 2024-03-04T17:39:18.563 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/_myobj 2024-03-04T17:39:18.563 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:el6EYvSE4ndoASoeWf/SLY9DSYk= 2024-03-04T17:39:18.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:el6EYvSE4ndoASoeWf/SLY9DSYk=', 'Content-Length': '0'} 2024-03-04T17:39:18.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b181f4704fa8fd49-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.566 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.566 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.566 DEBUG:boto:path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:18.566 DEBUG:boto:auth_path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:18.566 DEBUG:boto:Method: GET 2024-03-04T17:39:18.566 DEBUG:boto:Path: /ppfahn-12/?prefix=_myobj 2024-03-04T17:39:18.566 DEBUG:boto:Data: 2024-03-04T17:39:18.567 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.567 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.567 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.567 DEBUG:boto:Params: {} 2024-03-04T17:39:18.567 DEBUG:boto:Token: None 2024-03-04T17:39:18.567 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.567 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007db8cd5de884405f-0065e60746-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.612 DEBUG:boto:b'ppfahn-12_myobj1000false_myobj2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.612 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.612 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.613 DEBUG:boto:path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:18.613 DEBUG:boto:auth_path=/ppfahn-12/?prefix=_myobj 2024-03-04T17:39:18.613 DEBUG:boto:Method: GET 2024-03-04T17:39:18.613 DEBUG:boto:Path: /ppfahn-12/?prefix=_myobj 2024-03-04T17:39:18.613 DEBUG:boto:Data: 2024-03-04T17:39:18.613 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.613 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.613 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.613 DEBUG:boto:Params: {} 2024-03-04T17:39:18.613 DEBUG:boto:Token: None 2024-03-04T17:39:18.613 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.614 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070dccc129c4230db-0065e60746-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.618 DEBUG:boto:b'ppfahn-12_myobj1000false_myobj2024-03-04T17:36:30.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:18.619 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:18.619 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:18.619 DEBUG:boto:path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.619 DEBUG:boto:auth_path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.619 DEBUG:boto:Method: GET 2024-03-04T17:39:18.619 DEBUG:boto:Path: /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.619 DEBUG:boto:Data: 2024-03-04T17:39:18.620 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.620 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.620 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.620 DEBUG:boto:Params: {} 2024-03-04T17:39:18.620 DEBUG:boto:Token: None 2024-03-04T17:39:18.620 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.620 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s1gtLiLnz/tejJGsy/POpgP+IbM= 2024-03-04T17:39:18.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s1gtLiLnz/tejJGsy/POpgP+IbM=', 'Content-Length': '0'} 2024-03-04T17:39:18.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddb0e68d9b20c2ef-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.624 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:18.624 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:18.624 DEBUG:boto:path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.624 DEBUG:boto:auth_path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.624 DEBUG:boto:Method: GET 2024-03-04T17:39:18.624 DEBUG:boto:Path: /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.624 DEBUG:boto:Data: 2024-03-04T17:39:18.624 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.624 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.624 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.624 DEBUG:boto:Params: {} 2024-03-04T17:39:18.625 DEBUG:boto:Token: None 2024-03-04T17:39:18.625 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.625 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s1gtLiLnz/tejJGsy/POpgP+IbM= 2024-03-04T17:39:18.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s1gtLiLnz/tejJGsy/POpgP+IbM=', 'Content-Length': '0'} 2024-03-04T17:39:18.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081748f1410eae81c-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.668 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:18.668 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:18.668 DEBUG:boto:path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.668 DEBUG:boto:auth_path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.668 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.669 DEBUG:boto:Path: /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.669 DEBUG:boto:Data: 2024-03-04T17:39:18.669 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.669 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.669 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.669 DEBUG:boto:Params: {} 2024-03-04T17:39:18.669 DEBUG:boto:Token: None 2024-03-04T17:39:18.669 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.669 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6CWPAdU2NI+vyk2OEsPXPItzuSs= 2024-03-04T17:39:18.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6CWPAdU2NI+vyk2OEsPXPItzuSs=', 'Content-Length': '0'} 2024-03-04T17:39:18.672 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b402e59e0458ff81-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.672 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:18.672 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:18.672 DEBUG:boto:path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.673 DEBUG:boto:auth_path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.673 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.673 DEBUG:boto:Path: /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.673 DEBUG:boto:Data: 2024-03-04T17:39:18.673 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.673 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.673 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.673 DEBUG:boto:Params: {} 2024-03-04T17:39:18.673 DEBUG:boto:Token: None 2024-03-04T17:39:18.673 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.673 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6CWPAdU2NI+vyk2OEsPXPItzuSs= 2024-03-04T17:39:18.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6CWPAdU2NI+vyk2OEsPXPItzuSs=', 'Content-Length': '0'} 2024-03-04T17:39:18.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e978636617adaf19-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:18.677 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:18.677 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:18.677 DEBUG:boto:path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.677 DEBUG:boto:auth_path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.677 DEBUG:boto:Method: GET 2024-03-04T17:39:18.677 DEBUG:boto:Path: /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.677 DEBUG:boto:Data: 2024-03-04T17:39:18.677 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.677 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.677 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.678 DEBUG:boto:Params: {} 2024-03-04T17:39:18.678 DEBUG:boto:Token: None 2024-03-04T17:39:18.678 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.678 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s1gtLiLnz/tejJGsy/POpgP+IbM= 2024-03-04T17:39:18.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s1gtLiLnz/tejJGsy/POpgP+IbM=', 'Content-Length': '0'} 2024-03-04T17:39:18.681 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e33d80a828b9c3a-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.681 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:18.681 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:18.681 DEBUG:boto:path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.682 DEBUG:boto:auth_path=/ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.682 DEBUG:boto:Method: GET 2024-03-04T17:39:18.682 DEBUG:boto:Path: /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.682 DEBUG:boto:Data: 2024-03-04T17:39:18.682 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.682 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.682 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.682 DEBUG:boto:Params: {} 2024-03-04T17:39:18.682 DEBUG:boto:Token: None 2024-03-04T17:39:18.682 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/myobj?versionId=null 2024-03-04T17:39:18.683 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s1gtLiLnz/tejJGsy/POpgP+IbM= 2024-03-04T17:39:18.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s1gtLiLnz/tejJGsy/POpgP+IbM=', 'Content-Length': '0'} 2024-03-04T17:39:18.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea2d0dac3f08ba0e-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.686 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.686 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.686 DEBUG:boto:path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:18.686 DEBUG:boto:auth_path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:18.686 DEBUG:boto:Method: GET 2024-03-04T17:39:18.687 DEBUG:boto:Path: /ppfahn-12/?prefix=myobj 2024-03-04T17:39:18.687 DEBUG:boto:Data: 2024-03-04T17:39:18.687 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.687 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.687 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.687 DEBUG:boto:Params: {} 2024-03-04T17:39:18.687 DEBUG:boto:Token: None 2024-03-04T17:39:18.687 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.687 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000111a38c65ca25012-0065e60746-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.732 DEBUG:boto:b'ppfahn-12myobj1000falsemyobj2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.733 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.733 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.733 DEBUG:boto:path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:18.733 DEBUG:boto:auth_path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:18.733 DEBUG:boto:Method: GET 2024-03-04T17:39:18.733 DEBUG:boto:Path: /ppfahn-12/?prefix=myobj 2024-03-04T17:39:18.733 DEBUG:boto:Data: 2024-03-04T17:39:18.733 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.733 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.733 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.734 DEBUG:boto:Params: {} 2024-03-04T17:39:18.734 DEBUG:boto:Token: None 2024-03-04T17:39:18.734 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.734 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cffddd9f07e6710-0065e60746-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.738 DEBUG:boto:b'ppfahn-12myobj1000falsemyobj2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.739 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:18.739 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:18.739 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.739 DEBUG:boto:Path: /ppfahn-12/myobj 2024-03-04T17:39:18.739 DEBUG:boto:Data: 2024-03-04T17:39:18.739 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.739 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.740 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.740 DEBUG:boto:Params: {} 2024-03-04T17:39:18.740 DEBUG:boto:Token: None 2024-03-04T17:39:18.740 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/myobj 2024-03-04T17:39:18.740 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S9GFdlAEatmIy/vtm2FWYfdb4rQ= 2024-03-04T17:39:18.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S9GFdlAEatmIy/vtm2FWYfdb4rQ=', 'Content-Length': '0'} 2024-03-04T17:39:18.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd847720914428b8-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.743 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:18.743 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:18.743 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.743 DEBUG:boto:Path: /ppfahn-12/myobj 2024-03-04T17:39:18.743 DEBUG:boto:Data: 2024-03-04T17:39:18.744 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.744 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.744 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.744 DEBUG:boto:Params: {} 2024-03-04T17:39:18.744 DEBUG:boto:Token: None 2024-03-04T17:39:18.744 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/myobj 2024-03-04T17:39:18.744 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S9GFdlAEatmIy/vtm2FWYfdb4rQ= 2024-03-04T17:39:18.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S9GFdlAEatmIy/vtm2FWYfdb4rQ=', 'Content-Length': '0'} 2024-03-04T17:39:18.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000620a5712a18d4fa7-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:18.747 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:18.747 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:18.747 DEBUG:boto:Method: GET 2024-03-04T17:39:18.747 DEBUG:boto:Path: /ppfahn-12/myobj 2024-03-04T17:39:18.747 DEBUG:boto:Data: 2024-03-04T17:39:18.747 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.748 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.748 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.748 DEBUG:boto:Params: {} 2024-03-04T17:39:18.748 DEBUG:boto:Token: None 2024-03-04T17:39:18.748 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/myobj 2024-03-04T17:39:18.748 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IB0QgPFQe8GVAcySxChEHWHzM6o= 2024-03-04T17:39:18.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IB0QgPFQe8GVAcySxChEHWHzM6o=', 'Content-Length': '0'} 2024-03-04T17:39:18.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-12:d96636ac-054d-40e3-90f3-e3912da784b2.4349.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4c9536ae7e4f69f-0065e60746-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.751 DEBUG:boto:path=/ppfahn-12/myobj 2024-03-04T17:39:18.751 DEBUG:boto:auth_path=/ppfahn-12/myobj 2024-03-04T17:39:18.752 DEBUG:boto:Method: GET 2024-03-04T17:39:18.752 DEBUG:boto:Path: /ppfahn-12/myobj 2024-03-04T17:39:18.752 DEBUG:boto:Data: 2024-03-04T17:39:18.752 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.752 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.752 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.752 DEBUG:boto:Params: {} 2024-03-04T17:39:18.752 DEBUG:boto:Token: None 2024-03-04T17:39:18.752 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/myobj 2024-03-04T17:39:18.752 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IB0QgPFQe8GVAcySxChEHWHzM6o= 2024-03-04T17:39:18.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IB0QgPFQe8GVAcySxChEHWHzM6o=', 'Content-Length': '0'} 2024-03-04T17:39:18.755 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:36:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da454de40ee7a048-0065e60746-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.796 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.796 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.796 DEBUG:boto:path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:18.796 DEBUG:boto:auth_path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:18.796 DEBUG:boto:Method: GET 2024-03-04T17:39:18.797 DEBUG:boto:Path: /ppfahn-12/?prefix=myobj 2024-03-04T17:39:18.797 DEBUG:boto:Data: 2024-03-04T17:39:18.797 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.797 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.797 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.797 DEBUG:boto:Params: {} 2024-03-04T17:39:18.797 DEBUG:boto:Token: None 2024-03-04T17:39:18.797 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.797 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0f604058b2fc363-0065e60746-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.802 DEBUG:boto:b'ppfahn-12myobj1000falsemyobj2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.802 DEBUG:boto:path=/ppfahn-12/ 2024-03-04T17:39:18.802 DEBUG:boto:auth_path=/ppfahn-12/ 2024-03-04T17:39:18.802 DEBUG:boto:path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:18.802 DEBUG:boto:auth_path=/ppfahn-12/?prefix=myobj 2024-03-04T17:39:18.802 DEBUG:boto:Method: GET 2024-03-04T17:39:18.803 DEBUG:boto:Path: /ppfahn-12/?prefix=myobj 2024-03-04T17:39:18.803 DEBUG:boto:Data: 2024-03-04T17:39:18.803 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.803 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.803 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.803 DEBUG:boto:Params: {} 2024-03-04T17:39:18.803 DEBUG:boto:Token: None 2024-03-04T17:39:18.803 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-12/ 2024-03-04T17:39:18.803 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU= 2024-03-04T17:39:18.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6NcmY4neqvzJu2XotOF5hkITsaU=', 'Content-Length': '0'} 2024-03-04T17:39:18.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090899dd2b38cb625-0065e60746-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.808 DEBUG:boto:b'ppfahn-12myobj1000falsemyobj2024-03-04T17:36:30.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:18.808 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-12 zones={test-zone2, test-zone3} 2024-03-04T17:39:18.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2024-03-04T17:39:18.810 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-13 2024-03-04T17:39:18.810 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:18.810 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:18.810 DEBUG:boto:Method: PUT 2024-03-04T17:39:18.811 DEBUG:boto:Path: /ppfahn-13/ 2024-03-04T17:39:18.811 DEBUG:boto:Data: 2024-03-04T17:39:18.811 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.811 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:18.811 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.811 DEBUG:boto:Params: {} 2024-03-04T17:39:18.811 DEBUG:boto:Token: None 2024-03-04T17:39:18.811 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-13/ 2024-03-04T17:39:18.811 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BhBZV0O4NPYkrvRuf77ySpAsQ7g= 2024-03-04T17:39:18.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BhBZV0O4NPYkrvRuf77ySpAsQ7g=', 'Content-Length': '0'} 2024-03-04T17:39:18.845 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000797c895758493ea6-0065e60746-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.845 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-14 2024-03-04T17:39:18.845 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:18.845 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:18.845 DEBUG:boto:Method: PUT 2024-03-04T17:39:18.845 DEBUG:boto:Path: /ppfahn-14/ 2024-03-04T17:39:18.845 DEBUG:boto:Data: 2024-03-04T17:39:18.845 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.845 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.845 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.845 DEBUG:boto:Params: {} 2024-03-04T17:39:18.846 DEBUG:boto:Token: None 2024-03-04T17:39:18.846 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-14/ 2024-03-04T17:39:18.846 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iOl8QgESV9Ck00lzaeHMNDpHjwo= 2024-03-04T17:39:18.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iOl8QgESV9Ck00lzaeHMNDpHjwo=', 'Content-Length': '0'} 2024-03-04T17:39:18.906 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000082626eaa8f178e11-0065e60746-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.906 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-15 2024-03-04T17:39:18.906 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:18.906 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:18.906 DEBUG:boto:Method: PUT 2024-03-04T17:39:18.906 DEBUG:boto:Path: /ppfahn-15/ 2024-03-04T17:39:18.906 DEBUG:boto:Data: 2024-03-04T17:39:18.906 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.906 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:18.907 DEBUG:boto:Port: 8001 2024-03-04T17:39:18.907 DEBUG:boto:Params: {} 2024-03-04T17:39:18.907 DEBUG:boto:Token: None 2024-03-04T17:39:18.907 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-15/ 2024-03-04T17:39:18.907 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4d0tbRnFQNa8dOcPPz12j0LhzWY= 2024-03-04T17:39:18.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4d0tbRnFQNa8dOcPPz12j0LhzWY=', 'Content-Length': '0'} 2024-03-04T17:39:18.972 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000c49e9985fe6ddd6-0065e60746-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.972 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:18.972 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:18.972 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.972 DEBUG:boto:Path: /ppfahn-13/ 2024-03-04T17:39:18.972 DEBUG:boto:Data: 2024-03-04T17:39:18.973 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.973 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:18.973 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.973 DEBUG:boto:Params: {} 2024-03-04T17:39:18.973 DEBUG:boto:Token: None 2024-03-04T17:39:18.973 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-13/ 2024-03-04T17:39:18.973 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:767z/PviY0lp5Hkmep3MWmQDtEI= 2024-03-04T17:39:18.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:767z/PviY0lp5Hkmep3MWmQDtEI=', 'Content-Length': '0'} 2024-03-04T17:39:18.977 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e11e1adb7a754bb-0065e60746-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.977 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:18.977 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:18.977 DEBUG:boto:Method: PUT 2024-03-04T17:39:18.977 DEBUG:boto:Path: /ppfahn-13/myobj 2024-03-04T17:39:18.977 DEBUG:boto:Data: 2024-03-04T17:39:18.978 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:39:18.978 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:18.978 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.978 DEBUG:boto:Params: {} 2024-03-04T17:39:18.978 DEBUG:boto:Token: None 2024-03-04T17:39:18.978 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-13/myobj 2024-03-04T17:39:18.978 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/w5ZPf0ZOXGPiwzgjmh8JDOoXHQ= 2024-03-04T17:39:18.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/w5ZPf0ZOXGPiwzgjmh8JDOoXHQ='} 2024-03-04T17:39:18.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c7f2cd3f0b42238-0065e60746-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.988 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:18.988 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:18.988 DEBUG:boto:Method: HEAD 2024-03-04T17:39:18.988 DEBUG:boto:Path: /ppfahn-14/ 2024-03-04T17:39:18.988 DEBUG:boto:Data: 2024-03-04T17:39:18.988 DEBUG:boto:Headers: {} 2024-03-04T17:39:18.988 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.988 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.988 DEBUG:boto:Params: {} 2024-03-04T17:39:18.988 DEBUG:boto:Token: None 2024-03-04T17:39:18.989 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-14/ 2024-03-04T17:39:18.989 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bbf1FHgNdMJG462Fk0/dp0TFTjM= 2024-03-04T17:39:18.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bbf1FHgNdMJG462Fk0/dp0TFTjM=', 'Content-Length': '0'} 2024-03-04T17:39:18.992 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067bd022cb4c11903-0065e60746-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:18.992 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:18.992 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:18.992 DEBUG:boto:Method: PUT 2024-03-04T17:39:18.992 DEBUG:boto:Path: /ppfahn-14/myobj 2024-03-04T17:39:18.992 DEBUG:boto:Data: 2024-03-04T17:39:18.993 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:39:18.993 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:18.993 DEBUG:boto:Port: 8000 2024-03-04T17:39:18.993 DEBUG:boto:Params: {} 2024-03-04T17:39:18.993 DEBUG:boto:Token: None 2024-03-04T17:39:18.993 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:39:18 GMT /ppfahn-14/myobj 2024-03-04T17:39:18.993 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4++/ZoVVVTCycutLC+GZ0FexdYM= 2024-03-04T17:39:18.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:39:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4++/ZoVVVTCycutLC+GZ0FexdYM='} 2024-03-04T17:39:19.009 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b324797c72b8986e-0065e60746-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:19.009 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:19.009 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:19.010 DEBUG:boto:Method: HEAD 2024-03-04T17:39:19.010 DEBUG:boto:Path: /ppfahn-15/ 2024-03-04T17:39:19.010 DEBUG:boto:Data: 2024-03-04T17:39:19.010 DEBUG:boto:Headers: {} 2024-03-04T17:39:19.010 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:19.010 DEBUG:boto:Port: 8001 2024-03-04T17:39:19.010 DEBUG:boto:Params: {} 2024-03-04T17:39:19.010 DEBUG:boto:Token: None 2024-03-04T17:39:19.010 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:19 GMT /ppfahn-15/ 2024-03-04T17:39:19.010 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G30pGdgLCiK04lMd7KB13SBu4EQ= 2024-03-04T17:39:19.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G30pGdgLCiK04lMd7KB13SBu4EQ=', 'Content-Length': '0'} 2024-03-04T17:39:19.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ea0f07d95971406-0065e60747-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:19.016 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:19.016 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:19.016 DEBUG:boto:Method: PUT 2024-03-04T17:39:19.017 DEBUG:boto:Path: /ppfahn-15/myobj 2024-03-04T17:39:19.017 DEBUG:boto:Data: 2024-03-04T17:39:19.017 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:39:19.017 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:19.017 DEBUG:boto:Port: 8001 2024-03-04T17:39:19.017 DEBUG:boto:Params: {} 2024-03-04T17:39:19.017 DEBUG:boto:Token: None 2024-03-04T17:39:19.017 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:39:19 GMT /ppfahn-15/myobj 2024-03-04T17:39:19.017 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MHhl0RvUPENkCNxJk10IrzF6M4M= 2024-03-04T17:39:19.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:39:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MHhl0RvUPENkCNxJk10IrzF6M4M='} 2024-03-04T17:39:19.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000134ce715ee7b63ce-0065e60747-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:19.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:19.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:19.028 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:19.066 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:19.066 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:19.120 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:39:19.146 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709573958.942642_78.1","last_update":"2024-03-04T17:39:18.942642Z"}] 2024-03-04T17:39:19.147 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573958.942642_78.1'} 2024-03-04T17:39:19.147 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:39:19.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:19.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:19.147 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:19.239+0000 7fd1fabc5980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:19.239+0000 7fd1fabc5980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:19.269 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573958.942642_78.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:39:18.942642Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:19Z"} 2024-03-04T17:39:19.269 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573958.942642_78.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:39:18.942642Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:19Z"} 2024-03-04T17:39:19.269 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709573958.942642_78.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:39:18.942642Z', 'realm_epoch': 2}}] 2024-03-04T17:39:19.269 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573958.942642_78.1'} 2024-03-04T17:39:19.270 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573958.942642_78.1'} 2024-03-04T17:39:19.270 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:39:19.270 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:39:19.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:19.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:19.270 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:19.315 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:19.315 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:19.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:19.368+0000 7fe2bfdf7980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:19.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:19.372+0000 7fe2bfdf7980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:19.396 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573958.935581_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:39:18.935581Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:19Z"} 2024-03-04T17:39:19.397 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573958.935581_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:39:18.935581Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:19Z"} 2024-03-04T17:39:19.397 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709573958.935581_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:39:18.935581Z', 'realm_epoch': 2}}] 2024-03-04T17:39:19.397 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573958.942642_78.1'} 2024-03-04T17:39:19.397 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573958.935581_77.1'} 2024-03-04T17:39:19.397 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1709573958.942642_78.1 target=1_1709573958.935581_77.1 2024-03-04T17:39:24.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:24.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:24.399 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:24.442 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:24.442 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:24.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:24.492+0000 7f974ebbb980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:24.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:24.496+0000 7f974ebbb980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:24.518 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573958.942642_78.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:39:18.942642Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:24Z"} 2024-03-04T17:39:24.518 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709573958.942642_78.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:39:18.942642Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:24Z"} 2024-03-04T17:39:24.519 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709573958.942642_78.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:39:18.942642Z', 'realm_epoch': 2}}] 2024-03-04T17:39:24.519 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573958.942642_78.1'} 2024-03-04T17:39:24.519 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709573958.942642_78.1'} 2024-03-04T17:39:24.519 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:39:24.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:24.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:24.520 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:39:24.571 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:24.571 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:24.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:24.631+0000 7f2db4c54980 1 bucket sync caught up with source: 2024-03-04T17:39:24.634 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-03-04T17:39:24.634 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000001.13.6, , , , , , , , , ] 2024-03-04T17:39:24.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:24.631+0000 7f2db4c54980 0 bucket checkpoint complete 2024-03-04T17:39:24.657 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-13 zones={test-zone2, test-zone1} 2024-03-04T17:39:24.657 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:24.657 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:24.657 DEBUG:boto:Method: HEAD 2024-03-04T17:39:24.657 DEBUG:boto:Path: /ppfahn-13/ 2024-03-04T17:39:24.657 DEBUG:boto:Data: 2024-03-04T17:39:24.657 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.657 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:24.657 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.658 DEBUG:boto:Params: {} 2024-03-04T17:39:24.658 DEBUG:boto:Token: None 2024-03-04T17:39:24.658 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/ 2024-03-04T17:39:24.658 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XlgFLPjs24rVvfgwfnP2v82B32k= 2024-03-04T17:39:24.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XlgFLPjs24rVvfgwfnP2v82B32k=', 'Content-Length': '0'} 2024-03-04T17:39:24.660 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000473cfa6d4bc77a2c-0065e6074c-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.661 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:24.661 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:24.661 DEBUG:boto:Method: HEAD 2024-03-04T17:39:24.661 DEBUG:boto:Path: /ppfahn-13/ 2024-03-04T17:39:24.661 DEBUG:boto:Data: 2024-03-04T17:39:24.661 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.661 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:24.661 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.661 DEBUG:boto:Params: {} 2024-03-04T17:39:24.662 DEBUG:boto:Token: None 2024-03-04T17:39:24.662 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/ 2024-03-04T17:39:24.662 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XlgFLPjs24rVvfgwfnP2v82B32k= 2024-03-04T17:39:24.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XlgFLPjs24rVvfgwfnP2v82B32k=', 'Content-Length': '0'} 2024-03-04T17:39:24.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d82091ca24083237-0065e6074c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.666 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:39:24.666 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:24.666 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:24.666 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:24.666 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:24.666 DEBUG:boto:Method: GET 2024-03-04T17:39:24.666 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:24.666 DEBUG:boto:Data: 2024-03-04T17:39:24.667 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.667 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:24.667 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.667 DEBUG:boto:Params: {} 2024-03-04T17:39:24.667 DEBUG:boto:Token: None 2024-03-04T17:39:24.667 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/?versions 2024-03-04T17:39:24.667 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZxcFA9qsgtQIqsj842MrBjTwQZs= 2024-03-04T17:39:24.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZxcFA9qsgtQIqsj842MrBjTwQZs=', 'Content-Length': '0'} 2024-03-04T17:39:24.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000146bdfac9aa443f5-0065e6074c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.670 DEBUG:boto:b'ppfahn-13999falsemyobjnulltrue2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:24.670 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:24.670 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:39:24.670 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:24.671 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:24.671 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:24.671 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:24.671 DEBUG:boto:Method: GET 2024-03-04T17:39:24.671 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:24.671 DEBUG:boto:Data: 2024-03-04T17:39:24.671 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.671 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:24.671 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.671 DEBUG:boto:Params: {} 2024-03-04T17:39:24.672 DEBUG:boto:Token: None 2024-03-04T17:39:24.672 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/?versions 2024-03-04T17:39:24.672 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZxcFA9qsgtQIqsj842MrBjTwQZs= 2024-03-04T17:39:24.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZxcFA9qsgtQIqsj842MrBjTwQZs=', 'Content-Length': '0'} 2024-03-04T17:39:24.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef211a028fe493a4-0065e6074c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.716 DEBUG:boto:b'ppfahn-13999falsemyobjnulltrue2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:24.716 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:24.717 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:24.717 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:24.717 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:24.717 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:24.717 DEBUG:boto:Method: GET 2024-03-04T17:39:24.717 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:24.717 DEBUG:boto:Data: 2024-03-04T17:39:24.717 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.717 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:24.717 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.718 DEBUG:boto:Params: {} 2024-03-04T17:39:24.718 DEBUG:boto:Token: None 2024-03-04T17:39:24.718 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/?versions 2024-03-04T17:39:24.718 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZxcFA9qsgtQIqsj842MrBjTwQZs= 2024-03-04T17:39:24.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZxcFA9qsgtQIqsj842MrBjTwQZs=', 'Content-Length': '0'} 2024-03-04T17:39:24.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047e5e090ea5e6d42-0065e6074c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.722 DEBUG:boto:b'ppfahn-13999falsemyobjnulltrue2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:24.722 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:24.722 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:24.723 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:24.723 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:24.723 DEBUG:boto:Method: GET 2024-03-04T17:39:24.723 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:24.723 DEBUG:boto:Data: 2024-03-04T17:39:24.723 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.723 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:24.723 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.723 DEBUG:boto:Params: {} 2024-03-04T17:39:24.723 DEBUG:boto:Token: None 2024-03-04T17:39:24.724 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/?versions 2024-03-04T17:39:24.724 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZxcFA9qsgtQIqsj842MrBjTwQZs= 2024-03-04T17:39:24.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZxcFA9qsgtQIqsj842MrBjTwQZs=', 'Content-Length': '0'} 2024-03-04T17:39:24.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000143f9b6766e938d6-0065e6074c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.728 DEBUG:boto:b'ppfahn-13999falsemyobjnulltrue2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:24.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:24.729 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:24.729 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:24.729 DEBUG:boto:path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.729 DEBUG:boto:auth_path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.729 DEBUG:boto:Method: GET 2024-03-04T17:39:24.729 DEBUG:boto:Path: /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.729 DEBUG:boto:Data: 2024-03-04T17:39:24.729 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.729 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:24.730 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.730 DEBUG:boto:Params: {} 2024-03-04T17:39:24.730 DEBUG:boto:Token: None 2024-03-04T17:39:24.730 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.730 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PniVrylXrQ8GaDefDaVuNmjboZk= 2024-03-04T17:39:24.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PniVrylXrQ8GaDefDaVuNmjboZk=', 'Content-Length': '0'} 2024-03-04T17:39:24.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-13:d96636ac-054d-40e3-90f3-e3912da784b2.4349.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3c8c5e81552a8d8-0065e6074c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.776 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:24.776 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:24.776 DEBUG:boto:path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.776 DEBUG:boto:auth_path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.776 DEBUG:boto:Method: GET 2024-03-04T17:39:24.777 DEBUG:boto:Path: /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.777 DEBUG:boto:Data: 2024-03-04T17:39:24.777 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.777 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:24.777 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.777 DEBUG:boto:Params: {} 2024-03-04T17:39:24.777 DEBUG:boto:Token: None 2024-03-04T17:39:24.777 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.777 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PniVrylXrQ8GaDefDaVuNmjboZk= 2024-03-04T17:39:24.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PniVrylXrQ8GaDefDaVuNmjboZk=', 'Content-Length': '0'} 2024-03-04T17:39:24.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9bd245ada4ad532-0065e6074c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.780 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:24.781 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:24.781 DEBUG:boto:path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.781 DEBUG:boto:auth_path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.781 DEBUG:boto:Method: HEAD 2024-03-04T17:39:24.781 DEBUG:boto:Path: /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.781 DEBUG:boto:Data: 2024-03-04T17:39:24.781 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.781 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:24.781 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.781 DEBUG:boto:Params: {} 2024-03-04T17:39:24.781 DEBUG:boto:Token: None 2024-03-04T17:39:24.782 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.782 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4zSfNG/4WKTdvv4vkjvBxGhP+Ck= 2024-03-04T17:39:24.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4zSfNG/4WKTdvv4vkjvBxGhP+Ck=', 'Content-Length': '0'} 2024-03-04T17:39:24.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-13:d96636ac-054d-40e3-90f3-e3912da784b2.4349.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022f66dca356ac18c-0065e6074c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.785 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:24.785 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:24.785 DEBUG:boto:path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.785 DEBUG:boto:auth_path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.785 DEBUG:boto:Method: HEAD 2024-03-04T17:39:24.785 DEBUG:boto:Path: /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.785 DEBUG:boto:Data: 2024-03-04T17:39:24.785 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.785 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:24.785 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.785 DEBUG:boto:Params: {} 2024-03-04T17:39:24.785 DEBUG:boto:Token: None 2024-03-04T17:39:24.786 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.786 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4zSfNG/4WKTdvv4vkjvBxGhP+Ck= 2024-03-04T17:39:24.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4zSfNG/4WKTdvv4vkjvBxGhP+Ck=', 'Content-Length': '0'} 2024-03-04T17:39:24.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed7b2d172db19360-0065e6074c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:24.789 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:24.789 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:24.789 DEBUG:boto:path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.789 DEBUG:boto:auth_path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.789 DEBUG:boto:Method: GET 2024-03-04T17:39:24.789 DEBUG:boto:Path: /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.789 DEBUG:boto:Data: 2024-03-04T17:39:24.790 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.790 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:24.790 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.790 DEBUG:boto:Params: {} 2024-03-04T17:39:24.790 DEBUG:boto:Token: None 2024-03-04T17:39:24.790 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.790 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PniVrylXrQ8GaDefDaVuNmjboZk= 2024-03-04T17:39:24.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PniVrylXrQ8GaDefDaVuNmjboZk=', 'Content-Length': '0'} 2024-03-04T17:39:24.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-13:d96636ac-054d-40e3-90f3-e3912da784b2.4349.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0f553cad0ce2e72-0065e6074c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.794 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:24.794 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:24.794 DEBUG:boto:path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.794 DEBUG:boto:auth_path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.794 DEBUG:boto:Method: GET 2024-03-04T17:39:24.794 DEBUG:boto:Path: /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.794 DEBUG:boto:Data: 2024-03-04T17:39:24.794 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.794 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:24.794 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.794 DEBUG:boto:Params: {} 2024-03-04T17:39:24.795 DEBUG:boto:Token: None 2024-03-04T17:39:24.795 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:24.795 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PniVrylXrQ8GaDefDaVuNmjboZk= 2024-03-04T17:39:24.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PniVrylXrQ8GaDefDaVuNmjboZk=', 'Content-Length': '0'} 2024-03-04T17:39:24.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a23d1d7f2bc53750-0065e6074c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.840 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:24.840 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:24.840 DEBUG:boto:path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:24.840 DEBUG:boto:auth_path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:24.840 DEBUG:boto:Method: GET 2024-03-04T17:39:24.840 DEBUG:boto:Path: /ppfahn-13/?prefix=myobj 2024-03-04T17:39:24.841 DEBUG:boto:Data: 2024-03-04T17:39:24.841 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.841 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:24.841 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.841 DEBUG:boto:Params: {} 2024-03-04T17:39:24.841 DEBUG:boto:Token: None 2024-03-04T17:39:24.841 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/ 2024-03-04T17:39:24.841 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:b+BV01lwSJ8yH0Q6/GLFbeq89M0= 2024-03-04T17:39:24.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:b+BV01lwSJ8yH0Q6/GLFbeq89M0=', 'Content-Length': '0'} 2024-03-04T17:39:24.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fffb478814415faa-0065e6074c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.845 DEBUG:boto:b'ppfahn-13myobj1000falsemyobj2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:24.846 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:24.846 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:24.846 DEBUG:boto:path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:24.846 DEBUG:boto:auth_path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:24.846 DEBUG:boto:Method: GET 2024-03-04T17:39:24.846 DEBUG:boto:Path: /ppfahn-13/?prefix=myobj 2024-03-04T17:39:24.846 DEBUG:boto:Data: 2024-03-04T17:39:24.846 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.846 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:24.846 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.846 DEBUG:boto:Params: {} 2024-03-04T17:39:24.847 DEBUG:boto:Token: None 2024-03-04T17:39:24.847 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/ 2024-03-04T17:39:24.847 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:b+BV01lwSJ8yH0Q6/GLFbeq89M0= 2024-03-04T17:39:24.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:b+BV01lwSJ8yH0Q6/GLFbeq89M0=', 'Content-Length': '0'} 2024-03-04T17:39:24.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e75fc16961aa62d0-0065e6074c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.851 DEBUG:boto:b'ppfahn-13myobj1000falsemyobj2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:24.851 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:24.851 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:24.851 DEBUG:boto:Method: HEAD 2024-03-04T17:39:24.851 DEBUG:boto:Path: /ppfahn-13/myobj 2024-03-04T17:39:24.851 DEBUG:boto:Data: 2024-03-04T17:39:24.851 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.851 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:24.852 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.852 DEBUG:boto:Params: {} 2024-03-04T17:39:24.852 DEBUG:boto:Token: None 2024-03-04T17:39:24.852 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/myobj 2024-03-04T17:39:24.852 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VB/k5pxuLwMOQLWggBlVvd5EKoE= 2024-03-04T17:39:24.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VB/k5pxuLwMOQLWggBlVvd5EKoE=', 'Content-Length': '0'} 2024-03-04T17:39:24.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-13:d96636ac-054d-40e3-90f3-e3912da784b2.4349.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b347a9537e85a75-0065e6074c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.855 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:24.855 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:24.855 DEBUG:boto:Method: HEAD 2024-03-04T17:39:24.855 DEBUG:boto:Path: /ppfahn-13/myobj 2024-03-04T17:39:24.855 DEBUG:boto:Data: 2024-03-04T17:39:24.855 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.855 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:24.856 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.856 DEBUG:boto:Params: {} 2024-03-04T17:39:24.856 DEBUG:boto:Token: None 2024-03-04T17:39:24.856 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/myobj 2024-03-04T17:39:24.856 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VB/k5pxuLwMOQLWggBlVvd5EKoE= 2024-03-04T17:39:24.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VB/k5pxuLwMOQLWggBlVvd5EKoE=', 'Content-Length': '0'} 2024-03-04T17:39:24.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0d57ed0dd9f6495-0065e6074c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:24.859 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:24.859 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:24.859 DEBUG:boto:Method: GET 2024-03-04T17:39:24.859 DEBUG:boto:Path: /ppfahn-13/myobj 2024-03-04T17:39:24.859 DEBUG:boto:Data: 2024-03-04T17:39:24.859 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.859 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:24.860 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.860 DEBUG:boto:Params: {} 2024-03-04T17:39:24.860 DEBUG:boto:Token: None 2024-03-04T17:39:24.860 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/myobj 2024-03-04T17:39:24.860 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cksuWtPeZ4cSuVF4yqqnMUt4TcM= 2024-03-04T17:39:24.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cksuWtPeZ4cSuVF4yqqnMUt4TcM=', 'Content-Length': '0'} 2024-03-04T17:39:24.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-13:d96636ac-054d-40e3-90f3-e3912da784b2.4349.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff36394785003ebb-0065e6074c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.904 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:24.904 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:24.904 DEBUG:boto:Method: GET 2024-03-04T17:39:24.904 DEBUG:boto:Path: /ppfahn-13/myobj 2024-03-04T17:39:24.904 DEBUG:boto:Data: 2024-03-04T17:39:24.904 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.904 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:24.905 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.905 DEBUG:boto:Params: {} 2024-03-04T17:39:24.905 DEBUG:boto:Token: None 2024-03-04T17:39:24.905 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/myobj 2024-03-04T17:39:24.905 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cksuWtPeZ4cSuVF4yqqnMUt4TcM= 2024-03-04T17:39:24.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cksuWtPeZ4cSuVF4yqqnMUt4TcM=', 'Content-Length': '0'} 2024-03-04T17:39:24.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8e93f958ef950b2-0065e6074c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.908 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:24.908 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:24.908 DEBUG:boto:path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:24.908 DEBUG:boto:auth_path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:24.908 DEBUG:boto:Method: GET 2024-03-04T17:39:24.909 DEBUG:boto:Path: /ppfahn-13/?prefix=myobj 2024-03-04T17:39:24.909 DEBUG:boto:Data: 2024-03-04T17:39:24.909 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.909 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:24.909 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.909 DEBUG:boto:Params: {} 2024-03-04T17:39:24.909 DEBUG:boto:Token: None 2024-03-04T17:39:24.909 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/ 2024-03-04T17:39:24.909 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:b+BV01lwSJ8yH0Q6/GLFbeq89M0= 2024-03-04T17:39:24.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:b+BV01lwSJ8yH0Q6/GLFbeq89M0=', 'Content-Length': '0'} 2024-03-04T17:39:24.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fb6b1b7e7c14bd6-0065e6074c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.914 DEBUG:boto:b'ppfahn-13myobj1000falsemyobj2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:24.914 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:24.914 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:24.914 DEBUG:boto:path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:24.914 DEBUG:boto:auth_path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:24.914 DEBUG:boto:Method: GET 2024-03-04T17:39:24.914 DEBUG:boto:Path: /ppfahn-13/?prefix=myobj 2024-03-04T17:39:24.915 DEBUG:boto:Data: 2024-03-04T17:39:24.915 DEBUG:boto:Headers: {} 2024-03-04T17:39:24.915 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:24.915 DEBUG:boto:Port: 8000 2024-03-04T17:39:24.915 DEBUG:boto:Params: {} 2024-03-04T17:39:24.915 DEBUG:boto:Token: None 2024-03-04T17:39:24.915 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:24 GMT /ppfahn-13/ 2024-03-04T17:39:24.915 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:b+BV01lwSJ8yH0Q6/GLFbeq89M0= 2024-03-04T17:39:24.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:b+BV01lwSJ8yH0Q6/GLFbeq89M0=', 'Content-Length': '0'} 2024-03-04T17:39:24.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000311bf0cdfe4dbb37-0065e6074c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:24.919 DEBUG:boto:b'ppfahn-13myobj1000falsemyobj2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:24.919 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-13 zones={test-zone2, test-zone1} 2024-03-04T17:39:24.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:24.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:24.920 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:39:24.971 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:24.972 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:25.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:25.028+0000 7fcca5ee1980 1 waiting to reach incremental sync.. 2024-03-04T17:39:30.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:30.032+0000 7fcca5ee1980 1 bucket sync caught up with source: 2024-03-04T17:39:30.033 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-03-04T17:39:30.033 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000001.13.6, , , , , , , , , ] 2024-03-04T17:39:30.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:30.032+0000 7fcca5ee1980 0 bucket checkpoint complete 2024-03-04T17:39:30.066 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-13 zones={test-zone3, test-zone1} 2024-03-04T17:39:30.066 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:30.066 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:30.066 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.066 DEBUG:boto:Path: /ppfahn-13/ 2024-03-04T17:39:30.066 DEBUG:boto:Data: 2024-03-04T17:39:30.066 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.066 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.066 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.066 DEBUG:boto:Params: {} 2024-03-04T17:39:30.067 DEBUG:boto:Token: None 2024-03-04T17:39:30.067 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/ 2024-03-04T17:39:30.067 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vkqYL/Vb/uXb5LZnqtoEV70trAA= 2024-03-04T17:39:30.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vkqYL/Vb/uXb5LZnqtoEV70trAA=', 'Content-Length': '0'} 2024-03-04T17:39:30.070 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ec48772761f1fce-0065e60752-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.070 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:30.070 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:30.070 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.071 DEBUG:boto:Path: /ppfahn-13/ 2024-03-04T17:39:30.071 DEBUG:boto:Data: 2024-03-04T17:39:30.071 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.071 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.071 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.071 DEBUG:boto:Params: {} 2024-03-04T17:39:30.071 DEBUG:boto:Token: None 2024-03-04T17:39:30.071 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/ 2024-03-04T17:39:30.071 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vkqYL/Vb/uXb5LZnqtoEV70trAA= 2024-03-04T17:39:30.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vkqYL/Vb/uXb5LZnqtoEV70trAA=', 'Content-Length': '0'} 2024-03-04T17:39:30.074 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011084b5c2af62fb6-0065e60752-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.074 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:39:30.074 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:30.075 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:30.075 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:30.075 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:30.075 DEBUG:boto:Method: GET 2024-03-04T17:39:30.075 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:30.075 DEBUG:boto:Data: 2024-03-04T17:39:30.075 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.075 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.075 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.075 DEBUG:boto:Params: {} 2024-03-04T17:39:30.075 DEBUG:boto:Token: None 2024-03-04T17:39:30.076 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/?versions 2024-03-04T17:39:30.076 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YQW0bHu+kz3BwCiXLOTKZpY3OdU= 2024-03-04T17:39:30.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YQW0bHu+kz3BwCiXLOTKZpY3OdU=', 'Content-Length': '0'} 2024-03-04T17:39:30.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018cdedbc8c2e8914-0065e60752-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.120 DEBUG:boto:b'ppfahn-13999falsemyobjnulltrue2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.120 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:30.121 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:39:30.121 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:30.121 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:30.121 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:30.121 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:30.121 DEBUG:boto:Method: GET 2024-03-04T17:39:30.121 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:30.121 DEBUG:boto:Data: 2024-03-04T17:39:30.121 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.121 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.121 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.122 DEBUG:boto:Params: {} 2024-03-04T17:39:30.122 DEBUG:boto:Token: None 2024-03-04T17:39:30.122 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/?versions 2024-03-04T17:39:30.122 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YQW0bHu+kz3BwCiXLOTKZpY3OdU= 2024-03-04T17:39:30.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YQW0bHu+kz3BwCiXLOTKZpY3OdU=', 'Content-Length': '0'} 2024-03-04T17:39:30.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052f7a9493738e772-0065e60752-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.125 DEBUG:boto:b'ppfahn-13999falsemyobjnulltrue2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.126 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:30.126 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:30.126 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:30.126 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:30.126 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:30.126 DEBUG:boto:Method: GET 2024-03-04T17:39:30.126 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:30.126 DEBUG:boto:Data: 2024-03-04T17:39:30.126 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.126 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.127 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.127 DEBUG:boto:Params: {} 2024-03-04T17:39:30.127 DEBUG:boto:Token: None 2024-03-04T17:39:30.127 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/?versions 2024-03-04T17:39:30.127 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YQW0bHu+kz3BwCiXLOTKZpY3OdU= 2024-03-04T17:39:30.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YQW0bHu+kz3BwCiXLOTKZpY3OdU=', 'Content-Length': '0'} 2024-03-04T17:39:30.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a8b0f6ed5189ff6-0065e60752-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.130 DEBUG:boto:b'ppfahn-13999falsemyobjnulltrue2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.131 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:30.131 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:30.131 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:30.131 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:30.131 DEBUG:boto:Method: GET 2024-03-04T17:39:30.131 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:39:30.131 DEBUG:boto:Data: 2024-03-04T17:39:30.131 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.131 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.132 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.132 DEBUG:boto:Params: {} 2024-03-04T17:39:30.132 DEBUG:boto:Token: None 2024-03-04T17:39:30.132 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/?versions 2024-03-04T17:39:30.132 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YQW0bHu+kz3BwCiXLOTKZpY3OdU= 2024-03-04T17:39:30.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YQW0bHu+kz3BwCiXLOTKZpY3OdU=', 'Content-Length': '0'} 2024-03-04T17:39:30.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009523e7d6fc066dab-0065e60752-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.176 DEBUG:boto:b'ppfahn-13999falsemyobjnulltrue2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:30.177 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:30.177 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:30.177 DEBUG:boto:path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.177 DEBUG:boto:auth_path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.177 DEBUG:boto:Method: GET 2024-03-04T17:39:30.177 DEBUG:boto:Path: /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.177 DEBUG:boto:Data: 2024-03-04T17:39:30.177 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.177 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.178 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.178 DEBUG:boto:Params: {} 2024-03-04T17:39:30.178 DEBUG:boto:Token: None 2024-03-04T17:39:30.178 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.178 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5zbiMd1EfbNgAtmbl9s+byd80Tg= 2024-03-04T17:39:30.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5zbiMd1EfbNgAtmbl9s+byd80Tg=', 'Content-Length': '0'} 2024-03-04T17:39:30.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-13:d96636ac-054d-40e3-90f3-e3912da784b2.4349.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a578c3cc211cb3e3-0065e60752-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.182 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:30.182 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:30.182 DEBUG:boto:path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.182 DEBUG:boto:auth_path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.182 DEBUG:boto:Method: GET 2024-03-04T17:39:30.182 DEBUG:boto:Path: /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.182 DEBUG:boto:Data: 2024-03-04T17:39:30.182 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.182 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.183 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.183 DEBUG:boto:Params: {} 2024-03-04T17:39:30.183 DEBUG:boto:Token: None 2024-03-04T17:39:30.183 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.183 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5zbiMd1EfbNgAtmbl9s+byd80Tg= 2024-03-04T17:39:30.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5zbiMd1EfbNgAtmbl9s+byd80Tg=', 'Content-Length': '0'} 2024-03-04T17:39:30.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c65c3c06896c7fb-0065e60752-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.186 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:30.187 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:30.187 DEBUG:boto:path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.187 DEBUG:boto:auth_path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.187 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.187 DEBUG:boto:Path: /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.187 DEBUG:boto:Data: 2024-03-04T17:39:30.187 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.187 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.187 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.187 DEBUG:boto:Params: {} 2024-03-04T17:39:30.188 DEBUG:boto:Token: None 2024-03-04T17:39:30.188 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.188 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DDMEs/2uXgxBNwle/Fzz4PEE/t8= 2024-03-04T17:39:30.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DDMEs/2uXgxBNwle/Fzz4PEE/t8=', 'Content-Length': '0'} 2024-03-04T17:39:30.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-13:d96636ac-054d-40e3-90f3-e3912da784b2.4349.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009aa585536714aece-0065e60752-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.191 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:30.191 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:30.191 DEBUG:boto:path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.191 DEBUG:boto:auth_path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.191 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.191 DEBUG:boto:Path: /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.191 DEBUG:boto:Data: 2024-03-04T17:39:30.192 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.192 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.192 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.192 DEBUG:boto:Params: {} 2024-03-04T17:39:30.192 DEBUG:boto:Token: None 2024-03-04T17:39:30.192 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.192 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DDMEs/2uXgxBNwle/Fzz4PEE/t8= 2024-03-04T17:39:30.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DDMEs/2uXgxBNwle/Fzz4PEE/t8=', 'Content-Length': '0'} 2024-03-04T17:39:30.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf582bdb6df0d168-0065e60752-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:30.195 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:30.195 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:30.196 DEBUG:boto:path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.196 DEBUG:boto:auth_path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.196 DEBUG:boto:Method: GET 2024-03-04T17:39:30.196 DEBUG:boto:Path: /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.196 DEBUG:boto:Data: 2024-03-04T17:39:30.196 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.196 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.196 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.196 DEBUG:boto:Params: {} 2024-03-04T17:39:30.196 DEBUG:boto:Token: None 2024-03-04T17:39:30.197 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.197 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5zbiMd1EfbNgAtmbl9s+byd80Tg= 2024-03-04T17:39:30.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5zbiMd1EfbNgAtmbl9s+byd80Tg=', 'Content-Length': '0'} 2024-03-04T17:39:30.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-13:d96636ac-054d-40e3-90f3-e3912da784b2.4349.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e75a4ad59468a1a8-0065e60752-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.200 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:30.200 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:30.200 DEBUG:boto:path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.200 DEBUG:boto:auth_path=/ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.200 DEBUG:boto:Method: GET 2024-03-04T17:39:30.200 DEBUG:boto:Path: /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.200 DEBUG:boto:Data: 2024-03-04T17:39:30.200 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.201 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.201 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.201 DEBUG:boto:Params: {} 2024-03-04T17:39:30.201 DEBUG:boto:Token: None 2024-03-04T17:39:30.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/myobj?versionId=null 2024-03-04T17:39:30.201 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5zbiMd1EfbNgAtmbl9s+byd80Tg= 2024-03-04T17:39:30.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5zbiMd1EfbNgAtmbl9s+byd80Tg=', 'Content-Length': '0'} 2024-03-04T17:39:30.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b8c6898b01c9313-0065e60752-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.204 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:30.204 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:30.204 DEBUG:boto:path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:30.205 DEBUG:boto:auth_path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:30.205 DEBUG:boto:Method: GET 2024-03-04T17:39:30.205 DEBUG:boto:Path: /ppfahn-13/?prefix=myobj 2024-03-04T17:39:30.205 DEBUG:boto:Data: 2024-03-04T17:39:30.205 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.205 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.205 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.205 DEBUG:boto:Params: {} 2024-03-04T17:39:30.205 DEBUG:boto:Token: None 2024-03-04T17:39:30.205 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/ 2024-03-04T17:39:30.206 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HhcNvW9w65aSbSGEpcSfrPHEtq0= 2024-03-04T17:39:30.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HhcNvW9w65aSbSGEpcSfrPHEtq0=', 'Content-Length': '0'} 2024-03-04T17:39:30.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba4904b4b686c797-0065e60752-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.210 DEBUG:boto:b'ppfahn-13myobj1000falsemyobj2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.210 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:30.210 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:30.210 DEBUG:boto:path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:30.210 DEBUG:boto:auth_path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:30.210 DEBUG:boto:Method: GET 2024-03-04T17:39:30.211 DEBUG:boto:Path: /ppfahn-13/?prefix=myobj 2024-03-04T17:39:30.211 DEBUG:boto:Data: 2024-03-04T17:39:30.211 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.211 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.211 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.211 DEBUG:boto:Params: {} 2024-03-04T17:39:30.211 DEBUG:boto:Token: None 2024-03-04T17:39:30.211 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/ 2024-03-04T17:39:30.212 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HhcNvW9w65aSbSGEpcSfrPHEtq0= 2024-03-04T17:39:30.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HhcNvW9w65aSbSGEpcSfrPHEtq0=', 'Content-Length': '0'} 2024-03-04T17:39:30.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae20821977fb67ab-0065e60752-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.256 DEBUG:boto:b'ppfahn-13myobj1000falsemyobj2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.256 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:30.256 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:30.256 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.257 DEBUG:boto:Path: /ppfahn-13/myobj 2024-03-04T17:39:30.257 DEBUG:boto:Data: 2024-03-04T17:39:30.257 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.257 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.257 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.257 DEBUG:boto:Params: {} 2024-03-04T17:39:30.257 DEBUG:boto:Token: None 2024-03-04T17:39:30.257 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/myobj 2024-03-04T17:39:30.257 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8Lw98KNycVX9z5CzXXJAlCsJJNY= 2024-03-04T17:39:30.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8Lw98KNycVX9z5CzXXJAlCsJJNY=', 'Content-Length': '0'} 2024-03-04T17:39:30.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-13:d96636ac-054d-40e3-90f3-e3912da784b2.4349.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcb7f2c06e682008-0065e60752-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.260 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:30.260 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:30.260 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.261 DEBUG:boto:Path: /ppfahn-13/myobj 2024-03-04T17:39:30.261 DEBUG:boto:Data: 2024-03-04T17:39:30.261 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.261 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.261 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.261 DEBUG:boto:Params: {} 2024-03-04T17:39:30.261 DEBUG:boto:Token: None 2024-03-04T17:39:30.261 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/myobj 2024-03-04T17:39:30.261 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8Lw98KNycVX9z5CzXXJAlCsJJNY= 2024-03-04T17:39:30.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8Lw98KNycVX9z5CzXXJAlCsJJNY=', 'Content-Length': '0'} 2024-03-04T17:39:30.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c53a227bc27684fd-0065e60752-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:30.264 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:30.265 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:30.265 DEBUG:boto:Method: GET 2024-03-04T17:39:30.265 DEBUG:boto:Path: /ppfahn-13/myobj 2024-03-04T17:39:30.265 DEBUG:boto:Data: 2024-03-04T17:39:30.265 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.265 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.265 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.265 DEBUG:boto:Params: {} 2024-03-04T17:39:30.265 DEBUG:boto:Token: None 2024-03-04T17:39:30.265 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/myobj 2024-03-04T17:39:30.266 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ypNhd+iJ/MzvwWIzvdhfpGoyvjs= 2024-03-04T17:39:30.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ypNhd+iJ/MzvwWIzvdhfpGoyvjs=', 'Content-Length': '0'} 2024-03-04T17:39:30.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-13:d96636ac-054d-40e3-90f3-e3912da784b2.4349.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086a2cbd2395db658-0065e60752-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.312 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:30.312 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:30.312 DEBUG:boto:Method: GET 2024-03-04T17:39:30.312 DEBUG:boto:Path: /ppfahn-13/myobj 2024-03-04T17:39:30.312 DEBUG:boto:Data: 2024-03-04T17:39:30.313 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.313 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.313 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.313 DEBUG:boto:Params: {} 2024-03-04T17:39:30.313 DEBUG:boto:Token: None 2024-03-04T17:39:30.313 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/myobj 2024-03-04T17:39:30.313 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ypNhd+iJ/MzvwWIzvdhfpGoyvjs= 2024-03-04T17:39:30.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ypNhd+iJ/MzvwWIzvdhfpGoyvjs=', 'Content-Length': '0'} 2024-03-04T17:39:30.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025a5bc5b314bad37-0065e60752-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.316 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:30.316 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:30.316 DEBUG:boto:path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:30.316 DEBUG:boto:auth_path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:30.317 DEBUG:boto:Method: GET 2024-03-04T17:39:30.317 DEBUG:boto:Path: /ppfahn-13/?prefix=myobj 2024-03-04T17:39:30.317 DEBUG:boto:Data: 2024-03-04T17:39:30.317 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.317 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.317 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.317 DEBUG:boto:Params: {} 2024-03-04T17:39:30.317 DEBUG:boto:Token: None 2024-03-04T17:39:30.317 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/ 2024-03-04T17:39:30.317 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HhcNvW9w65aSbSGEpcSfrPHEtq0= 2024-03-04T17:39:30.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HhcNvW9w65aSbSGEpcSfrPHEtq0=', 'Content-Length': '0'} 2024-03-04T17:39:30.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b683a91f219c885c-0065e60752-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.322 DEBUG:boto:b'ppfahn-13myobj1000falsemyobj2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.322 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:30.322 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:30.322 DEBUG:boto:path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:30.322 DEBUG:boto:auth_path=/ppfahn-13/?prefix=myobj 2024-03-04T17:39:30.322 DEBUG:boto:Method: GET 2024-03-04T17:39:30.322 DEBUG:boto:Path: /ppfahn-13/?prefix=myobj 2024-03-04T17:39:30.322 DEBUG:boto:Data: 2024-03-04T17:39:30.323 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.323 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.323 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.323 DEBUG:boto:Params: {} 2024-03-04T17:39:30.323 DEBUG:boto:Token: None 2024-03-04T17:39:30.323 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-13/ 2024-03-04T17:39:30.323 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HhcNvW9w65aSbSGEpcSfrPHEtq0= 2024-03-04T17:39:30.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HhcNvW9w65aSbSGEpcSfrPHEtq0=', 'Content-Length': '0'} 2024-03-04T17:39:30.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa768f37f191ca3a-0065e60752-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.327 DEBUG:boto:b'ppfahn-13myobj1000falsemyobj2024-03-04T17:39:18.982Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.327 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-13 zones={test-zone3, test-zone1} 2024-03-04T17:39:30.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:30.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:30.328 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:39:30.377 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:30.377 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:30.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:30.440+0000 7f292a18d980 1 bucket sync caught up with source: 2024-03-04T17:39:30.443 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:39:30.443 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:39:30.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:30.440+0000 7f292a18d980 0 bucket checkpoint complete 2024-03-04T17:39:30.465 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-14 zones={test-zone1, test-zone2} 2024-03-04T17:39:30.465 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.466 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.466 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.466 DEBUG:boto:Path: /ppfahn-14/ 2024-03-04T17:39:30.466 DEBUG:boto:Data: 2024-03-04T17:39:30.466 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.466 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.466 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.466 DEBUG:boto:Params: {} 2024-03-04T17:39:30.466 DEBUG:boto:Token: None 2024-03-04T17:39:30.467 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/ 2024-03-04T17:39:30.467 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DkoEBQy6ejBAYlxFYTsuvt9J0/I= 2024-03-04T17:39:30.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DkoEBQy6ejBAYlxFYTsuvt9J0/I=', 'Content-Length': '0'} 2024-03-04T17:39:30.469 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000043a45800a68dcfc-0065e60752-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.469 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.469 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.470 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.470 DEBUG:boto:Path: /ppfahn-14/ 2024-03-04T17:39:30.470 DEBUG:boto:Data: 2024-03-04T17:39:30.470 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.470 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.470 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.470 DEBUG:boto:Params: {} 2024-03-04T17:39:30.470 DEBUG:boto:Token: None 2024-03-04T17:39:30.470 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/ 2024-03-04T17:39:30.471 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DkoEBQy6ejBAYlxFYTsuvt9J0/I= 2024-03-04T17:39:30.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DkoEBQy6ejBAYlxFYTsuvt9J0/I=', 'Content-Length': '0'} 2024-03-04T17:39:30.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000782f979018524325-0065e60752-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.475 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:39:30.475 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.475 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.475 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.475 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.476 DEBUG:boto:Method: GET 2024-03-04T17:39:30.476 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.476 DEBUG:boto:Data: 2024-03-04T17:39:30.476 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.476 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.476 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.476 DEBUG:boto:Params: {} 2024-03-04T17:39:30.476 DEBUG:boto:Token: None 2024-03-04T17:39:30.476 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/?versions 2024-03-04T17:39:30.477 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE= 2024-03-04T17:39:30.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE=', 'Content-Length': '0'} 2024-03-04T17:39:30.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af1f48d5a1530f08-0065e60752-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.520 DEBUG:boto:b'ppfahn-14999falsemyobjnulltrue2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.543 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:30.544 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:39:30.544 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.544 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.544 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.544 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.544 DEBUG:boto:Method: GET 2024-03-04T17:39:30.544 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.544 DEBUG:boto:Data: 2024-03-04T17:39:30.544 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.545 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.545 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.545 DEBUG:boto:Params: {} 2024-03-04T17:39:30.545 DEBUG:boto:Token: None 2024-03-04T17:39:30.545 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/?versions 2024-03-04T17:39:30.545 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE= 2024-03-04T17:39:30.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE=', 'Content-Length': '0'} 2024-03-04T17:39:30.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099dfe3cb0c4b230e-0065e60752-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.550 DEBUG:boto:b'ppfahn-14999falsemyobjnulltrue2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.550 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:30.550 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.550 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.550 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.550 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.551 DEBUG:boto:Method: GET 2024-03-04T17:39:30.551 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.551 DEBUG:boto:Data: 2024-03-04T17:39:30.551 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.551 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.551 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.551 DEBUG:boto:Params: {} 2024-03-04T17:39:30.551 DEBUG:boto:Token: None 2024-03-04T17:39:30.551 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/?versions 2024-03-04T17:39:30.552 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE= 2024-03-04T17:39:30.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE=', 'Content-Length': '0'} 2024-03-04T17:39:30.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061c3ed2f0503491c-0065e60752-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.556 DEBUG:boto:b'ppfahn-14999falsemyobjnulltrue2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.556 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.556 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.557 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.557 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.557 DEBUG:boto:Method: GET 2024-03-04T17:39:30.557 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.557 DEBUG:boto:Data: 2024-03-04T17:39:30.557 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.557 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.557 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.557 DEBUG:boto:Params: {} 2024-03-04T17:39:30.557 DEBUG:boto:Token: None 2024-03-04T17:39:30.558 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/?versions 2024-03-04T17:39:30.558 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE= 2024-03-04T17:39:30.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE=', 'Content-Length': '0'} 2024-03-04T17:39:30.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070ca25028b60b628-0065e60752-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.562 DEBUG:boto:b'ppfahn-14999falsemyobjnulltrue2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:30.562 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.562 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.563 DEBUG:boto:path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.563 DEBUG:boto:auth_path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.563 DEBUG:boto:Method: GET 2024-03-04T17:39:30.563 DEBUG:boto:Path: /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.563 DEBUG:boto:Data: 2024-03-04T17:39:30.563 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.563 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.563 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.563 DEBUG:boto:Params: {} 2024-03-04T17:39:30.563 DEBUG:boto:Token: None 2024-03-04T17:39:30.563 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.564 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg= 2024-03-04T17:39:30.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg=', 'Content-Length': '0'} 2024-03-04T17:39:30.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-14:d96636ac-054d-40e3-90f3-e3912da784b2.4349.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000677d283f75c7d340-0065e60752-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.567 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.567 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.567 DEBUG:boto:path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.567 DEBUG:boto:auth_path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.567 DEBUG:boto:Method: GET 2024-03-04T17:39:30.567 DEBUG:boto:Path: /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.567 DEBUG:boto:Data: 2024-03-04T17:39:30.567 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.567 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.568 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.568 DEBUG:boto:Params: {} 2024-03-04T17:39:30.568 DEBUG:boto:Token: None 2024-03-04T17:39:30.568 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.568 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg= 2024-03-04T17:39:30.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg=', 'Content-Length': '0'} 2024-03-04T17:39:30.571 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000417d55f0913632ca-0065e60752-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.612 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.612 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.612 DEBUG:boto:path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.612 DEBUG:boto:auth_path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.613 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.613 DEBUG:boto:Path: /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.613 DEBUG:boto:Data: 2024-03-04T17:39:30.613 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.613 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.613 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.613 DEBUG:boto:Params: {} 2024-03-04T17:39:30.613 DEBUG:boto:Token: None 2024-03-04T17:39:30.613 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.614 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uRHCl+wVfNzvs5yN/UzyvPiKJsQ= 2024-03-04T17:39:30.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uRHCl+wVfNzvs5yN/UzyvPiKJsQ=', 'Content-Length': '0'} 2024-03-04T17:39:30.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-14:d96636ac-054d-40e3-90f3-e3912da784b2.4349.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a605ce4cf641c0be-0065e60752-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.616 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.617 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.617 DEBUG:boto:path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.617 DEBUG:boto:auth_path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.617 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.617 DEBUG:boto:Path: /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.617 DEBUG:boto:Data: 2024-03-04T17:39:30.617 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.617 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.617 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.617 DEBUG:boto:Params: {} 2024-03-04T17:39:30.617 DEBUG:boto:Token: None 2024-03-04T17:39:30.618 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.618 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uRHCl+wVfNzvs5yN/UzyvPiKJsQ= 2024-03-04T17:39:30.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uRHCl+wVfNzvs5yN/UzyvPiKJsQ=', 'Content-Length': '0'} 2024-03-04T17:39:30.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002914a04cedfe6ab1-0065e60752-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:30.621 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.621 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.621 DEBUG:boto:path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.621 DEBUG:boto:auth_path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.621 DEBUG:boto:Method: GET 2024-03-04T17:39:30.622 DEBUG:boto:Path: /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.622 DEBUG:boto:Data: 2024-03-04T17:39:30.622 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.622 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.622 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.622 DEBUG:boto:Params: {} 2024-03-04T17:39:30.622 DEBUG:boto:Token: None 2024-03-04T17:39:30.622 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.622 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg= 2024-03-04T17:39:30.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg=', 'Content-Length': '0'} 2024-03-04T17:39:30.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-14:d96636ac-054d-40e3-90f3-e3912da784b2.4349.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cbafc727d3befb7-0065e60752-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.668 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.668 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.668 DEBUG:boto:path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.668 DEBUG:boto:auth_path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.668 DEBUG:boto:Method: GET 2024-03-04T17:39:30.669 DEBUG:boto:Path: /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.669 DEBUG:boto:Data: 2024-03-04T17:39:30.669 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.669 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.669 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.669 DEBUG:boto:Params: {} 2024-03-04T17:39:30.669 DEBUG:boto:Token: None 2024-03-04T17:39:30.669 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.669 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg= 2024-03-04T17:39:30.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg=', 'Content-Length': '0'} 2024-03-04T17:39:30.672 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b9db729d052e51f-0065e60752-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.673 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.673 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.673 DEBUG:boto:path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:30.673 DEBUG:boto:auth_path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:30.673 DEBUG:boto:Method: GET 2024-03-04T17:39:30.673 DEBUG:boto:Path: /ppfahn-14/?prefix=myobj 2024-03-04T17:39:30.673 DEBUG:boto:Data: 2024-03-04T17:39:30.673 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.673 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.673 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.674 DEBUG:boto:Params: {} 2024-03-04T17:39:30.674 DEBUG:boto:Token: None 2024-03-04T17:39:30.674 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/ 2024-03-04T17:39:30.674 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fR0gA9LmSZN/Tdq1td3+uD9TqXs= 2024-03-04T17:39:30.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fR0gA9LmSZN/Tdq1td3+uD9TqXs=', 'Content-Length': '0'} 2024-03-04T17:39:30.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dc3c43aa800f82d-0065e60752-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.678 DEBUG:boto:b'ppfahn-14myobj1000falsemyobj2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.679 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.679 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.679 DEBUG:boto:path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:30.679 DEBUG:boto:auth_path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:30.679 DEBUG:boto:Method: GET 2024-03-04T17:39:30.679 DEBUG:boto:Path: /ppfahn-14/?prefix=myobj 2024-03-04T17:39:30.679 DEBUG:boto:Data: 2024-03-04T17:39:30.679 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.679 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.679 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.680 DEBUG:boto:Params: {} 2024-03-04T17:39:30.680 DEBUG:boto:Token: None 2024-03-04T17:39:30.680 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/ 2024-03-04T17:39:30.680 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fR0gA9LmSZN/Tdq1td3+uD9TqXs= 2024-03-04T17:39:30.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fR0gA9LmSZN/Tdq1td3+uD9TqXs=', 'Content-Length': '0'} 2024-03-04T17:39:30.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff2f9a7c1fd43561-0065e60752-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.684 DEBUG:boto:b'ppfahn-14myobj1000falsemyobj2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.685 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.685 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.685 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.685 DEBUG:boto:Path: /ppfahn-14/myobj 2024-03-04T17:39:30.685 DEBUG:boto:Data: 2024-03-04T17:39:30.685 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.685 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.685 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.686 DEBUG:boto:Params: {} 2024-03-04T17:39:30.686 DEBUG:boto:Token: None 2024-03-04T17:39:30.686 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj 2024-03-04T17:39:30.686 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Jato9ydsTZu18GlRlmz3cYG9Xf0= 2024-03-04T17:39:30.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Jato9ydsTZu18GlRlmz3cYG9Xf0=', 'Content-Length': '0'} 2024-03-04T17:39:30.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-14:d96636ac-054d-40e3-90f3-e3912da784b2.4349.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbd9d3f66471ca57-0065e60752-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.689 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.689 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.689 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.689 DEBUG:boto:Path: /ppfahn-14/myobj 2024-03-04T17:39:30.689 DEBUG:boto:Data: 2024-03-04T17:39:30.689 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.689 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.689 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.690 DEBUG:boto:Params: {} 2024-03-04T17:39:30.690 DEBUG:boto:Token: None 2024-03-04T17:39:30.690 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj 2024-03-04T17:39:30.690 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Jato9ydsTZu18GlRlmz3cYG9Xf0= 2024-03-04T17:39:30.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Jato9ydsTZu18GlRlmz3cYG9Xf0=', 'Content-Length': '0'} 2024-03-04T17:39:30.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005747cbf2781f6bb8-0065e60752-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:30.693 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.693 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.693 DEBUG:boto:Method: GET 2024-03-04T17:39:30.693 DEBUG:boto:Path: /ppfahn-14/myobj 2024-03-04T17:39:30.694 DEBUG:boto:Data: 2024-03-04T17:39:30.694 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.694 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.694 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.694 DEBUG:boto:Params: {} 2024-03-04T17:39:30.694 DEBUG:boto:Token: None 2024-03-04T17:39:30.694 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj 2024-03-04T17:39:30.694 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dVs9jsmOnVhn0/vkQW7JhgoaTCM= 2024-03-04T17:39:30.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dVs9jsmOnVhn0/vkQW7JhgoaTCM=', 'Content-Length': '0'} 2024-03-04T17:39:30.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-14:d96636ac-054d-40e3-90f3-e3912da784b2.4349.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004f0c7f023e9e217-0065e60752-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.740 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.740 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.740 DEBUG:boto:Method: GET 2024-03-04T17:39:30.740 DEBUG:boto:Path: /ppfahn-14/myobj 2024-03-04T17:39:30.740 DEBUG:boto:Data: 2024-03-04T17:39:30.741 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.741 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.741 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.741 DEBUG:boto:Params: {} 2024-03-04T17:39:30.741 DEBUG:boto:Token: None 2024-03-04T17:39:30.741 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj 2024-03-04T17:39:30.741 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dVs9jsmOnVhn0/vkQW7JhgoaTCM= 2024-03-04T17:39:30.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dVs9jsmOnVhn0/vkQW7JhgoaTCM=', 'Content-Length': '0'} 2024-03-04T17:39:30.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001827b5b24d753786-0065e60752-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.744 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.744 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.745 DEBUG:boto:path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:30.745 DEBUG:boto:auth_path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:30.745 DEBUG:boto:Method: GET 2024-03-04T17:39:30.745 DEBUG:boto:Path: /ppfahn-14/?prefix=myobj 2024-03-04T17:39:30.745 DEBUG:boto:Data: 2024-03-04T17:39:30.745 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.745 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:30.745 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.745 DEBUG:boto:Params: {} 2024-03-04T17:39:30.745 DEBUG:boto:Token: None 2024-03-04T17:39:30.745 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/ 2024-03-04T17:39:30.746 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fR0gA9LmSZN/Tdq1td3+uD9TqXs= 2024-03-04T17:39:30.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fR0gA9LmSZN/Tdq1td3+uD9TqXs=', 'Content-Length': '0'} 2024-03-04T17:39:30.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009ae0fc7de643364-0065e60752-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.749 DEBUG:boto:b'ppfahn-14myobj1000falsemyobj2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.750 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.750 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.750 DEBUG:boto:path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:30.750 DEBUG:boto:auth_path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:30.750 DEBUG:boto:Method: GET 2024-03-04T17:39:30.750 DEBUG:boto:Path: /ppfahn-14/?prefix=myobj 2024-03-04T17:39:30.750 DEBUG:boto:Data: 2024-03-04T17:39:30.751 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.751 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.751 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.751 DEBUG:boto:Params: {} 2024-03-04T17:39:30.751 DEBUG:boto:Token: None 2024-03-04T17:39:30.751 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/ 2024-03-04T17:39:30.751 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fR0gA9LmSZN/Tdq1td3+uD9TqXs= 2024-03-04T17:39:30.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fR0gA9LmSZN/Tdq1td3+uD9TqXs=', 'Content-Length': '0'} 2024-03-04T17:39:30.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026bb9966b1ba7627-0065e60752-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.755 DEBUG:boto:b'ppfahn-14myobj1000falsemyobj2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.756 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-14 zones={test-zone1, test-zone2} 2024-03-04T17:39:30.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:30.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:30.756 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:39:30.806 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:30.806 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:30.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:30.868+0000 7f0b4f2a3980 1 bucket sync caught up with source: 2024-03-04T17:39:30.869 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:39:30.869 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:39:30.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:30.868+0000 7f0b4f2a3980 0 bucket checkpoint complete 2024-03-04T17:39:30.891 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-14 zones={test-zone3, test-zone2} 2024-03-04T17:39:30.891 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.891 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.891 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.891 DEBUG:boto:Path: /ppfahn-14/ 2024-03-04T17:39:30.891 DEBUG:boto:Data: 2024-03-04T17:39:30.891 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.891 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.892 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.892 DEBUG:boto:Params: {} 2024-03-04T17:39:30.892 DEBUG:boto:Token: None 2024-03-04T17:39:30.892 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/ 2024-03-04T17:39:30.892 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DkoEBQy6ejBAYlxFYTsuvt9J0/I= 2024-03-04T17:39:30.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DkoEBQy6ejBAYlxFYTsuvt9J0/I=', 'Content-Length': '0'} 2024-03-04T17:39:30.894 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008be14ff33b0cd043-0065e60752-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.895 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.895 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.895 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.895 DEBUG:boto:Path: /ppfahn-14/ 2024-03-04T17:39:30.895 DEBUG:boto:Data: 2024-03-04T17:39:30.895 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.895 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.895 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.895 DEBUG:boto:Params: {} 2024-03-04T17:39:30.895 DEBUG:boto:Token: None 2024-03-04T17:39:30.896 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/ 2024-03-04T17:39:30.896 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DkoEBQy6ejBAYlxFYTsuvt9J0/I= 2024-03-04T17:39:30.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DkoEBQy6ejBAYlxFYTsuvt9J0/I=', 'Content-Length': '0'} 2024-03-04T17:39:30.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085c88fe34d9d1562-0065e60752-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.900 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:39:30.900 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.900 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.901 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.901 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.901 DEBUG:boto:Method: GET 2024-03-04T17:39:30.901 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.901 DEBUG:boto:Data: 2024-03-04T17:39:30.901 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.901 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.901 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.901 DEBUG:boto:Params: {} 2024-03-04T17:39:30.901 DEBUG:boto:Token: None 2024-03-04T17:39:30.902 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/?versions 2024-03-04T17:39:30.902 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE= 2024-03-04T17:39:30.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE=', 'Content-Length': '0'} 2024-03-04T17:39:30.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd813e048a1b9e15-0065e60752-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.904 DEBUG:boto:b'ppfahn-14999falsemyobjnulltrue2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.904 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:30.904 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:39:30.905 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.905 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.905 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.905 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.905 DEBUG:boto:Method: GET 2024-03-04T17:39:30.905 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.905 DEBUG:boto:Data: 2024-03-04T17:39:30.905 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.905 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.905 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.906 DEBUG:boto:Params: {} 2024-03-04T17:39:30.906 DEBUG:boto:Token: None 2024-03-04T17:39:30.906 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/?versions 2024-03-04T17:39:30.906 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE= 2024-03-04T17:39:30.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE=', 'Content-Length': '0'} 2024-03-04T17:39:30.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020f35f11900014a7-0065e60752-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.911 DEBUG:boto:b'ppfahn-14999falsemyobjnulltrue2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.911 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:30.911 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.911 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.911 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.911 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.911 DEBUG:boto:Method: GET 2024-03-04T17:39:30.912 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.912 DEBUG:boto:Data: 2024-03-04T17:39:30.912 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.912 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.912 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.912 DEBUG:boto:Params: {} 2024-03-04T17:39:30.912 DEBUG:boto:Token: None 2024-03-04T17:39:30.912 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/?versions 2024-03-04T17:39:30.912 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE= 2024-03-04T17:39:30.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE=', 'Content-Length': '0'} 2024-03-04T17:39:30.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cef4bdd10fe4362f-0065e60752-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.915 DEBUG:boto:b'ppfahn-14999falsemyobjnulltrue2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.915 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:30.915 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:30.915 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.915 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.916 DEBUG:boto:Method: GET 2024-03-04T17:39:30.916 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:39:30.916 DEBUG:boto:Data: 2024-03-04T17:39:30.916 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.916 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.916 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.916 DEBUG:boto:Params: {} 2024-03-04T17:39:30.916 DEBUG:boto:Token: None 2024-03-04T17:39:30.916 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/?versions 2024-03-04T17:39:30.917 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE= 2024-03-04T17:39:30.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+XDSG4m7pnZgh6IpQTwowG1LxwE=', 'Content-Length': '0'} 2024-03-04T17:39:30.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af147a691691de20-0065e60752-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.964 DEBUG:boto:b'ppfahn-14999falsemyobjnulltrue2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:30.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:30.964 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.964 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.965 DEBUG:boto:path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.965 DEBUG:boto:auth_path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.965 DEBUG:boto:Method: GET 2024-03-04T17:39:30.965 DEBUG:boto:Path: /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.965 DEBUG:boto:Data: 2024-03-04T17:39:30.965 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.965 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.965 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.965 DEBUG:boto:Params: {} 2024-03-04T17:39:30.965 DEBUG:boto:Token: None 2024-03-04T17:39:30.966 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.966 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg= 2024-03-04T17:39:30.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg=', 'Content-Length': '0'} 2024-03-04T17:39:30.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-14:d96636ac-054d-40e3-90f3-e3912da784b2.4349.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061825e24e5dd904f-0065e60752-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.969 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.970 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.970 DEBUG:boto:path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.970 DEBUG:boto:auth_path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.970 DEBUG:boto:Method: GET 2024-03-04T17:39:30.970 DEBUG:boto:Path: /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.970 DEBUG:boto:Data: 2024-03-04T17:39:30.970 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.970 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.970 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.970 DEBUG:boto:Params: {} 2024-03-04T17:39:30.970 DEBUG:boto:Token: None 2024-03-04T17:39:30.971 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.971 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg= 2024-03-04T17:39:30.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg=', 'Content-Length': '0'} 2024-03-04T17:39:30.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042bb6bd2b25b2d64-0065e60752-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.974 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.974 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.974 DEBUG:boto:path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.974 DEBUG:boto:auth_path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.974 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.974 DEBUG:boto:Path: /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.974 DEBUG:boto:Data: 2024-03-04T17:39:30.974 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.975 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.975 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.975 DEBUG:boto:Params: {} 2024-03-04T17:39:30.975 DEBUG:boto:Token: None 2024-03-04T17:39:30.975 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.975 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uRHCl+wVfNzvs5yN/UzyvPiKJsQ= 2024-03-04T17:39:30.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uRHCl+wVfNzvs5yN/UzyvPiKJsQ=', 'Content-Length': '0'} 2024-03-04T17:39:30.978 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-14:d96636ac-054d-40e3-90f3-e3912da784b2.4349.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077f8a9ff6ce0a5a8-0065e60752-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.978 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.978 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.978 DEBUG:boto:path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.979 DEBUG:boto:auth_path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.979 DEBUG:boto:Method: HEAD 2024-03-04T17:39:30.979 DEBUG:boto:Path: /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.979 DEBUG:boto:Data: 2024-03-04T17:39:30.979 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.979 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:30.979 DEBUG:boto:Port: 8000 2024-03-04T17:39:30.979 DEBUG:boto:Params: {} 2024-03-04T17:39:30.979 DEBUG:boto:Token: None 2024-03-04T17:39:30.979 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.980 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uRHCl+wVfNzvs5yN/UzyvPiKJsQ= 2024-03-04T17:39:30.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uRHCl+wVfNzvs5yN/UzyvPiKJsQ=', 'Content-Length': '0'} 2024-03-04T17:39:30.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000647dec817c6e5141-0065e60752-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:30.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:30.983 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:30.983 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:30.983 DEBUG:boto:path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.983 DEBUG:boto:auth_path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.983 DEBUG:boto:Method: GET 2024-03-04T17:39:30.983 DEBUG:boto:Path: /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.984 DEBUG:boto:Data: 2024-03-04T17:39:30.984 DEBUG:boto:Headers: {} 2024-03-04T17:39:30.984 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:30.984 DEBUG:boto:Port: 8001 2024-03-04T17:39:30.984 DEBUG:boto:Params: {} 2024-03-04T17:39:30.984 DEBUG:boto:Token: None 2024-03-04T17:39:30.984 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:30 GMT /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:30.984 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg= 2024-03-04T17:39:30.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EjVgwjIYPSxWgYNegkuZzTG0Zyg=', 'Content-Length': '0'} 2024-03-04T17:39:30.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-14:d96636ac-054d-40e3-90f3-e3912da784b2.4349.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db07142d6f628ece-0065e60752-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.028 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:31.028 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:31.028 DEBUG:boto:path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:31.028 DEBUG:boto:auth_path=/ppfahn-14/myobj?versionId=null 2024-03-04T17:39:31.028 DEBUG:boto:Method: GET 2024-03-04T17:39:31.029 DEBUG:boto:Path: /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:31.029 DEBUG:boto:Data: 2024-03-04T17:39:31.029 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.029 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.029 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.029 DEBUG:boto:Params: {} 2024-03-04T17:39:31.029 DEBUG:boto:Token: None 2024-03-04T17:39:31.029 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-14/myobj?versionId=null 2024-03-04T17:39:31.029 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T/JYTQRFhUhVoLNpcU+x7YG2w5g= 2024-03-04T17:39:31.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T/JYTQRFhUhVoLNpcU+x7YG2w5g=', 'Content-Length': '0'} 2024-03-04T17:39:31.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034929db63916b37a-0065e60753-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.033 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:31.033 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:31.033 DEBUG:boto:path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:31.033 DEBUG:boto:auth_path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:31.033 DEBUG:boto:Method: GET 2024-03-04T17:39:31.033 DEBUG:boto:Path: /ppfahn-14/?prefix=myobj 2024-03-04T17:39:31.033 DEBUG:boto:Data: 2024-03-04T17:39:31.033 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.033 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.034 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.034 DEBUG:boto:Params: {} 2024-03-04T17:39:31.034 DEBUG:boto:Token: None 2024-03-04T17:39:31.034 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-14/ 2024-03-04T17:39:31.034 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3o0bMixj1f3FrwCA3lrZBGuFJns= 2024-03-04T17:39:31.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3o0bMixj1f3FrwCA3lrZBGuFJns=', 'Content-Length': '0'} 2024-03-04T17:39:31.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000751e32dccec38b42-0065e60753-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.039 DEBUG:boto:b'ppfahn-14myobj1000falsemyobj2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.039 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:31.039 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:31.039 DEBUG:boto:path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:31.039 DEBUG:boto:auth_path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:31.039 DEBUG:boto:Method: GET 2024-03-04T17:39:31.040 DEBUG:boto:Path: /ppfahn-14/?prefix=myobj 2024-03-04T17:39:31.040 DEBUG:boto:Data: 2024-03-04T17:39:31.040 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.040 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.040 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.040 DEBUG:boto:Params: {} 2024-03-04T17:39:31.040 DEBUG:boto:Token: None 2024-03-04T17:39:31.040 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-14/ 2024-03-04T17:39:31.040 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3o0bMixj1f3FrwCA3lrZBGuFJns= 2024-03-04T17:39:31.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3o0bMixj1f3FrwCA3lrZBGuFJns=', 'Content-Length': '0'} 2024-03-04T17:39:31.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dccdd1a8a5cb88d4-0065e60753-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.088 DEBUG:boto:b'ppfahn-14myobj1000falsemyobj2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.088 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:31.088 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:31.089 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.089 DEBUG:boto:Path: /ppfahn-14/myobj 2024-03-04T17:39:31.089 DEBUG:boto:Data: 2024-03-04T17:39:31.089 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.089 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.089 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.089 DEBUG:boto:Params: {} 2024-03-04T17:39:31.089 DEBUG:boto:Token: None 2024-03-04T17:39:31.089 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-14/myobj 2024-03-04T17:39:31.089 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:81dxyy4j0YXOTmnb9o6Ge75rZDs= 2024-03-04T17:39:31.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:81dxyy4j0YXOTmnb9o6Ge75rZDs=', 'Content-Length': '0'} 2024-03-04T17:39:31.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-14:d96636ac-054d-40e3-90f3-e3912da784b2.4349.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027edf6934ff72bbe-0065e60753-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.093 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:31.093 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:31.093 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.093 DEBUG:boto:Path: /ppfahn-14/myobj 2024-03-04T17:39:31.093 DEBUG:boto:Data: 2024-03-04T17:39:31.093 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.093 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.093 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.093 DEBUG:boto:Params: {} 2024-03-04T17:39:31.093 DEBUG:boto:Token: None 2024-03-04T17:39:31.094 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-14/myobj 2024-03-04T17:39:31.094 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:81dxyy4j0YXOTmnb9o6Ge75rZDs= 2024-03-04T17:39:31.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:81dxyy4j0YXOTmnb9o6Ge75rZDs=', 'Content-Length': '0'} 2024-03-04T17:39:31.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000804ca6424f3e2438-0065e60753-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:31.097 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:31.097 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:31.097 DEBUG:boto:Method: GET 2024-03-04T17:39:31.097 DEBUG:boto:Path: /ppfahn-14/myobj 2024-03-04T17:39:31.097 DEBUG:boto:Data: 2024-03-04T17:39:31.097 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.097 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.097 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.097 DEBUG:boto:Params: {} 2024-03-04T17:39:31.098 DEBUG:boto:Token: None 2024-03-04T17:39:31.098 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-14/myobj 2024-03-04T17:39:31.098 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UD5WSU+0vLpLG+Py7+qvMIKTq9U= 2024-03-04T17:39:31.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UD5WSU+0vLpLG+Py7+qvMIKTq9U=', 'Content-Length': '0'} 2024-03-04T17:39:31.100 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-14:d96636ac-054d-40e3-90f3-e3912da784b2.4349.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004941f96aefb52bab-0065e60753-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.144 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:39:31.144 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:39:31.144 DEBUG:boto:Method: GET 2024-03-04T17:39:31.144 DEBUG:boto:Path: /ppfahn-14/myobj 2024-03-04T17:39:31.144 DEBUG:boto:Data: 2024-03-04T17:39:31.145 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.145 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.145 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.145 DEBUG:boto:Params: {} 2024-03-04T17:39:31.145 DEBUG:boto:Token: None 2024-03-04T17:39:31.145 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-14/myobj 2024-03-04T17:39:31.145 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UD5WSU+0vLpLG+Py7+qvMIKTq9U= 2024-03-04T17:39:31.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UD5WSU+0vLpLG+Py7+qvMIKTq9U=', 'Content-Length': '0'} 2024-03-04T17:39:31.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a38608a15b2b9675-0065e60753-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.148 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:31.148 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:31.148 DEBUG:boto:path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:31.148 DEBUG:boto:auth_path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:31.148 DEBUG:boto:Method: GET 2024-03-04T17:39:31.149 DEBUG:boto:Path: /ppfahn-14/?prefix=myobj 2024-03-04T17:39:31.149 DEBUG:boto:Data: 2024-03-04T17:39:31.149 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.149 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.149 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.149 DEBUG:boto:Params: {} 2024-03-04T17:39:31.149 DEBUG:boto:Token: None 2024-03-04T17:39:31.149 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-14/ 2024-03-04T17:39:31.149 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3o0bMixj1f3FrwCA3lrZBGuFJns= 2024-03-04T17:39:31.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3o0bMixj1f3FrwCA3lrZBGuFJns=', 'Content-Length': '0'} 2024-03-04T17:39:31.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032f2629ec2889684-0065e60753-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.196 DEBUG:boto:b'ppfahn-14myobj1000falsemyobj2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.196 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:39:31.196 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:39:31.197 DEBUG:boto:path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:31.197 DEBUG:boto:auth_path=/ppfahn-14/?prefix=myobj 2024-03-04T17:39:31.197 DEBUG:boto:Method: GET 2024-03-04T17:39:31.197 DEBUG:boto:Path: /ppfahn-14/?prefix=myobj 2024-03-04T17:39:31.197 DEBUG:boto:Data: 2024-03-04T17:39:31.197 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.197 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.197 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.197 DEBUG:boto:Params: {} 2024-03-04T17:39:31.197 DEBUG:boto:Token: None 2024-03-04T17:39:31.197 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-14/ 2024-03-04T17:39:31.198 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3o0bMixj1f3FrwCA3lrZBGuFJns= 2024-03-04T17:39:31.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3o0bMixj1f3FrwCA3lrZBGuFJns=', 'Content-Length': '0'} 2024-03-04T17:39:31.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fed3f9ec223756b-0065e60753-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.202 DEBUG:boto:b'ppfahn-14myobj1000falsemyobj2024-03-04T17:39:18.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.202 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-14 zones={test-zone3, test-zone2} 2024-03-04T17:39:31.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:31.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:31.203 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:39:31.252 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:31.253 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:31.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:31.316+0000 7f05a995d980 1 bucket sync caught up with source: 2024-03-04T17:39:31.315 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:39:31.316 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:39:31.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:39:31.316+0000 7f05a995d980 0 bucket checkpoint complete 2024-03-04T17:39:31.337 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-15 zones={test-zone1, test-zone3} 2024-03-04T17:39:31.337 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.337 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.337 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.337 DEBUG:boto:Path: /ppfahn-15/ 2024-03-04T17:39:31.338 DEBUG:boto:Data: 2024-03-04T17:39:31.338 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.338 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:31.338 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.338 DEBUG:boto:Params: {} 2024-03-04T17:39:31.338 DEBUG:boto:Token: None 2024-03-04T17:39:31.338 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/ 2024-03-04T17:39:31.338 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7sZm/jRH8ZO0+20Z0RUTee+JCek= 2024-03-04T17:39:31.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7sZm/jRH8ZO0+20Z0RUTee+JCek=', 'Content-Length': '0'} 2024-03-04T17:39:31.341 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f280d19f0484bfd7-0065e60753-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.341 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.341 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.341 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.341 DEBUG:boto:Path: /ppfahn-15/ 2024-03-04T17:39:31.341 DEBUG:boto:Data: 2024-03-04T17:39:31.341 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.341 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.341 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.342 DEBUG:boto:Params: {} 2024-03-04T17:39:31.342 DEBUG:boto:Token: None 2024-03-04T17:39:31.342 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/ 2024-03-04T17:39:31.342 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7sZm/jRH8ZO0+20Z0RUTee+JCek= 2024-03-04T17:39:31.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7sZm/jRH8ZO0+20Z0RUTee+JCek=', 'Content-Length': '0'} 2024-03-04T17:39:31.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c2fbfb754ea192b-0065e60753-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.344 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:39:31.344 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.345 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.345 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.345 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.345 DEBUG:boto:Method: GET 2024-03-04T17:39:31.345 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.345 DEBUG:boto:Data: 2024-03-04T17:39:31.345 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.345 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:31.345 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.345 DEBUG:boto:Params: {} 2024-03-04T17:39:31.345 DEBUG:boto:Token: None 2024-03-04T17:39:31.346 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/?versions 2024-03-04T17:39:31.346 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0= 2024-03-04T17:39:31.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0=', 'Content-Length': '0'} 2024-03-04T17:39:31.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000871db738c35b35f7-0065e60753-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.348 DEBUG:boto:b'ppfahn-15999falsemyobjnulltrue2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.348 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:31.348 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:39:31.349 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.349 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.349 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.349 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.349 DEBUG:boto:Method: GET 2024-03-04T17:39:31.349 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.349 DEBUG:boto:Data: 2024-03-04T17:39:31.349 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.349 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.349 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.350 DEBUG:boto:Params: {} 2024-03-04T17:39:31.350 DEBUG:boto:Token: None 2024-03-04T17:39:31.350 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/?versions 2024-03-04T17:39:31.350 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0= 2024-03-04T17:39:31.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0=', 'Content-Length': '0'} 2024-03-04T17:39:31.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f53dbfe77d23b71-0065e60753-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.352 DEBUG:boto:b'ppfahn-15999falsemyobjnulltrue2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.352 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:31.353 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.353 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.353 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.353 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.353 DEBUG:boto:Method: GET 2024-03-04T17:39:31.353 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.353 DEBUG:boto:Data: 2024-03-04T17:39:31.353 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.353 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:31.354 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.354 DEBUG:boto:Params: {} 2024-03-04T17:39:31.354 DEBUG:boto:Token: None 2024-03-04T17:39:31.354 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/?versions 2024-03-04T17:39:31.354 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0= 2024-03-04T17:39:31.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0=', 'Content-Length': '0'} 2024-03-04T17:39:31.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048faaae9ddec7718-0065e60753-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.400 DEBUG:boto:b'ppfahn-15999falsemyobjnulltrue2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.400 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.401 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.401 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.401 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.401 DEBUG:boto:Method: GET 2024-03-04T17:39:31.401 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.401 DEBUG:boto:Data: 2024-03-04T17:39:31.401 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.401 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.401 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.401 DEBUG:boto:Params: {} 2024-03-04T17:39:31.402 DEBUG:boto:Token: None 2024-03-04T17:39:31.402 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/?versions 2024-03-04T17:39:31.402 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0= 2024-03-04T17:39:31.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0=', 'Content-Length': '0'} 2024-03-04T17:39:31.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da784a62b4d99a33-0065e60753-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.405 DEBUG:boto:b'ppfahn-15999falsemyobjnulltrue2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:31.405 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.405 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.405 DEBUG:boto:path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.406 DEBUG:boto:auth_path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.406 DEBUG:boto:Method: GET 2024-03-04T17:39:31.406 DEBUG:boto:Path: /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.406 DEBUG:boto:Data: 2024-03-04T17:39:31.406 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.406 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:31.406 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.406 DEBUG:boto:Params: {} 2024-03-04T17:39:31.406 DEBUG:boto:Token: None 2024-03-04T17:39:31.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.406 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM= 2024-03-04T17:39:31.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM=', 'Content-Length': '0'} 2024-03-04T17:39:31.409 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-15:d96636ac-054d-40e3-90f3-e3912da784b2.4349.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad556d79d1b9d240-0065e60753-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.410 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.410 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.410 DEBUG:boto:path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.410 DEBUG:boto:auth_path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.410 DEBUG:boto:Method: GET 2024-03-04T17:39:31.410 DEBUG:boto:Path: /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.410 DEBUG:boto:Data: 2024-03-04T17:39:31.410 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.410 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.410 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.411 DEBUG:boto:Params: {} 2024-03-04T17:39:31.411 DEBUG:boto:Token: None 2024-03-04T17:39:31.411 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.411 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM= 2024-03-04T17:39:31.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM=', 'Content-Length': '0'} 2024-03-04T17:39:31.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa5a2c2782acccd9-0065e60753-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.456 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.456 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.456 DEBUG:boto:path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.456 DEBUG:boto:auth_path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.457 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.457 DEBUG:boto:Path: /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.457 DEBUG:boto:Data: 2024-03-04T17:39:31.457 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.457 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:31.457 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.457 DEBUG:boto:Params: {} 2024-03-04T17:39:31.457 DEBUG:boto:Token: None 2024-03-04T17:39:31.457 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.457 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ip7s9kd6J2HSi2ED627UI0ms0fk= 2024-03-04T17:39:31.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ip7s9kd6J2HSi2ED627UI0ms0fk=', 'Content-Length': '0'} 2024-03-04T17:39:31.460 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-15:d96636ac-054d-40e3-90f3-e3912da784b2.4349.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028f56b69ece8f7f3-0065e60753-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.461 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.461 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.461 DEBUG:boto:path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.461 DEBUG:boto:auth_path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.461 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.461 DEBUG:boto:Path: /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.461 DEBUG:boto:Data: 2024-03-04T17:39:31.461 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.461 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.462 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.462 DEBUG:boto:Params: {} 2024-03-04T17:39:31.462 DEBUG:boto:Token: None 2024-03-04T17:39:31.462 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.462 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ip7s9kd6J2HSi2ED627UI0ms0fk= 2024-03-04T17:39:31.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ip7s9kd6J2HSi2ED627UI0ms0fk=', 'Content-Length': '0'} 2024-03-04T17:39:31.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b5400f3e2c78d66-0065e60753-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:31.465 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.465 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.466 DEBUG:boto:path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.466 DEBUG:boto:auth_path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.466 DEBUG:boto:Method: GET 2024-03-04T17:39:31.466 DEBUG:boto:Path: /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.466 DEBUG:boto:Data: 2024-03-04T17:39:31.466 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.466 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:31.466 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.466 DEBUG:boto:Params: {} 2024-03-04T17:39:31.466 DEBUG:boto:Token: None 2024-03-04T17:39:31.466 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.467 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM= 2024-03-04T17:39:31.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM=', 'Content-Length': '0'} 2024-03-04T17:39:31.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-15:d96636ac-054d-40e3-90f3-e3912da784b2.4349.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000923fc23866d36f63-0065e60753-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.470 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.470 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.470 DEBUG:boto:path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.470 DEBUG:boto:auth_path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.470 DEBUG:boto:Method: GET 2024-03-04T17:39:31.470 DEBUG:boto:Path: /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.470 DEBUG:boto:Data: 2024-03-04T17:39:31.470 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.470 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.471 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.471 DEBUG:boto:Params: {} 2024-03-04T17:39:31.471 DEBUG:boto:Token: None 2024-03-04T17:39:31.471 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.471 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM= 2024-03-04T17:39:31.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM=', 'Content-Length': '0'} 2024-03-04T17:39:31.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000550f904d8d58bcbf-0065e60753-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.474 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.475 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.475 DEBUG:boto:path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.475 DEBUG:boto:auth_path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.475 DEBUG:boto:Method: GET 2024-03-04T17:39:31.475 DEBUG:boto:Path: /ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.475 DEBUG:boto:Data: 2024-03-04T17:39:31.475 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.475 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:31.475 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.475 DEBUG:boto:Params: {} 2024-03-04T17:39:31.476 DEBUG:boto:Token: None 2024-03-04T17:39:31.476 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/ 2024-03-04T17:39:31.476 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA= 2024-03-04T17:39:31.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA=', 'Content-Length': '0'} 2024-03-04T17:39:31.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006593a8c06c0af994-0065e60753-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.524 DEBUG:boto:b'ppfahn-15myobj1000falsemyobj2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.524 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.524 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.525 DEBUG:boto:path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.525 DEBUG:boto:auth_path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.525 DEBUG:boto:Method: GET 2024-03-04T17:39:31.525 DEBUG:boto:Path: /ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.525 DEBUG:boto:Data: 2024-03-04T17:39:31.525 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.525 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.525 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.525 DEBUG:boto:Params: {} 2024-03-04T17:39:31.525 DEBUG:boto:Token: None 2024-03-04T17:39:31.526 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/ 2024-03-04T17:39:31.526 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA= 2024-03-04T17:39:31.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA=', 'Content-Length': '0'} 2024-03-04T17:39:31.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026f9472cd37b03a2-0065e60753-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.530 DEBUG:boto:b'ppfahn-15myobj1000falsemyobj2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.530 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.530 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.530 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.531 DEBUG:boto:Path: /ppfahn-15/myobj 2024-03-04T17:39:31.531 DEBUG:boto:Data: 2024-03-04T17:39:31.531 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.531 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:31.531 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.531 DEBUG:boto:Params: {} 2024-03-04T17:39:31.531 DEBUG:boto:Token: None 2024-03-04T17:39:31.531 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj 2024-03-04T17:39:31.531 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:z4XCK//nQ3nTi2y4NdLfOG57280= 2024-03-04T17:39:31.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:z4XCK//nQ3nTi2y4NdLfOG57280=', 'Content-Length': '0'} 2024-03-04T17:39:31.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-15:d96636ac-054d-40e3-90f3-e3912da784b2.4349.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e33635426b06b8f-0065e60753-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.534 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.534 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.534 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.535 DEBUG:boto:Path: /ppfahn-15/myobj 2024-03-04T17:39:31.535 DEBUG:boto:Data: 2024-03-04T17:39:31.535 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.535 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.535 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.535 DEBUG:boto:Params: {} 2024-03-04T17:39:31.535 DEBUG:boto:Token: None 2024-03-04T17:39:31.535 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj 2024-03-04T17:39:31.535 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:z4XCK//nQ3nTi2y4NdLfOG57280= 2024-03-04T17:39:31.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:z4XCK//nQ3nTi2y4NdLfOG57280=', 'Content-Length': '0'} 2024-03-04T17:39:31.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000942ab7af305a4ac1-0065e60753-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:31.538 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.538 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.538 DEBUG:boto:Method: GET 2024-03-04T17:39:31.538 DEBUG:boto:Path: /ppfahn-15/myobj 2024-03-04T17:39:31.539 DEBUG:boto:Data: 2024-03-04T17:39:31.539 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.539 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:31.539 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.539 DEBUG:boto:Params: {} 2024-03-04T17:39:31.539 DEBUG:boto:Token: None 2024-03-04T17:39:31.539 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj 2024-03-04T17:39:31.539 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3hKciDUnA0wQkekN7yLp8Vsn/Q= 2024-03-04T17:39:31.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3hKciDUnA0wQkekN7yLp8Vsn/Q=', 'Content-Length': '0'} 2024-03-04T17:39:31.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-15:d96636ac-054d-40e3-90f3-e3912da784b2.4349.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9bb6f331a1951af-0065e60753-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.542 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.542 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.542 DEBUG:boto:Method: GET 2024-03-04T17:39:31.542 DEBUG:boto:Path: /ppfahn-15/myobj 2024-03-04T17:39:31.542 DEBUG:boto:Data: 2024-03-04T17:39:31.542 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.543 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.543 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.543 DEBUG:boto:Params: {} 2024-03-04T17:39:31.543 DEBUG:boto:Token: None 2024-03-04T17:39:31.543 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj 2024-03-04T17:39:31.543 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3hKciDUnA0wQkekN7yLp8Vsn/Q= 2024-03-04T17:39:31.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3hKciDUnA0wQkekN7yLp8Vsn/Q=', 'Content-Length': '0'} 2024-03-04T17:39:31.546 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd2f7babfc06998e-0065e60753-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.588 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.588 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.588 DEBUG:boto:path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.588 DEBUG:boto:auth_path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.589 DEBUG:boto:Method: GET 2024-03-04T17:39:31.589 DEBUG:boto:Path: /ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.589 DEBUG:boto:Data: 2024-03-04T17:39:31.589 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.589 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:31.589 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.589 DEBUG:boto:Params: {} 2024-03-04T17:39:31.589 DEBUG:boto:Token: None 2024-03-04T17:39:31.589 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/ 2024-03-04T17:39:31.589 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA= 2024-03-04T17:39:31.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA=', 'Content-Length': '0'} 2024-03-04T17:39:31.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006976ff3160fdcd3b-0065e60753-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.594 DEBUG:boto:b'ppfahn-15myobj1000falsemyobj2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.594 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.594 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.594 DEBUG:boto:path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.594 DEBUG:boto:auth_path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.594 DEBUG:boto:Method: GET 2024-03-04T17:39:31.595 DEBUG:boto:Path: /ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.595 DEBUG:boto:Data: 2024-03-04T17:39:31.595 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.595 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.595 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.595 DEBUG:boto:Params: {} 2024-03-04T17:39:31.595 DEBUG:boto:Token: None 2024-03-04T17:39:31.595 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/ 2024-03-04T17:39:31.595 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA= 2024-03-04T17:39:31.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA=', 'Content-Length': '0'} 2024-03-04T17:39:31.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc9ba43d6a0c650d-0065e60753-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.600 DEBUG:boto:b'ppfahn-15myobj1000falsemyobj2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.600 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-15 zones={test-zone1, test-zone3} 2024-03-04T17:39:31.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:31.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:31.601 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:39:31.651 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:31.651 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:31.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:31.711+0000 7f7c5dcfd980 1 bucket sync caught up with source: 2024-03-04T17:39:31.714 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:39:31.714 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:39:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:31.711+0000 7f7c5dcfd980 0 bucket checkpoint complete 2024-03-04T17:39:31.736 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-15 zones={test-zone2, test-zone3} 2024-03-04T17:39:31.736 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.736 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.737 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.737 DEBUG:boto:Path: /ppfahn-15/ 2024-03-04T17:39:31.737 DEBUG:boto:Data: 2024-03-04T17:39:31.737 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.737 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.737 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.737 DEBUG:boto:Params: {} 2024-03-04T17:39:31.737 DEBUG:boto:Token: None 2024-03-04T17:39:31.737 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/ 2024-03-04T17:39:31.737 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7sZm/jRH8ZO0+20Z0RUTee+JCek= 2024-03-04T17:39:31.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7sZm/jRH8ZO0+20Z0RUTee+JCek=', 'Content-Length': '0'} 2024-03-04T17:39:31.740 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a7fca3ced70063d-0065e60753-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.740 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.740 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.740 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.740 DEBUG:boto:Path: /ppfahn-15/ 2024-03-04T17:39:31.740 DEBUG:boto:Data: 2024-03-04T17:39:31.740 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.740 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.741 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.741 DEBUG:boto:Params: {} 2024-03-04T17:39:31.741 DEBUG:boto:Token: None 2024-03-04T17:39:31.741 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/ 2024-03-04T17:39:31.741 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7sZm/jRH8ZO0+20Z0RUTee+JCek= 2024-03-04T17:39:31.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7sZm/jRH8ZO0+20Z0RUTee+JCek=', 'Content-Length': '0'} 2024-03-04T17:39:31.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081e05822d7f5723b-0065e60753-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.745 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:39:31.745 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.745 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.746 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.746 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.746 DEBUG:boto:Method: GET 2024-03-04T17:39:31.746 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.746 DEBUG:boto:Data: 2024-03-04T17:39:31.746 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.746 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.746 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.746 DEBUG:boto:Params: {} 2024-03-04T17:39:31.746 DEBUG:boto:Token: None 2024-03-04T17:39:31.746 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/?versions 2024-03-04T17:39:31.747 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0= 2024-03-04T17:39:31.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0=', 'Content-Length': '0'} 2024-03-04T17:39:31.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1c23aa4862a88b2-0065e60753-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.749 DEBUG:boto:b'ppfahn-15999falsemyobjnulltrue2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.749 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:31.749 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:39:31.749 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.750 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.750 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.750 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.750 DEBUG:boto:Method: GET 2024-03-04T17:39:31.750 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.750 DEBUG:boto:Data: 2024-03-04T17:39:31.750 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.750 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.750 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.750 DEBUG:boto:Params: {} 2024-03-04T17:39:31.750 DEBUG:boto:Token: None 2024-03-04T17:39:31.751 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/?versions 2024-03-04T17:39:31.751 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0= 2024-03-04T17:39:31.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0=', 'Content-Length': '0'} 2024-03-04T17:39:31.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed6994cea7ae74d9-0065e60753-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.755 DEBUG:boto:b'ppfahn-15999falsemyobjnulltrue2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.755 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T17:39:31.755 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.755 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.755 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.756 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.756 DEBUG:boto:Method: GET 2024-03-04T17:39:31.756 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.756 DEBUG:boto:Data: 2024-03-04T17:39:31.756 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.756 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.756 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.756 DEBUG:boto:Params: {} 2024-03-04T17:39:31.756 DEBUG:boto:Token: None 2024-03-04T17:39:31.756 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/?versions 2024-03-04T17:39:31.757 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0= 2024-03-04T17:39:31.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0=', 'Content-Length': '0'} 2024-03-04T17:39:31.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bce571bc039f745-0065e60753-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.759 DEBUG:boto:b'ppfahn-15999falsemyobjnulltrue2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.759 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.759 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.759 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.760 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.760 DEBUG:boto:Method: GET 2024-03-04T17:39:31.760 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:39:31.760 DEBUG:boto:Data: 2024-03-04T17:39:31.760 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.760 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.760 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.760 DEBUG:boto:Params: {} 2024-03-04T17:39:31.760 DEBUG:boto:Token: None 2024-03-04T17:39:31.760 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/?versions 2024-03-04T17:39:31.761 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0= 2024-03-04T17:39:31.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pF8Z1jhxjZ9GwA1iISF5KJ3JoM0=', 'Content-Length': '0'} 2024-03-04T17:39:31.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a19a7cdfc4912a0e-0065e60753-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.808 DEBUG:boto:b'ppfahn-15999falsemyobjnulltrue2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:31.808 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.809 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.809 DEBUG:boto:path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.809 DEBUG:boto:auth_path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.809 DEBUG:boto:Method: GET 2024-03-04T17:39:31.809 DEBUG:boto:Path: /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.809 DEBUG:boto:Data: 2024-03-04T17:39:31.809 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.809 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.809 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.809 DEBUG:boto:Params: {} 2024-03-04T17:39:31.810 DEBUG:boto:Token: None 2024-03-04T17:39:31.810 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.810 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM= 2024-03-04T17:39:31.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM=', 'Content-Length': '0'} 2024-03-04T17:39:31.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-15:d96636ac-054d-40e3-90f3-e3912da784b2.4349.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000443ba050bc46d55a-0065e60753-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.812 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.813 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.813 DEBUG:boto:path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.813 DEBUG:boto:auth_path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.813 DEBUG:boto:Method: GET 2024-03-04T17:39:31.813 DEBUG:boto:Path: /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.813 DEBUG:boto:Data: 2024-03-04T17:39:31.813 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.813 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.813 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.813 DEBUG:boto:Params: {} 2024-03-04T17:39:31.814 DEBUG:boto:Token: None 2024-03-04T17:39:31.814 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.814 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM= 2024-03-04T17:39:31.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM=', 'Content-Length': '0'} 2024-03-04T17:39:31.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006731b4b643d910b3-0065e60753-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.817 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.817 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.817 DEBUG:boto:path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.818 DEBUG:boto:auth_path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.818 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.818 DEBUG:boto:Path: /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.818 DEBUG:boto:Data: 2024-03-04T17:39:31.818 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.818 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.818 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.818 DEBUG:boto:Params: {} 2024-03-04T17:39:31.818 DEBUG:boto:Token: None 2024-03-04T17:39:31.818 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.819 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ip7s9kd6J2HSi2ED627UI0ms0fk= 2024-03-04T17:39:31.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ip7s9kd6J2HSi2ED627UI0ms0fk=', 'Content-Length': '0'} 2024-03-04T17:39:31.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-15:d96636ac-054d-40e3-90f3-e3912da784b2.4349.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000081a180d31d9e4b7-0065e60753-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.822 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.822 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.822 DEBUG:boto:path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.822 DEBUG:boto:auth_path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.822 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.822 DEBUG:boto:Path: /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.822 DEBUG:boto:Data: 2024-03-04T17:39:31.822 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.822 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.822 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.823 DEBUG:boto:Params: {} 2024-03-04T17:39:31.823 DEBUG:boto:Token: None 2024-03-04T17:39:31.823 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.823 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ip7s9kd6J2HSi2ED627UI0ms0fk= 2024-03-04T17:39:31.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ip7s9kd6J2HSi2ED627UI0ms0fk=', 'Content-Length': '0'} 2024-03-04T17:39:31.826 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005038305c70fea91e-0065e60753-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:31.826 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.826 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.826 DEBUG:boto:path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.826 DEBUG:boto:auth_path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.826 DEBUG:boto:Method: GET 2024-03-04T17:39:31.826 DEBUG:boto:Path: /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.827 DEBUG:boto:Data: 2024-03-04T17:39:31.827 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.827 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.827 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.827 DEBUG:boto:Params: {} 2024-03-04T17:39:31.827 DEBUG:boto:Token: None 2024-03-04T17:39:31.827 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.827 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM= 2024-03-04T17:39:31.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM=', 'Content-Length': '0'} 2024-03-04T17:39:31.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-15:d96636ac-054d-40e3-90f3-e3912da784b2.4349.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7a5651c188f473e-0065e60753-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.872 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.872 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.872 DEBUG:boto:path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.872 DEBUG:boto:auth_path=/ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.872 DEBUG:boto:Method: GET 2024-03-04T17:39:31.873 DEBUG:boto:Path: /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.873 DEBUG:boto:Data: 2024-03-04T17:39:31.873 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.873 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.873 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.873 DEBUG:boto:Params: {} 2024-03-04T17:39:31.873 DEBUG:boto:Token: None 2024-03-04T17:39:31.873 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj?versionId=null 2024-03-04T17:39:31.873 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM= 2024-03-04T17:39:31.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iJV4TkWuL2/Yl4XxL77pNWulIJM=', 'Content-Length': '0'} 2024-03-04T17:39:31.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083baee19bbe23288-0065e60753-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.877 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.877 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.877 DEBUG:boto:path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.877 DEBUG:boto:auth_path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.877 DEBUG:boto:Method: GET 2024-03-04T17:39:31.877 DEBUG:boto:Path: /ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.877 DEBUG:boto:Data: 2024-03-04T17:39:31.877 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.877 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.877 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.878 DEBUG:boto:Params: {} 2024-03-04T17:39:31.878 DEBUG:boto:Token: None 2024-03-04T17:39:31.878 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/ 2024-03-04T17:39:31.878 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA= 2024-03-04T17:39:31.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA=', 'Content-Length': '0'} 2024-03-04T17:39:31.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cad821f8bbdf5be2-0065e60753-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.882 DEBUG:boto:b'ppfahn-15myobj1000falsemyobj2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.882 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.883 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.883 DEBUG:boto:path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.883 DEBUG:boto:auth_path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.883 DEBUG:boto:Method: GET 2024-03-04T17:39:31.883 DEBUG:boto:Path: /ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.883 DEBUG:boto:Data: 2024-03-04T17:39:31.883 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.883 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.883 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.883 DEBUG:boto:Params: {} 2024-03-04T17:39:31.883 DEBUG:boto:Token: None 2024-03-04T17:39:31.883 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/ 2024-03-04T17:39:31.884 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA= 2024-03-04T17:39:31.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA=', 'Content-Length': '0'} 2024-03-04T17:39:31.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e7bdd70321ef4f0-0065e60753-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.928 DEBUG:boto:b'ppfahn-15myobj1000falsemyobj2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.928 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.929 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.929 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.929 DEBUG:boto:Path: /ppfahn-15/myobj 2024-03-04T17:39:31.929 DEBUG:boto:Data: 2024-03-04T17:39:31.929 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.929 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.929 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.929 DEBUG:boto:Params: {} 2024-03-04T17:39:31.929 DEBUG:boto:Token: None 2024-03-04T17:39:31.929 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj 2024-03-04T17:39:31.929 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:z4XCK//nQ3nTi2y4NdLfOG57280= 2024-03-04T17:39:31.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:z4XCK//nQ3nTi2y4NdLfOG57280=', 'Content-Length': '0'} 2024-03-04T17:39:31.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-15:d96636ac-054d-40e3-90f3-e3912da784b2.4349.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009232461a7c6a34a9-0065e60753-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.932 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.933 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.933 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.933 DEBUG:boto:Path: /ppfahn-15/myobj 2024-03-04T17:39:31.933 DEBUG:boto:Data: 2024-03-04T17:39:31.933 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.933 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.933 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.933 DEBUG:boto:Params: {} 2024-03-04T17:39:31.933 DEBUG:boto:Token: None 2024-03-04T17:39:31.933 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj 2024-03-04T17:39:31.933 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:z4XCK//nQ3nTi2y4NdLfOG57280= 2024-03-04T17:39:31.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:z4XCK//nQ3nTi2y4NdLfOG57280=', 'Content-Length': '0'} 2024-03-04T17:39:31.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abdaee8cb55d9794-0065e60753-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T17:39:31.937 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.937 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.937 DEBUG:boto:Method: GET 2024-03-04T17:39:31.937 DEBUG:boto:Path: /ppfahn-15/myobj 2024-03-04T17:39:31.937 DEBUG:boto:Data: 2024-03-04T17:39:31.937 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.937 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.937 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.938 DEBUG:boto:Params: {} 2024-03-04T17:39:31.938 DEBUG:boto:Token: None 2024-03-04T17:39:31.938 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj 2024-03-04T17:39:31.938 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3hKciDUnA0wQkekN7yLp8Vsn/Q= 2024-03-04T17:39:31.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3hKciDUnA0wQkekN7yLp8Vsn/Q=', 'Content-Length': '0'} 2024-03-04T17:39:31.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-15:d96636ac-054d-40e3-90f3-e3912da784b2.4349.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bdcabf9d9caf215-0065e60753-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.941 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:39:31.941 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:39:31.941 DEBUG:boto:Method: GET 2024-03-04T17:39:31.942 DEBUG:boto:Path: /ppfahn-15/myobj 2024-03-04T17:39:31.942 DEBUG:boto:Data: 2024-03-04T17:39:31.942 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.942 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.942 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.942 DEBUG:boto:Params: {} 2024-03-04T17:39:31.942 DEBUG:boto:Token: None 2024-03-04T17:39:31.942 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/myobj 2024-03-04T17:39:31.942 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3hKciDUnA0wQkekN7yLp8Vsn/Q= 2024-03-04T17:39:31.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3hKciDUnA0wQkekN7yLp8Vsn/Q=', 'Content-Length': '0'} 2024-03-04T17:39:31.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7f99b8148bdb65e-0065e60753-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.946 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.946 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.946 DEBUG:boto:path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.946 DEBUG:boto:auth_path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.946 DEBUG:boto:Method: GET 2024-03-04T17:39:31.946 DEBUG:boto:Path: /ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.946 DEBUG:boto:Data: 2024-03-04T17:39:31.946 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.946 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:39:31.946 DEBUG:boto:Port: 8000 2024-03-04T17:39:31.946 DEBUG:boto:Params: {} 2024-03-04T17:39:31.947 DEBUG:boto:Token: None 2024-03-04T17:39:31.947 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/ 2024-03-04T17:39:31.947 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA= 2024-03-04T17:39:31.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA=', 'Content-Length': '0'} 2024-03-04T17:39:31.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079632dd65721c842-0065e60753-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.992 DEBUG:boto:b'ppfahn-15myobj1000falsemyobj2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.992 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:39:31.993 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:39:31.993 DEBUG:boto:path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.993 DEBUG:boto:auth_path=/ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.993 DEBUG:boto:Method: GET 2024-03-04T17:39:31.993 DEBUG:boto:Path: /ppfahn-15/?prefix=myobj 2024-03-04T17:39:31.993 DEBUG:boto:Data: 2024-03-04T17:39:31.993 DEBUG:boto:Headers: {} 2024-03-04T17:39:31.993 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:39:31.993 DEBUG:boto:Port: 8001 2024-03-04T17:39:31.993 DEBUG:boto:Params: {} 2024-03-04T17:39:31.993 DEBUG:boto:Token: None 2024-03-04T17:39:31.994 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:39:31 GMT /ppfahn-15/ 2024-03-04T17:39:31.994 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA= 2024-03-04T17:39:31.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6Ijcs8ODtvrdxUClwULLWenwAQA=', 'Content-Length': '0'} 2024-03-04T17:39:31.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f44eaee4e24562a-0065e60753-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:31.999 DEBUG:boto:b'ppfahn-15myobj1000falsemyobj2024-03-04T17:39:19.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:39:31.999 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-15 zones={test-zone2, test-zone3} 2024-03-04T17:39:31.999 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:39:31.999 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:39:31.999 DEBUG:boto:Method: HEAD 2024-03-04T17:39:31.999 DEBUG:boto:Path: /ppfahn-13/ 2024-03-04T17:39:31.999 DEBUG:boto:Data: 2024-03-04T17:39:32.000 DEBUG:boto:Headers: {} 2024-03-04T17:39:32.000 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:32.000 DEBUG:boto:Port: 8000 2024-03-04T17:39:32.000 DEBUG:boto:Params: {} 2024-03-04T17:39:32.000 DEBUG:boto:Token: None 2024-03-04T17:39:32.000 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:32 GMT /ppfahn-13/ 2024-03-04T17:39:32.000 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IYsWip1u1STrmJHB/2p9bSfXnXA= 2024-03-04T17:39:32.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IYsWip1u1STrmJHB/2p9bSfXnXA=', 'Content-Length': '0'} 2024-03-04T17:39:32.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e5672c181b89e63-0065e60754-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:32.005 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:32.005 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:32.005 DEBUG:boto:Method: HEAD 2024-03-04T17:39:32.005 DEBUG:boto:Path: /ppfahn-13/myobj 2024-03-04T17:39:32.005 DEBUG:boto:Data: 2024-03-04T17:39:32.005 DEBUG:boto:Headers: {} 2024-03-04T17:39:32.006 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:32.006 DEBUG:boto:Port: 8000 2024-03-04T17:39:32.006 DEBUG:boto:Params: {} 2024-03-04T17:39:32.006 DEBUG:boto:Token: None 2024-03-04T17:39:32.006 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:39:32 GMT /ppfahn-13/myobj 2024-03-04T17:39:32.006 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xVXgej7RIxrrRTctFCW2wINQoH0= 2024-03-04T17:39:32.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xVXgej7RIxrrRTctFCW2wINQoH0=', 'Content-Length': '0'} 2024-03-04T17:39:32.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000216ebf4755e5451c-0065e60754-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:32.009 DEBUG:boto:path=/ppfahn-13/myobj 2024-03-04T17:39:32.009 DEBUG:boto:auth_path=/ppfahn-13/myobj 2024-03-04T17:39:32.009 DEBUG:boto:Method: DELETE 2024-03-04T17:39:32.009 DEBUG:boto:Path: /ppfahn-13/myobj 2024-03-04T17:39:32.009 DEBUG:boto:Data: 2024-03-04T17:39:32.010 DEBUG:boto:Headers: {} 2024-03-04T17:39:32.010 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:39:32.010 DEBUG:boto:Port: 8000 2024-03-04T17:39:32.010 DEBUG:boto:Params: {} 2024-03-04T17:39:32.010 DEBUG:boto:Token: None 2024-03-04T17:39:32.010 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:39:32 GMT /ppfahn-13/myobj 2024-03-04T17:39:32.010 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hqpm2MS3GSkGG7vXPVk1EjoTgTU= 2024-03-04T17:39:32.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:39:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hqpm2MS3GSkGG7vXPVk1EjoTgTU=', 'Content-Length': '0'} 2024-03-04T17:39:32.018 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006a82a4f6242f469a-0065e60754-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:39:32.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:32.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:39:32.019 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:39:32.069 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:32.069 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:32.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:32.131+0000 7f97a933e980 1 waiting for incremental sync to catch up: 2024-03-04T17:39:32.135 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-03-04T17:39:32.135 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-03-04T17:39:37.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:37.131+0000 7f97a933e980 1 waiting for incremental sync to catch up: 2024-03-04T17:39:37.137 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-03-04T17:39:37.137 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-03-04T17:39:42.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:42.135+0000 7f97a933e980 1 waiting for incremental sync to catch up: 2024-03-04T17:39:42.140 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-03-04T17:39:42.140 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-03-04T17:39:47.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:47.139+0000 7f97a933e980 1 waiting for incremental sync to catch up: 2024-03-04T17:39:47.142 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-03-04T17:39:47.142 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-03-04T17:39:52.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:52.143+0000 7f97a933e980 1 waiting for incremental sync to catch up: 2024-03-04T17:39:52.144 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-03-04T17:39:52.144 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-03-04T17:39:57.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:39:57.147+0000 7f97a933e980 1 waiting for incremental sync to catch up: 2024-03-04T17:39:57.147 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-03-04T17:39:57.147 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-03-04T17:40:02.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:40:02.147+0000 7f97a933e980 1 bucket sync caught up with source: 2024-03-04T17:40:02.149 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000002.15.6, , , , , , , , , ] 2024-03-04T17:40:02.149 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-03-04T17:40:02.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:40:02.147+0000 7f97a933e980 0 bucket checkpoint complete 2024-03-04T17:40:02.190 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-13 zones={test-zone2, test-zone1} 2024-03-04T17:40:02.190 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:40:02.190 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:40:02.190 DEBUG:boto:Method: HEAD 2024-03-04T17:40:02.190 DEBUG:boto:Path: /ppfahn-13/ 2024-03-04T17:40:02.190 DEBUG:boto:Data: 2024-03-04T17:40:02.191 DEBUG:boto:Headers: {} 2024-03-04T17:40:02.191 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:02.191 DEBUG:boto:Port: 8000 2024-03-04T17:40:02.191 DEBUG:boto:Params: {} 2024-03-04T17:40:02.191 DEBUG:boto:Token: None 2024-03-04T17:40:02.191 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:02 GMT /ppfahn-13/ 2024-03-04T17:40:02.191 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ipu+p6265KcF+TbbmjdnysfGjEQ= 2024-03-04T17:40:02.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:02 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ipu+p6265KcF+TbbmjdnysfGjEQ=', 'Content-Length': '0'} 2024-03-04T17:40:02.195 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000187458f59b27b232-0065e60772-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:02 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:02.195 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:40:02.195 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:40:02.195 DEBUG:boto:Method: HEAD 2024-03-04T17:40:02.195 DEBUG:boto:Path: /ppfahn-13/ 2024-03-04T17:40:02.196 DEBUG:boto:Data: 2024-03-04T17:40:02.196 DEBUG:boto:Headers: {} 2024-03-04T17:40:02.196 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:02.196 DEBUG:boto:Port: 8000 2024-03-04T17:40:02.196 DEBUG:boto:Params: {} 2024-03-04T17:40:02.196 DEBUG:boto:Token: None 2024-03-04T17:40:02.196 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:02 GMT /ppfahn-13/ 2024-03-04T17:40:02.196 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ipu+p6265KcF+TbbmjdnysfGjEQ= 2024-03-04T17:40:02.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:02 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ipu+p6265KcF+TbbmjdnysfGjEQ=', 'Content-Length': '0'} 2024-03-04T17:40:02.200 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021effaca9cb4c424-0065e60772-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:02 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:02.200 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:40:02.201 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:40:02.201 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:40:02.201 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:02.201 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:02.201 DEBUG:boto:Method: GET 2024-03-04T17:40:02.201 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:02.201 DEBUG:boto:Data: 2024-03-04T17:40:02.201 DEBUG:boto:Headers: {} 2024-03-04T17:40:02.201 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:02.201 DEBUG:boto:Port: 8000 2024-03-04T17:40:02.202 DEBUG:boto:Params: {} 2024-03-04T17:40:02.202 DEBUG:boto:Token: None 2024-03-04T17:40:02.202 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:02 GMT /ppfahn-13/?versions 2024-03-04T17:40:02.202 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pU5CsH/xiFprsS88nJvOHdB/dtI= 2024-03-04T17:40:02.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:02 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pU5CsH/xiFprsS88nJvOHdB/dtI=', 'Content-Length': '0'} 2024-03-04T17:40:02.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d0e08be54a3f33-0065e60772-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:02 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:02.206 DEBUG:boto:b'ppfahn-13999false' 2024-03-04T17:40:02.206 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:40:02.206 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:40:02.206 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:40:02.207 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:02.207 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:02.207 DEBUG:boto:Method: GET 2024-03-04T17:40:02.207 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:02.207 DEBUG:boto:Data: 2024-03-04T17:40:02.207 DEBUG:boto:Headers: {} 2024-03-04T17:40:02.207 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:02.207 DEBUG:boto:Port: 8000 2024-03-04T17:40:02.207 DEBUG:boto:Params: {} 2024-03-04T17:40:02.208 DEBUG:boto:Token: None 2024-03-04T17:40:02.208 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:02 GMT /ppfahn-13/?versions 2024-03-04T17:40:02.208 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pU5CsH/xiFprsS88nJvOHdB/dtI= 2024-03-04T17:40:02.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:02 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pU5CsH/xiFprsS88nJvOHdB/dtI=', 'Content-Length': '0'} 2024-03-04T17:40:02.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a583d6ce4f0a3882-0065e60772-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:02 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:02.212 DEBUG:boto:b'ppfahn-13999false' 2024-03-04T17:40:02.213 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:40:02.213 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:40:02.213 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:02.213 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:02.213 DEBUG:boto:Method: GET 2024-03-04T17:40:02.213 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:02.213 DEBUG:boto:Data: 2024-03-04T17:40:02.213 DEBUG:boto:Headers: {} 2024-03-04T17:40:02.214 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:02.214 DEBUG:boto:Port: 8000 2024-03-04T17:40:02.214 DEBUG:boto:Params: {} 2024-03-04T17:40:02.214 DEBUG:boto:Token: None 2024-03-04T17:40:02.214 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:02 GMT /ppfahn-13/?versions 2024-03-04T17:40:02.214 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pU5CsH/xiFprsS88nJvOHdB/dtI= 2024-03-04T17:40:02.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:02 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pU5CsH/xiFprsS88nJvOHdB/dtI=', 'Content-Length': '0'} 2024-03-04T17:40:02.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cd2e3e5df964a5a-0065e60772-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:02 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:02.218 DEBUG:boto:b'ppfahn-13999false' 2024-03-04T17:40:02.218 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:40:02.218 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:40:02.218 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:02.218 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:02.218 DEBUG:boto:Method: GET 2024-03-04T17:40:02.218 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:02.218 DEBUG:boto:Data: 2024-03-04T17:40:02.218 DEBUG:boto:Headers: {} 2024-03-04T17:40:02.219 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:02.219 DEBUG:boto:Port: 8000 2024-03-04T17:40:02.219 DEBUG:boto:Params: {} 2024-03-04T17:40:02.219 DEBUG:boto:Token: None 2024-03-04T17:40:02.219 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:02 GMT /ppfahn-13/?versions 2024-03-04T17:40:02.219 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pU5CsH/xiFprsS88nJvOHdB/dtI= 2024-03-04T17:40:02.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:02 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pU5CsH/xiFprsS88nJvOHdB/dtI=', 'Content-Length': '0'} 2024-03-04T17:40:02.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a3334d826da27bc-0065e60772-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:02 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:02.223 DEBUG:boto:b'ppfahn-13999false' 2024-03-04T17:40:02.223 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-13 zones={test-zone2, test-zone1} 2024-03-04T17:40:02.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:40:02.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:40:02.224 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:40:02.277 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:02.277 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:02.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:02.339+0000 7f9f1fdd4980 1 waiting for incremental sync to catch up: 2024-03-04T17:40:02.341 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-03-04T17:40:02.341 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-03-04T17:40:07.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:07.343+0000 7f9f1fdd4980 1 bucket sync caught up with source: 2024-03-04T17:40:07.344 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000002.15.6, , , , , , , , , ] 2024-03-04T17:40:07.344 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-03-04T17:40:07.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:07.343+0000 7f9f1fdd4980 0 bucket checkpoint complete 2024-03-04T17:40:07.379 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-13 zones={test-zone3, test-zone1} 2024-03-04T17:40:07.380 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:40:07.380 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:40:07.380 DEBUG:boto:Method: HEAD 2024-03-04T17:40:07.380 DEBUG:boto:Path: /ppfahn-13/ 2024-03-04T17:40:07.380 DEBUG:boto:Data: 2024-03-04T17:40:07.380 DEBUG:boto:Headers: {} 2024-03-04T17:40:07.380 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:07.380 DEBUG:boto:Port: 8001 2024-03-04T17:40:07.380 DEBUG:boto:Params: {} 2024-03-04T17:40:07.381 DEBUG:boto:Token: None 2024-03-04T17:40:07.381 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:07 GMT /ppfahn-13/ 2024-03-04T17:40:07.381 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zCb8DWffU28uOb3qeukpsFHkogk= 2024-03-04T17:40:07.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:07 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zCb8DWffU28uOb3qeukpsFHkogk=', 'Content-Length': '0'} 2024-03-04T17:40:07.384 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000141949f7c0299ff4-0065e60777-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:07.385 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:40:07.385 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:40:07.385 DEBUG:boto:Method: HEAD 2024-03-04T17:40:07.385 DEBUG:boto:Path: /ppfahn-13/ 2024-03-04T17:40:07.385 DEBUG:boto:Data: 2024-03-04T17:40:07.385 DEBUG:boto:Headers: {} 2024-03-04T17:40:07.385 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:07.385 DEBUG:boto:Port: 8000 2024-03-04T17:40:07.385 DEBUG:boto:Params: {} 2024-03-04T17:40:07.385 DEBUG:boto:Token: None 2024-03-04T17:40:07.386 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:07 GMT /ppfahn-13/ 2024-03-04T17:40:07.386 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zCb8DWffU28uOb3qeukpsFHkogk= 2024-03-04T17:40:07.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:07 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zCb8DWffU28uOb3qeukpsFHkogk=', 'Content-Length': '0'} 2024-03-04T17:40:07.388 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6677f162ff6084b-0065e60777-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:07.389 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:40:07.389 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:40:07.389 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:40:07.389 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:07.389 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:07.389 DEBUG:boto:Method: GET 2024-03-04T17:40:07.389 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:07.389 DEBUG:boto:Data: 2024-03-04T17:40:07.389 DEBUG:boto:Headers: {} 2024-03-04T17:40:07.389 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:07.389 DEBUG:boto:Port: 8001 2024-03-04T17:40:07.390 DEBUG:boto:Params: {} 2024-03-04T17:40:07.390 DEBUG:boto:Token: None 2024-03-04T17:40:07.390 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:07 GMT /ppfahn-13/?versions 2024-03-04T17:40:07.390 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hNPzhV0+gJ/TpIjxICJ0mDfGDrI= 2024-03-04T17:40:07.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:07 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hNPzhV0+gJ/TpIjxICJ0mDfGDrI=', 'Content-Length': '0'} 2024-03-04T17:40:07.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f74461df77ffd5c-0065e60777-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:07.436 DEBUG:boto:b'ppfahn-13999false' 2024-03-04T17:40:07.436 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:40:07.436 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:40:07.437 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:40:07.437 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:07.437 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:07.437 DEBUG:boto:Method: GET 2024-03-04T17:40:07.437 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:07.437 DEBUG:boto:Data: 2024-03-04T17:40:07.437 DEBUG:boto:Headers: {} 2024-03-04T17:40:07.437 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:07.437 DEBUG:boto:Port: 8000 2024-03-04T17:40:07.437 DEBUG:boto:Params: {} 2024-03-04T17:40:07.438 DEBUG:boto:Token: None 2024-03-04T17:40:07.438 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:07 GMT /ppfahn-13/?versions 2024-03-04T17:40:07.438 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hNPzhV0+gJ/TpIjxICJ0mDfGDrI= 2024-03-04T17:40:07.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:07 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hNPzhV0+gJ/TpIjxICJ0mDfGDrI=', 'Content-Length': '0'} 2024-03-04T17:40:07.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c6f56dd71a335d0-0065e60777-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:07.441 DEBUG:boto:b'ppfahn-13999false' 2024-03-04T17:40:07.442 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:40:07.442 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:40:07.442 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:07.442 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:07.442 DEBUG:boto:Method: GET 2024-03-04T17:40:07.442 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:07.442 DEBUG:boto:Data: 2024-03-04T17:40:07.442 DEBUG:boto:Headers: {} 2024-03-04T17:40:07.442 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:07.442 DEBUG:boto:Port: 8001 2024-03-04T17:40:07.442 DEBUG:boto:Params: {} 2024-03-04T17:40:07.443 DEBUG:boto:Token: None 2024-03-04T17:40:07.443 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:07 GMT /ppfahn-13/?versions 2024-03-04T17:40:07.443 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hNPzhV0+gJ/TpIjxICJ0mDfGDrI= 2024-03-04T17:40:07.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:07 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hNPzhV0+gJ/TpIjxICJ0mDfGDrI=', 'Content-Length': '0'} 2024-03-04T17:40:07.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015ffddff4a904292-0065e60777-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:07.446 DEBUG:boto:b'ppfahn-13999false' 2024-03-04T17:40:07.446 DEBUG:boto:path=/ppfahn-13/ 2024-03-04T17:40:07.446 DEBUG:boto:auth_path=/ppfahn-13/ 2024-03-04T17:40:07.446 DEBUG:boto:path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:07.446 DEBUG:boto:auth_path=/ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:07.446 DEBUG:boto:Method: GET 2024-03-04T17:40:07.447 DEBUG:boto:Path: /ppfahn-13/?versions&max-keys=999 2024-03-04T17:40:07.447 DEBUG:boto:Data: 2024-03-04T17:40:07.447 DEBUG:boto:Headers: {} 2024-03-04T17:40:07.447 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:07.447 DEBUG:boto:Port: 8000 2024-03-04T17:40:07.447 DEBUG:boto:Params: {} 2024-03-04T17:40:07.447 DEBUG:boto:Token: None 2024-03-04T17:40:07.447 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:07 GMT /ppfahn-13/?versions 2024-03-04T17:40:07.447 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hNPzhV0+gJ/TpIjxICJ0mDfGDrI= 2024-03-04T17:40:07.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:07 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hNPzhV0+gJ/TpIjxICJ0mDfGDrI=', 'Content-Length': '0'} 2024-03-04T17:40:07.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5dcda389585cc28-0065e60777-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:07.451 DEBUG:boto:b'ppfahn-13999false' 2024-03-04T17:40:07.451 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-13 zones={test-zone3, test-zone1} 2024-03-04T17:40:07.451 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:40:07.451 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:40:07.451 DEBUG:boto:Method: HEAD 2024-03-04T17:40:07.452 DEBUG:boto:Path: /ppfahn-14/ 2024-03-04T17:40:07.452 DEBUG:boto:Data: 2024-03-04T17:40:07.452 DEBUG:boto:Headers: {} 2024-03-04T17:40:07.452 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:07.452 DEBUG:boto:Port: 8000 2024-03-04T17:40:07.452 DEBUG:boto:Params: {} 2024-03-04T17:40:07.452 DEBUG:boto:Token: None 2024-03-04T17:40:07.452 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:07 GMT /ppfahn-14/ 2024-03-04T17:40:07.452 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F5eRyVRIZVuNAuNX8wOd9UM8DoE= 2024-03-04T17:40:07.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:07 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F5eRyVRIZVuNAuNX8wOd9UM8DoE=', 'Content-Length': '0'} 2024-03-04T17:40:07.456 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab129d3579d68ea1-0065e60777-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:07.457 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:40:07.457 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:40:07.457 DEBUG:boto:Method: HEAD 2024-03-04T17:40:07.457 DEBUG:boto:Path: /ppfahn-14/myobj 2024-03-04T17:40:07.457 DEBUG:boto:Data: 2024-03-04T17:40:07.457 DEBUG:boto:Headers: {} 2024-03-04T17:40:07.457 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:07.457 DEBUG:boto:Port: 8000 2024-03-04T17:40:07.457 DEBUG:boto:Params: {} 2024-03-04T17:40:07.457 DEBUG:boto:Token: None 2024-03-04T17:40:07.458 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:07 GMT /ppfahn-14/myobj 2024-03-04T17:40:07.458 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PA4CtD40WbQuZVn64ik+cGbgZqw= 2024-03-04T17:40:07.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:07 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PA4CtD40WbQuZVn64ik+cGbgZqw=', 'Content-Length': '0'} 2024-03-04T17:40:07.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000357cd129551d6ca8-0065e60777-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:07.461 DEBUG:boto:path=/ppfahn-14/myobj 2024-03-04T17:40:07.461 DEBUG:boto:auth_path=/ppfahn-14/myobj 2024-03-04T17:40:07.461 DEBUG:boto:Method: DELETE 2024-03-04T17:40:07.461 DEBUG:boto:Path: /ppfahn-14/myobj 2024-03-04T17:40:07.461 DEBUG:boto:Data: 2024-03-04T17:40:07.462 DEBUG:boto:Headers: {} 2024-03-04T17:40:07.462 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:07.462 DEBUG:boto:Port: 8000 2024-03-04T17:40:07.462 DEBUG:boto:Params: {} 2024-03-04T17:40:07.462 DEBUG:boto:Token: None 2024-03-04T17:40:07.462 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:40:07 GMT /ppfahn-14/myobj 2024-03-04T17:40:07.462 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/ja/QMT/Ix9MhUCJQav5Yfj65ws= 2024-03-04T17:40:07.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:07 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/ja/QMT/Ix9MhUCJQav5Yfj65ws=', 'Content-Length': '0'} 2024-03-04T17:40:07.474 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000816c8d1c75986e63-0065e60777-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:07.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:40:07.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:40:07.475 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:40:07.524 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:07.524 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:07.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:07.587+0000 7f0528161980 1 waiting for incremental sync to catch up: 2024-03-04T17:40:07.587 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:40:07.587 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2024-03-04T17:40:12.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:12.587+0000 7f0528161980 1 waiting for incremental sync to catch up: 2024-03-04T17:40:12.590 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:40:12.590 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2024-03-04T17:40:17.592 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:17.591+0000 7f0528161980 1 waiting for incremental sync to catch up: 2024-03-04T17:40:17.592 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:40:17.592 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2024-03-04T17:40:22.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:22.591+0000 7f0528161980 1 bucket sync caught up with source: 2024-03-04T17:40:22.594 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000002.10.6, , , , , , , , , ] 2024-03-04T17:40:22.594 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2024-03-04T17:40:22.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:22.591+0000 7f0528161980 0 bucket checkpoint complete 2024-03-04T17:40:22.638 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-14 zones={test-zone1, test-zone2} 2024-03-04T17:40:22.639 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:40:22.639 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:40:22.639 DEBUG:boto:Method: HEAD 2024-03-04T17:40:22.639 DEBUG:boto:Path: /ppfahn-14/ 2024-03-04T17:40:22.639 DEBUG:boto:Data: 2024-03-04T17:40:22.639 DEBUG:boto:Headers: {} 2024-03-04T17:40:22.639 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:22.639 DEBUG:boto:Port: 8000 2024-03-04T17:40:22.639 DEBUG:boto:Params: {} 2024-03-04T17:40:22.640 DEBUG:boto:Token: None 2024-03-04T17:40:22.640 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:22 GMT /ppfahn-14/ 2024-03-04T17:40:22.640 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Ftcvpfcn/+zbqnazhYqFPJ45JG0= 2024-03-04T17:40:22.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Ftcvpfcn/+zbqnazhYqFPJ45JG0=', 'Content-Length': '0'} 2024-03-04T17:40:22.643 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e7c7f33a515d8f5-0065e60786-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:22.644 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:40:22.644 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:40:22.644 DEBUG:boto:Method: HEAD 2024-03-04T17:40:22.644 DEBUG:boto:Path: /ppfahn-14/ 2024-03-04T17:40:22.644 DEBUG:boto:Data: 2024-03-04T17:40:22.644 DEBUG:boto:Headers: {} 2024-03-04T17:40:22.644 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:22.644 DEBUG:boto:Port: 8000 2024-03-04T17:40:22.644 DEBUG:boto:Params: {} 2024-03-04T17:40:22.644 DEBUG:boto:Token: None 2024-03-04T17:40:22.645 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:22 GMT /ppfahn-14/ 2024-03-04T17:40:22.645 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Ftcvpfcn/+zbqnazhYqFPJ45JG0= 2024-03-04T17:40:22.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Ftcvpfcn/+zbqnazhYqFPJ45JG0=', 'Content-Length': '0'} 2024-03-04T17:40:22.649 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfc25af2e2bb7520-0065e60786-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:22.649 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:40:22.649 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:40:22.649 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:40:22.650 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:22.650 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:22.650 DEBUG:boto:Method: GET 2024-03-04T17:40:22.650 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:22.650 DEBUG:boto:Data: 2024-03-04T17:40:22.650 DEBUG:boto:Headers: {} 2024-03-04T17:40:22.650 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:22.650 DEBUG:boto:Port: 8000 2024-03-04T17:40:22.650 DEBUG:boto:Params: {} 2024-03-04T17:40:22.650 DEBUG:boto:Token: None 2024-03-04T17:40:22.650 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:22 GMT /ppfahn-14/?versions 2024-03-04T17:40:22.651 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s4TET6CsV8nJ5rJ48/FGsC6PFss= 2024-03-04T17:40:22.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s4TET6CsV8nJ5rJ48/FGsC6PFss=', 'Content-Length': '0'} 2024-03-04T17:40:22.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1938eae96adf166-0065e60786-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:22.654 DEBUG:boto:b'ppfahn-14999false' 2024-03-04T17:40:22.654 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:40:22.654 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:40:22.654 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:40:22.655 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:22.655 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:22.655 DEBUG:boto:Method: GET 2024-03-04T17:40:22.655 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:22.655 DEBUG:boto:Data: 2024-03-04T17:40:22.655 DEBUG:boto:Headers: {} 2024-03-04T17:40:22.655 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:22.655 DEBUG:boto:Port: 8000 2024-03-04T17:40:22.655 DEBUG:boto:Params: {} 2024-03-04T17:40:22.655 DEBUG:boto:Token: None 2024-03-04T17:40:22.656 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:22 GMT /ppfahn-14/?versions 2024-03-04T17:40:22.656 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s4TET6CsV8nJ5rJ48/FGsC6PFss= 2024-03-04T17:40:22.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s4TET6CsV8nJ5rJ48/FGsC6PFss=', 'Content-Length': '0'} 2024-03-04T17:40:22.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006671243e0241e53e-0065e60786-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:22.660 DEBUG:boto:b'ppfahn-14999false' 2024-03-04T17:40:22.661 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:40:22.661 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:40:22.661 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:22.661 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:22.661 DEBUG:boto:Method: GET 2024-03-04T17:40:22.661 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:22.661 DEBUG:boto:Data: 2024-03-04T17:40:22.661 DEBUG:boto:Headers: {} 2024-03-04T17:40:22.661 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:22.661 DEBUG:boto:Port: 8000 2024-03-04T17:40:22.661 DEBUG:boto:Params: {} 2024-03-04T17:40:22.662 DEBUG:boto:Token: None 2024-03-04T17:40:22.662 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:22 GMT /ppfahn-14/?versions 2024-03-04T17:40:22.662 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s4TET6CsV8nJ5rJ48/FGsC6PFss= 2024-03-04T17:40:22.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s4TET6CsV8nJ5rJ48/FGsC6PFss=', 'Content-Length': '0'} 2024-03-04T17:40:22.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4c5daaae01a3194-0065e60786-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:22.708 DEBUG:boto:b'ppfahn-14999false' 2024-03-04T17:40:22.708 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:40:22.708 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:40:22.708 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:22.709 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:22.709 DEBUG:boto:Method: GET 2024-03-04T17:40:22.709 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:22.709 DEBUG:boto:Data: 2024-03-04T17:40:22.709 DEBUG:boto:Headers: {} 2024-03-04T17:40:22.709 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:22.709 DEBUG:boto:Port: 8000 2024-03-04T17:40:22.709 DEBUG:boto:Params: {} 2024-03-04T17:40:22.709 DEBUG:boto:Token: None 2024-03-04T17:40:22.709 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:22 GMT /ppfahn-14/?versions 2024-03-04T17:40:22.709 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s4TET6CsV8nJ5rJ48/FGsC6PFss= 2024-03-04T17:40:22.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s4TET6CsV8nJ5rJ48/FGsC6PFss=', 'Content-Length': '0'} 2024-03-04T17:40:22.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ca86456f405e531-0065e60786-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:22.714 DEBUG:boto:b'ppfahn-14999false' 2024-03-04T17:40:22.714 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-14 zones={test-zone1, test-zone2} 2024-03-04T17:40:22.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:40:22.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:40:22.715 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:40:22.766 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:22.766 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:22.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:22.827+0000 7fad62e60980 1 waiting for incremental sync to catch up: 2024-03-04T17:40:22.828 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:40:22.828 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2024-03-04T17:40:27.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:27.827+0000 7fad62e60980 1 bucket sync caught up with source: 2024-03-04T17:40:27.830 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000002.10.6, , , , , , , , , ] 2024-03-04T17:40:27.830 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2024-03-04T17:40:27.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:27.827+0000 7fad62e60980 0 bucket checkpoint complete 2024-03-04T17:40:27.866 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-14 zones={test-zone3, test-zone2} 2024-03-04T17:40:27.866 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:40:27.866 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:40:27.866 DEBUG:boto:Method: HEAD 2024-03-04T17:40:27.866 DEBUG:boto:Path: /ppfahn-14/ 2024-03-04T17:40:27.867 DEBUG:boto:Data: 2024-03-04T17:40:27.867 DEBUG:boto:Headers: {} 2024-03-04T17:40:27.867 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:27.867 DEBUG:boto:Port: 8001 2024-03-04T17:40:27.867 DEBUG:boto:Params: {} 2024-03-04T17:40:27.867 DEBUG:boto:Token: None 2024-03-04T17:40:27.867 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:27 GMT /ppfahn-14/ 2024-03-04T17:40:27.867 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mronkXTxxAKy7vK5p27DY6FDC3M= 2024-03-04T17:40:27.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mronkXTxxAKy7vK5p27DY6FDC3M=', 'Content-Length': '0'} 2024-03-04T17:40:27.871 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da036826c34b8063-0065e6078b-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:27.871 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:40:27.871 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:40:27.871 DEBUG:boto:Method: HEAD 2024-03-04T17:40:27.871 DEBUG:boto:Path: /ppfahn-14/ 2024-03-04T17:40:27.871 DEBUG:boto:Data: 2024-03-04T17:40:27.871 DEBUG:boto:Headers: {} 2024-03-04T17:40:27.871 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:27.871 DEBUG:boto:Port: 8000 2024-03-04T17:40:27.871 DEBUG:boto:Params: {} 2024-03-04T17:40:27.872 DEBUG:boto:Token: None 2024-03-04T17:40:27.872 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:27 GMT /ppfahn-14/ 2024-03-04T17:40:27.872 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mronkXTxxAKy7vK5p27DY6FDC3M= 2024-03-04T17:40:27.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mronkXTxxAKy7vK5p27DY6FDC3M=', 'Content-Length': '0'} 2024-03-04T17:40:27.877 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072cc7e3d53b7b446-0065e6078b-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:27.877 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:40:27.877 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:40:27.877 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:40:27.877 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:27.878 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:27.878 DEBUG:boto:Method: GET 2024-03-04T17:40:27.878 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:27.878 DEBUG:boto:Data: 2024-03-04T17:40:27.878 DEBUG:boto:Headers: {} 2024-03-04T17:40:27.878 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:27.878 DEBUG:boto:Port: 8001 2024-03-04T17:40:27.878 DEBUG:boto:Params: {} 2024-03-04T17:40:27.878 DEBUG:boto:Token: None 2024-03-04T17:40:27.878 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:27 GMT /ppfahn-14/?versions 2024-03-04T17:40:27.878 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/slqrj9QD68AxODFrDfRrhRL9LA= 2024-03-04T17:40:27.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/slqrj9QD68AxODFrDfRrhRL9LA=', 'Content-Length': '0'} 2024-03-04T17:40:27.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caf1a4add121c34e-0065e6078b-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:27.882 DEBUG:boto:b'ppfahn-14999false' 2024-03-04T17:40:27.882 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:40:27.882 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:40:27.882 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:40:27.882 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:27.883 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:27.883 DEBUG:boto:Method: GET 2024-03-04T17:40:27.883 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:27.883 DEBUG:boto:Data: 2024-03-04T17:40:27.883 DEBUG:boto:Headers: {} 2024-03-04T17:40:27.883 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:27.883 DEBUG:boto:Port: 8000 2024-03-04T17:40:27.883 DEBUG:boto:Params: {} 2024-03-04T17:40:27.883 DEBUG:boto:Token: None 2024-03-04T17:40:27.883 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:27 GMT /ppfahn-14/?versions 2024-03-04T17:40:27.883 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/slqrj9QD68AxODFrDfRrhRL9LA= 2024-03-04T17:40:27.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/slqrj9QD68AxODFrDfRrhRL9LA=', 'Content-Length': '0'} 2024-03-04T17:40:27.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b486896f873a8f17-0065e6078b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:27.887 DEBUG:boto:b'ppfahn-14999false' 2024-03-04T17:40:27.888 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:40:27.888 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:40:27.888 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:27.888 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:27.888 DEBUG:boto:Method: GET 2024-03-04T17:40:27.888 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:27.888 DEBUG:boto:Data: 2024-03-04T17:40:27.888 DEBUG:boto:Headers: {} 2024-03-04T17:40:27.888 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:27.889 DEBUG:boto:Port: 8001 2024-03-04T17:40:27.889 DEBUG:boto:Params: {} 2024-03-04T17:40:27.889 DEBUG:boto:Token: None 2024-03-04T17:40:27.889 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:27 GMT /ppfahn-14/?versions 2024-03-04T17:40:27.889 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/slqrj9QD68AxODFrDfRrhRL9LA= 2024-03-04T17:40:27.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/slqrj9QD68AxODFrDfRrhRL9LA=', 'Content-Length': '0'} 2024-03-04T17:40:27.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1fb792fc1b7adb1-0065e6078b-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:27.892 DEBUG:boto:b'ppfahn-14999false' 2024-03-04T17:40:27.893 DEBUG:boto:path=/ppfahn-14/ 2024-03-04T17:40:27.893 DEBUG:boto:auth_path=/ppfahn-14/ 2024-03-04T17:40:27.893 DEBUG:boto:path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:27.893 DEBUG:boto:auth_path=/ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:27.893 DEBUG:boto:Method: GET 2024-03-04T17:40:27.893 DEBUG:boto:Path: /ppfahn-14/?versions&max-keys=999 2024-03-04T17:40:27.893 DEBUG:boto:Data: 2024-03-04T17:40:27.893 DEBUG:boto:Headers: {} 2024-03-04T17:40:27.893 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:27.893 DEBUG:boto:Port: 8000 2024-03-04T17:40:27.894 DEBUG:boto:Params: {} 2024-03-04T17:40:27.894 DEBUG:boto:Token: None 2024-03-04T17:40:27.894 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:27 GMT /ppfahn-14/?versions 2024-03-04T17:40:27.894 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/slqrj9QD68AxODFrDfRrhRL9LA= 2024-03-04T17:40:27.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/slqrj9QD68AxODFrDfRrhRL9LA=', 'Content-Length': '0'} 2024-03-04T17:40:27.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066eaf7bc7cb2c3cc-0065e6078b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:27.898 DEBUG:boto:b'ppfahn-14999false' 2024-03-04T17:40:27.899 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-14 zones={test-zone3, test-zone2} 2024-03-04T17:40:27.899 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:40:27.899 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:40:27.899 DEBUG:boto:Method: HEAD 2024-03-04T17:40:27.899 DEBUG:boto:Path: /ppfahn-15/ 2024-03-04T17:40:27.899 DEBUG:boto:Data: 2024-03-04T17:40:27.899 DEBUG:boto:Headers: {} 2024-03-04T17:40:27.899 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:27.899 DEBUG:boto:Port: 8001 2024-03-04T17:40:27.899 DEBUG:boto:Params: {} 2024-03-04T17:40:27.900 DEBUG:boto:Token: None 2024-03-04T17:40:27.900 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:27 GMT /ppfahn-15/ 2024-03-04T17:40:27.900 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6MXpm0KeBN0NuC+svGI0g8Wfc7w= 2024-03-04T17:40:27.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6MXpm0KeBN0NuC+svGI0g8Wfc7w=', 'Content-Length': '0'} 2024-03-04T17:40:27.904 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ffdb4dff2e53acd-0065e6078b-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:27.904 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:40:27.904 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:40:27.904 DEBUG:boto:Method: HEAD 2024-03-04T17:40:27.904 DEBUG:boto:Path: /ppfahn-15/myobj 2024-03-04T17:40:27.904 DEBUG:boto:Data: 2024-03-04T17:40:27.905 DEBUG:boto:Headers: {} 2024-03-04T17:40:27.905 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:27.905 DEBUG:boto:Port: 8001 2024-03-04T17:40:27.905 DEBUG:boto:Params: {} 2024-03-04T17:40:27.905 DEBUG:boto:Token: None 2024-03-04T17:40:27.905 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:27 GMT /ppfahn-15/myobj 2024-03-04T17:40:27.905 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:10XgJu/VaoRogorEobVyMB+fi38= 2024-03-04T17:40:27.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:10XgJu/VaoRogorEobVyMB+fi38=', 'Content-Length': '0'} 2024-03-04T17:40:27.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bac7b5d62fe3837-0065e6078b-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:27.908 DEBUG:boto:path=/ppfahn-15/myobj 2024-03-04T17:40:27.908 DEBUG:boto:auth_path=/ppfahn-15/myobj 2024-03-04T17:40:27.908 DEBUG:boto:Method: DELETE 2024-03-04T17:40:27.909 DEBUG:boto:Path: /ppfahn-15/myobj 2024-03-04T17:40:27.909 DEBUG:boto:Data: 2024-03-04T17:40:27.909 DEBUG:boto:Headers: {} 2024-03-04T17:40:27.909 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:27.909 DEBUG:boto:Port: 8001 2024-03-04T17:40:27.909 DEBUG:boto:Params: {} 2024-03-04T17:40:27.909 DEBUG:boto:Token: None 2024-03-04T17:40:27.909 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:40:27 GMT /ppfahn-15/myobj 2024-03-04T17:40:27.909 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Ccem/bafoh3EH4vN989NGgg6eXc= 2024-03-04T17:40:27.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Ccem/bafoh3EH4vN989NGgg6eXc=', 'Content-Length': '0'} 2024-03-04T17:40:27.921 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e193c9a354ea6ba2-0065e6078b-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:27.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:40:27.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:40:27.922 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:40:27.971 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:27.971 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:28.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:28.031+0000 7f37acffb980 1 waiting for incremental sync to catch up: 2024-03-04T17:40:28.031 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:40:28.031 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.19.6, , , , , , , , , ] 2024-03-04T17:40:33.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:33.031+0000 7f37acffb980 1 waiting for incremental sync to catch up: 2024-03-04T17:40:33.034 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:40:33.034 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.19.6, , , , , , , , , ] 2024-03-04T17:40:38.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:38.035+0000 7f37acffb980 1 waiting for incremental sync to catch up: 2024-03-04T17:40:38.036 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2024-03-04T17:40:38.036 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.19.6, , , , , , , , , ] 2024-03-04T17:40:43.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:43.039+0000 7f37acffb980 1 bucket sync caught up with source: 2024-03-04T17:40:43.039 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000002.19.6, , , , , , , , , ] 2024-03-04T17:40:43.039 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.19.6, , , , , , , , , ] 2024-03-04T17:40:43.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:43.039+0000 7f37acffb980 0 bucket checkpoint complete 2024-03-04T17:40:43.084 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-15 zones={test-zone1, test-zone3} 2024-03-04T17:40:43.085 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:40:43.085 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:40:43.085 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.085 DEBUG:boto:Path: /ppfahn-15/ 2024-03-04T17:40:43.085 DEBUG:boto:Data: 2024-03-04T17:40:43.085 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.085 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.085 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.085 DEBUG:boto:Params: {} 2024-03-04T17:40:43.085 DEBUG:boto:Token: None 2024-03-04T17:40:43.086 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-15/ 2024-03-04T17:40:43.086 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BHCrpPBCTxDbKyYCFu3fJF48CtA= 2024-03-04T17:40:43.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BHCrpPBCTxDbKyYCFu3fJF48CtA=', 'Content-Length': '0'} 2024-03-04T17:40:43.089 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfe596ff74a53661-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.090 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:40:43.090 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:40:43.090 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.090 DEBUG:boto:Path: /ppfahn-15/ 2024-03-04T17:40:43.090 DEBUG:boto:Data: 2024-03-04T17:40:43.090 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.090 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:43.090 DEBUG:boto:Port: 8001 2024-03-04T17:40:43.090 DEBUG:boto:Params: {} 2024-03-04T17:40:43.090 DEBUG:boto:Token: None 2024-03-04T17:40:43.091 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-15/ 2024-03-04T17:40:43.091 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BHCrpPBCTxDbKyYCFu3fJF48CtA= 2024-03-04T17:40:43.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BHCrpPBCTxDbKyYCFu3fJF48CtA=', 'Content-Length': '0'} 2024-03-04T17:40:43.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028645d8f08e3e510-0065e6079b-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.094 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:40:43.094 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:40:43.094 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:40:43.094 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.094 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.095 DEBUG:boto:Method: GET 2024-03-04T17:40:43.095 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.095 DEBUG:boto:Data: 2024-03-04T17:40:43.095 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.095 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.095 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.095 DEBUG:boto:Params: {} 2024-03-04T17:40:43.095 DEBUG:boto:Token: None 2024-03-04T17:40:43.095 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-15/?versions 2024-03-04T17:40:43.095 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc= 2024-03-04T17:40:43.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc=', 'Content-Length': '0'} 2024-03-04T17:40:43.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0650058ec630af9-0065e6079b-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.099 DEBUG:boto:b'ppfahn-15999false' 2024-03-04T17:40:43.100 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:40:43.100 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:40:43.100 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:40:43.100 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.100 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.100 DEBUG:boto:Method: GET 2024-03-04T17:40:43.100 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.100 DEBUG:boto:Data: 2024-03-04T17:40:43.100 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.100 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:43.100 DEBUG:boto:Port: 8001 2024-03-04T17:40:43.101 DEBUG:boto:Params: {} 2024-03-04T17:40:43.101 DEBUG:boto:Token: None 2024-03-04T17:40:43.101 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-15/?versions 2024-03-04T17:40:43.101 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc= 2024-03-04T17:40:43.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc=', 'Content-Length': '0'} 2024-03-04T17:40:43.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a60ae09b901736f4-0065e6079b-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.105 DEBUG:boto:b'ppfahn-15999false' 2024-03-04T17:40:43.105 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:40:43.106 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:40:43.106 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.106 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.106 DEBUG:boto:Method: GET 2024-03-04T17:40:43.106 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.106 DEBUG:boto:Data: 2024-03-04T17:40:43.106 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.106 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.106 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.106 DEBUG:boto:Params: {} 2024-03-04T17:40:43.106 DEBUG:boto:Token: None 2024-03-04T17:40:43.106 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-15/?versions 2024-03-04T17:40:43.107 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc= 2024-03-04T17:40:43.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc=', 'Content-Length': '0'} 2024-03-04T17:40:43.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fdc412cab06a967-0065e6079b-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.110 DEBUG:boto:b'ppfahn-15999false' 2024-03-04T17:40:43.110 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:40:43.111 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:40:43.111 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.111 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.111 DEBUG:boto:Method: GET 2024-03-04T17:40:43.111 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.111 DEBUG:boto:Data: 2024-03-04T17:40:43.111 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.111 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:43.111 DEBUG:boto:Port: 8001 2024-03-04T17:40:43.111 DEBUG:boto:Params: {} 2024-03-04T17:40:43.111 DEBUG:boto:Token: None 2024-03-04T17:40:43.112 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-15/?versions 2024-03-04T17:40:43.112 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc= 2024-03-04T17:40:43.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc=', 'Content-Length': '0'} 2024-03-04T17:40:43.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdb3d49edbcba2bc-0065e6079b-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.115 DEBUG:boto:b'ppfahn-15999false' 2024-03-04T17:40:43.115 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-15 zones={test-zone1, test-zone3} 2024-03-04T17:40:43.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:40:43.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:40:43.116 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:40:43.167 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:43.167 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:43.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:40:43.230+0000 7f786b216980 1 bucket sync caught up with source: 2024-03-04T17:40:43.230 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000002.19.6, , , , , , , , , ] 2024-03-04T17:40:43.230 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000002.19.6, , , , , , , , , ] 2024-03-04T17:40:43.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:40:43.230+0000 7f786b216980 0 bucket checkpoint complete 2024-03-04T17:40:43.251 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-15 zones={test-zone2, test-zone3} 2024-03-04T17:40:43.251 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:40:43.251 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:40:43.251 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.251 DEBUG:boto:Path: /ppfahn-15/ 2024-03-04T17:40:43.251 DEBUG:boto:Data: 2024-03-04T17:40:43.251 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.251 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:43.252 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.252 DEBUG:boto:Params: {} 2024-03-04T17:40:43.252 DEBUG:boto:Token: None 2024-03-04T17:40:43.252 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-15/ 2024-03-04T17:40:43.252 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BHCrpPBCTxDbKyYCFu3fJF48CtA= 2024-03-04T17:40:43.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BHCrpPBCTxDbKyYCFu3fJF48CtA=', 'Content-Length': '0'} 2024-03-04T17:40:43.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4b1e83e4547e41a-0065e6079b-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.254 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:40:43.255 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:40:43.255 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.255 DEBUG:boto:Path: /ppfahn-15/ 2024-03-04T17:40:43.255 DEBUG:boto:Data: 2024-03-04T17:40:43.255 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.255 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:43.255 DEBUG:boto:Port: 8001 2024-03-04T17:40:43.255 DEBUG:boto:Params: {} 2024-03-04T17:40:43.255 DEBUG:boto:Token: None 2024-03-04T17:40:43.255 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-15/ 2024-03-04T17:40:43.256 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BHCrpPBCTxDbKyYCFu3fJF48CtA= 2024-03-04T17:40:43.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BHCrpPBCTxDbKyYCFu3fJF48CtA=', 'Content-Length': '0'} 2024-03-04T17:40:43.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093d40cb0bdcca157-0065e6079b-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.259 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:40:43.259 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:40:43.260 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:40:43.260 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.260 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.260 DEBUG:boto:Method: GET 2024-03-04T17:40:43.260 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.260 DEBUG:boto:Data: 2024-03-04T17:40:43.260 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.260 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:43.260 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.260 DEBUG:boto:Params: {} 2024-03-04T17:40:43.260 DEBUG:boto:Token: None 2024-03-04T17:40:43.261 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-15/?versions 2024-03-04T17:40:43.261 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc= 2024-03-04T17:40:43.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc=', 'Content-Length': '0'} 2024-03-04T17:40:43.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7d57882406214eb-0065e6079b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.263 DEBUG:boto:b'ppfahn-15999false' 2024-03-04T17:40:43.263 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:40:43.263 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:40:43.263 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:40:43.263 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.264 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.264 DEBUG:boto:Method: GET 2024-03-04T17:40:43.264 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.264 DEBUG:boto:Data: 2024-03-04T17:40:43.264 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.264 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:43.264 DEBUG:boto:Port: 8001 2024-03-04T17:40:43.264 DEBUG:boto:Params: {} 2024-03-04T17:40:43.264 DEBUG:boto:Token: None 2024-03-04T17:40:43.264 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-15/?versions 2024-03-04T17:40:43.265 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc= 2024-03-04T17:40:43.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc=', 'Content-Length': '0'} 2024-03-04T17:40:43.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5e6a9fafdab4aea-0065e6079b-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.268 DEBUG:boto:b'ppfahn-15999false' 2024-03-04T17:40:43.269 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:40:43.269 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:40:43.269 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.269 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.269 DEBUG:boto:Method: GET 2024-03-04T17:40:43.269 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.269 DEBUG:boto:Data: 2024-03-04T17:40:43.269 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.269 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:43.270 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.270 DEBUG:boto:Params: {} 2024-03-04T17:40:43.270 DEBUG:boto:Token: None 2024-03-04T17:40:43.270 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-15/?versions 2024-03-04T17:40:43.270 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc= 2024-03-04T17:40:43.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc=', 'Content-Length': '0'} 2024-03-04T17:40:43.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008106722698490284-0065e6079b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.273 DEBUG:boto:b'ppfahn-15999false' 2024-03-04T17:40:43.273 DEBUG:boto:path=/ppfahn-15/ 2024-03-04T17:40:43.273 DEBUG:boto:auth_path=/ppfahn-15/ 2024-03-04T17:40:43.273 DEBUG:boto:path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.273 DEBUG:boto:auth_path=/ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.273 DEBUG:boto:Method: GET 2024-03-04T17:40:43.274 DEBUG:boto:Path: /ppfahn-15/?versions&max-keys=999 2024-03-04T17:40:43.274 DEBUG:boto:Data: 2024-03-04T17:40:43.274 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.274 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:43.274 DEBUG:boto:Port: 8001 2024-03-04T17:40:43.274 DEBUG:boto:Params: {} 2024-03-04T17:40:43.274 DEBUG:boto:Token: None 2024-03-04T17:40:43.274 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-15/?versions 2024-03-04T17:40:43.274 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc= 2024-03-04T17:40:43.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xQ0HpRZYHc3/EBEPLTSB53ZaxVc=', 'Content-Length': '0'} 2024-03-04T17:40:43.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074be49841ea18c25-0065e6079b-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.320 DEBUG:boto:b'ppfahn-15999false' 2024-03-04T17:40:43.320 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-15 zones={test-zone2, test-zone3} 2024-03-04T17:40:43.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2024-03-04T17:40:43.322 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-16 2024-03-04T17:40:43.322 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.322 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.322 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.322 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.322 DEBUG:boto:Data: 2024-03-04T17:40:43.322 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.322 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.322 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.323 DEBUG:boto:Params: {} 2024-03-04T17:40:43.323 DEBUG:boto:Token: None 2024-03-04T17:40:43.323 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.323 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SZAEeLRPDDyXLmuFHftQL/jfxdE= 2024-03-04T17:40:43.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SZAEeLRPDDyXLmuFHftQL/jfxdE=', 'Content-Length': '0'} 2024-03-04T17:40:43.356 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ba544c897f8b488f-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.356 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-17 2024-03-04T17:40:43.356 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:43.356 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:43.357 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.357 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:43.357 DEBUG:boto:Data: 2024-03-04T17:40:43.357 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.357 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:43.357 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.357 DEBUG:boto:Params: {} 2024-03-04T17:40:43.358 DEBUG:boto:Token: None 2024-03-04T17:40:43.358 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-17/ 2024-03-04T17:40:43.358 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:myYNfZl6maRqmkUu/nPs3WVZCGk= 2024-03-04T17:40:43.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:myYNfZl6maRqmkUu/nPs3WVZCGk=', 'Content-Length': '0'} 2024-03-04T17:40:43.412 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e33b23da7274a63a-0065e6079b-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.412 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-18 2024-03-04T17:40:43.412 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:43.412 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:43.412 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.412 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:43.412 DEBUG:boto:Data: 2024-03-04T17:40:43.412 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.412 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:43.412 DEBUG:boto:Port: 8001 2024-03-04T17:40:43.413 DEBUG:boto:Params: {} 2024-03-04T17:40:43.413 DEBUG:boto:Token: None 2024-03-04T17:40:43.413 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-18/ 2024-03-04T17:40:43.413 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QWzbcXMDCWkAz/SXVm00zzHTNXs= 2024-03-04T17:40:43.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QWzbcXMDCWkAz/SXVm00zzHTNXs=', 'Content-Length': '0'} 2024-03-04T17:40:43.462 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000065dfcc1df8260b69-0065e6079b-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.462 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.462 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.462 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.462 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.462 DEBUG:boto:Data: 2024-03-04T17:40:43.462 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.462 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.463 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.463 DEBUG:boto:Params: {} 2024-03-04T17:40:43.463 DEBUG:boto:Token: None 2024-03-04T17:40:43.463 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.463 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bfb44596a9d55c9-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.466 DEBUG:boto:path=/ppfahn-16/obj1 2024-03-04T17:40:43.466 DEBUG:boto:auth_path=/ppfahn-16/obj1 2024-03-04T17:40:43.466 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.466 DEBUG:boto:Path: /ppfahn-16/obj1 2024-03-04T17:40:43.466 DEBUG:boto:Data: 2024-03-04T17:40:43.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.467 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.467 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.467 DEBUG:boto:Params: {} 2024-03-04T17:40:43.467 DEBUG:boto:Token: None 2024-03-04T17:40:43.467 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj1 2024-03-04T17:40:43.467 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:x06qphqdzidadB88XUxpVA6Dvlo= 2024-03-04T17:40:43.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:x06qphqdzidadB88XUxpVA6Dvlo='} 2024-03-04T17:40:43.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e89f7614d7ac46f-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.476 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.476 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.476 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.476 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.476 DEBUG:boto:Data: 2024-03-04T17:40:43.477 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.477 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.477 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.477 DEBUG:boto:Params: {} 2024-03-04T17:40:43.477 DEBUG:boto:Token: None 2024-03-04T17:40:43.477 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.477 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014bb6c8666b6894b-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.480 DEBUG:boto:path=/ppfahn-16/obj2 2024-03-04T17:40:43.480 DEBUG:boto:auth_path=/ppfahn-16/obj2 2024-03-04T17:40:43.480 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.480 DEBUG:boto:Path: /ppfahn-16/obj2 2024-03-04T17:40:43.480 DEBUG:boto:Data: 2024-03-04T17:40:43.480 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.481 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.481 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.481 DEBUG:boto:Params: {} 2024-03-04T17:40:43.481 DEBUG:boto:Token: None 2024-03-04T17:40:43.481 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj2 2024-03-04T17:40:43.481 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lS3q+vs1zI0NXmZXY/UnICOIi7k= 2024-03-04T17:40:43.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lS3q+vs1zI0NXmZXY/UnICOIi7k='} 2024-03-04T17:40:43.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae60b74d65c464f5-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.524 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.524 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.525 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.525 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.525 DEBUG:boto:Data: 2024-03-04T17:40:43.525 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.525 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.525 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.525 DEBUG:boto:Params: {} 2024-03-04T17:40:43.525 DEBUG:boto:Token: None 2024-03-04T17:40:43.525 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.525 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c4e0b659e66e95b-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.532 DEBUG:boto:path=/ppfahn-16/obj3 2024-03-04T17:40:43.532 DEBUG:boto:auth_path=/ppfahn-16/obj3 2024-03-04T17:40:43.532 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.532 DEBUG:boto:Path: /ppfahn-16/obj3 2024-03-04T17:40:43.532 DEBUG:boto:Data: 2024-03-04T17:40:43.532 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.532 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.532 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.532 DEBUG:boto:Params: {} 2024-03-04T17:40:43.532 DEBUG:boto:Token: None 2024-03-04T17:40:43.533 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj3 2024-03-04T17:40:43.533 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CVWEtZAxF8hXlHA/gu06cp9VFlA= 2024-03-04T17:40:43.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CVWEtZAxF8hXlHA/gu06cp9VFlA='} 2024-03-04T17:40:43.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d858ee910bcc9751-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.544 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.544 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.544 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.544 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.544 DEBUG:boto:Data: 2024-03-04T17:40:43.544 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.545 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.545 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.545 DEBUG:boto:Params: {} 2024-03-04T17:40:43.545 DEBUG:boto:Token: None 2024-03-04T17:40:43.545 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.545 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.549 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a26b20e4c4cf2356-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.549 DEBUG:boto:path=/ppfahn-16/obj4 2024-03-04T17:40:43.549 DEBUG:boto:auth_path=/ppfahn-16/obj4 2024-03-04T17:40:43.549 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.549 DEBUG:boto:Path: /ppfahn-16/obj4 2024-03-04T17:40:43.549 DEBUG:boto:Data: 2024-03-04T17:40:43.550 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.550 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.550 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.550 DEBUG:boto:Params: {} 2024-03-04T17:40:43.550 DEBUG:boto:Token: None 2024-03-04T17:40:43.550 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj4 2024-03-04T17:40:43.550 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IdId9DvdMYKcGG0PR5cavmxARFE= 2024-03-04T17:40:43.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IdId9DvdMYKcGG0PR5cavmxARFE='} 2024-03-04T17:40:43.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030f6a38bdcfc9ffd-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.592 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.592 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.592 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.592 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.593 DEBUG:boto:Data: 2024-03-04T17:40:43.593 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.593 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.593 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.593 DEBUG:boto:Params: {} 2024-03-04T17:40:43.593 DEBUG:boto:Token: None 2024-03-04T17:40:43.593 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.593 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.599 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000045ec365c162b89-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.599 DEBUG:boto:path=/ppfahn-16/obj5 2024-03-04T17:40:43.600 DEBUG:boto:auth_path=/ppfahn-16/obj5 2024-03-04T17:40:43.600 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.600 DEBUG:boto:Path: /ppfahn-16/obj5 2024-03-04T17:40:43.600 DEBUG:boto:Data: 2024-03-04T17:40:43.600 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.600 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.600 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.600 DEBUG:boto:Params: {} 2024-03-04T17:40:43.600 DEBUG:boto:Token: None 2024-03-04T17:40:43.600 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj5 2024-03-04T17:40:43.601 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tPxPb8+eWk6NBgy6zFn3J3bzACw= 2024-03-04T17:40:43.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tPxPb8+eWk6NBgy6zFn3J3bzACw='} 2024-03-04T17:40:43.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000719e5f8562aaa9e0-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.616 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.616 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.617 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.617 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.617 DEBUG:boto:Data: 2024-03-04T17:40:43.617 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.617 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.617 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.617 DEBUG:boto:Params: {} 2024-03-04T17:40:43.617 DEBUG:boto:Token: None 2024-03-04T17:40:43.617 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.617 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.621 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fa05ce393e5a320-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.621 DEBUG:boto:path=/ppfahn-16/obj6 2024-03-04T17:40:43.622 DEBUG:boto:auth_path=/ppfahn-16/obj6 2024-03-04T17:40:43.622 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.622 DEBUG:boto:Path: /ppfahn-16/obj6 2024-03-04T17:40:43.622 DEBUG:boto:Data: 2024-03-04T17:40:43.622 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.622 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.622 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.622 DEBUG:boto:Params: {} 2024-03-04T17:40:43.622 DEBUG:boto:Token: None 2024-03-04T17:40:43.622 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj6 2024-03-04T17:40:43.623 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VADglL/9xO9RUMHvdJ6wn1kQrq8= 2024-03-04T17:40:43.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VADglL/9xO9RUMHvdJ6wn1kQrq8='} 2024-03-04T17:40:43.664 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e70c17c179aaa190-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.664 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.664 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.664 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.665 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.665 DEBUG:boto:Data: 2024-03-04T17:40:43.665 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.665 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.665 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.665 DEBUG:boto:Params: {} 2024-03-04T17:40:43.665 DEBUG:boto:Token: None 2024-03-04T17:40:43.665 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.665 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.669 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a807b8ed05df750d-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.669 DEBUG:boto:path=/ppfahn-16/obj7 2024-03-04T17:40:43.669 DEBUG:boto:auth_path=/ppfahn-16/obj7 2024-03-04T17:40:43.669 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.669 DEBUG:boto:Path: /ppfahn-16/obj7 2024-03-04T17:40:43.669 DEBUG:boto:Data: 2024-03-04T17:40:43.669 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.670 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.670 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.670 DEBUG:boto:Params: {} 2024-03-04T17:40:43.670 DEBUG:boto:Token: None 2024-03-04T17:40:43.670 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj7 2024-03-04T17:40:43.670 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LB/E52PbEB1lZwlCoe8lll24Ekg= 2024-03-04T17:40:43.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LB/E52PbEB1lZwlCoe8lll24Ekg='} 2024-03-04T17:40:43.676 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae8607658889d40b-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.676 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.676 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.676 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.676 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.676 DEBUG:boto:Data: 2024-03-04T17:40:43.677 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.677 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.677 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.677 DEBUG:boto:Params: {} 2024-03-04T17:40:43.677 DEBUG:boto:Token: None 2024-03-04T17:40:43.677 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.677 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.680 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097163db60e045f20-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.680 DEBUG:boto:path=/ppfahn-16/obj8 2024-03-04T17:40:43.680 DEBUG:boto:auth_path=/ppfahn-16/obj8 2024-03-04T17:40:43.680 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.681 DEBUG:boto:Path: /ppfahn-16/obj8 2024-03-04T17:40:43.681 DEBUG:boto:Data: 2024-03-04T17:40:43.681 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.681 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.681 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.681 DEBUG:boto:Params: {} 2024-03-04T17:40:43.681 DEBUG:boto:Token: None 2024-03-04T17:40:43.681 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj8 2024-03-04T17:40:43.681 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZA4xG/plcG6CKkhy6Q9rIQ0dRjs= 2024-03-04T17:40:43.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZA4xG/plcG6CKkhy6Q9rIQ0dRjs='} 2024-03-04T17:40:43.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000700f9374dcf73a02-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.724 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.724 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.725 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.725 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.725 DEBUG:boto:Data: 2024-03-04T17:40:43.725 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.725 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.725 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.725 DEBUG:boto:Params: {} 2024-03-04T17:40:43.725 DEBUG:boto:Token: None 2024-03-04T17:40:43.725 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.725 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.729 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd8181700cafd230-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.729 DEBUG:boto:path=/ppfahn-16/obj9 2024-03-04T17:40:43.729 DEBUG:boto:auth_path=/ppfahn-16/obj9 2024-03-04T17:40:43.729 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.729 DEBUG:boto:Path: /ppfahn-16/obj9 2024-03-04T17:40:43.729 DEBUG:boto:Data: 2024-03-04T17:40:43.729 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.730 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.730 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.730 DEBUG:boto:Params: {} 2024-03-04T17:40:43.730 DEBUG:boto:Token: None 2024-03-04T17:40:43.730 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj9 2024-03-04T17:40:43.730 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:B+W+7a1FThloZRIUDr70JxEOlYQ= 2024-03-04T17:40:43.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:B+W+7a1FThloZRIUDr70JxEOlYQ='} 2024-03-04T17:40:43.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef0daa0897f9474c-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.739 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.739 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.739 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.739 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.740 DEBUG:boto:Data: 2024-03-04T17:40:43.740 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.740 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.740 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.740 DEBUG:boto:Params: {} 2024-03-04T17:40:43.740 DEBUG:boto:Token: None 2024-03-04T17:40:43.740 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.740 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b06ce9473add85ad-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.744 DEBUG:boto:path=/ppfahn-16/obj10 2024-03-04T17:40:43.744 DEBUG:boto:auth_path=/ppfahn-16/obj10 2024-03-04T17:40:43.745 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.745 DEBUG:boto:Path: /ppfahn-16/obj10 2024-03-04T17:40:43.745 DEBUG:boto:Data: 2024-03-04T17:40:43.745 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.745 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.745 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.745 DEBUG:boto:Params: {} 2024-03-04T17:40:43.745 DEBUG:boto:Token: None 2024-03-04T17:40:43.745 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj10 2024-03-04T17:40:43.745 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4ox8PXztuTPbHZo/XqJIPlsZE4Y= 2024-03-04T17:40:43.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4ox8PXztuTPbHZo/XqJIPlsZE4Y='} 2024-03-04T17:40:43.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0ad8071bae3e74f-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.792 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.793 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.793 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.793 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.793 DEBUG:boto:Data: 2024-03-04T17:40:43.793 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.793 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.793 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.793 DEBUG:boto:Params: {} 2024-03-04T17:40:43.793 DEBUG:boto:Token: None 2024-03-04T17:40:43.793 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.794 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e36a61c02cc93ce7-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.798 DEBUG:boto:path=/ppfahn-16/obj11 2024-03-04T17:40:43.798 DEBUG:boto:auth_path=/ppfahn-16/obj11 2024-03-04T17:40:43.798 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.798 DEBUG:boto:Path: /ppfahn-16/obj11 2024-03-04T17:40:43.799 DEBUG:boto:Data: 2024-03-04T17:40:43.799 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.799 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.799 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.799 DEBUG:boto:Params: {} 2024-03-04T17:40:43.799 DEBUG:boto:Token: None 2024-03-04T17:40:43.799 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj11 2024-03-04T17:40:43.799 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7+t+WTj8YhJ+lwVisopKdCjdG54= 2024-03-04T17:40:43.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7+t+WTj8YhJ+lwVisopKdCjdG54='} 2024-03-04T17:40:43.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8e2cea3e15061d4-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.811 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.811 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.811 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.811 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.811 DEBUG:boto:Data: 2024-03-04T17:40:43.811 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.811 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.811 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.811 DEBUG:boto:Params: {} 2024-03-04T17:40:43.811 DEBUG:boto:Token: None 2024-03-04T17:40:43.812 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.812 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006669ffd4d88f6237-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.816 DEBUG:boto:path=/ppfahn-16/obj12 2024-03-04T17:40:43.816 DEBUG:boto:auth_path=/ppfahn-16/obj12 2024-03-04T17:40:43.816 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.816 DEBUG:boto:Path: /ppfahn-16/obj12 2024-03-04T17:40:43.816 DEBUG:boto:Data: 2024-03-04T17:40:43.816 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.816 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.816 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.816 DEBUG:boto:Params: {} 2024-03-04T17:40:43.817 DEBUG:boto:Token: None 2024-03-04T17:40:43.817 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj12 2024-03-04T17:40:43.817 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jJGBJO4H2HvpPd0i2cAte2sfaCI= 2024-03-04T17:40:43.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jJGBJO4H2HvpPd0i2cAte2sfaCI='} 2024-03-04T17:40:43.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b02bafae4789d5d6-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.860 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.860 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.860 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.860 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.861 DEBUG:boto:Data: 2024-03-04T17:40:43.861 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.861 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.861 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.861 DEBUG:boto:Params: {} 2024-03-04T17:40:43.861 DEBUG:boto:Token: None 2024-03-04T17:40:43.861 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.861 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9d9f506a166e962-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.866 DEBUG:boto:path=/ppfahn-16/obj13 2024-03-04T17:40:43.866 DEBUG:boto:auth_path=/ppfahn-16/obj13 2024-03-04T17:40:43.866 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.866 DEBUG:boto:Path: /ppfahn-16/obj13 2024-03-04T17:40:43.866 DEBUG:boto:Data: 2024-03-04T17:40:43.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.866 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.866 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.866 DEBUG:boto:Params: {} 2024-03-04T17:40:43.866 DEBUG:boto:Token: None 2024-03-04T17:40:43.866 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj13 2024-03-04T17:40:43.867 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nLEdPd6nuql3/4YG1dW+tXVwtEQ= 2024-03-04T17:40:43.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nLEdPd6nuql3/4YG1dW+tXVwtEQ='} 2024-03-04T17:40:43.875 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8eebd9480098b7d-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.875 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.875 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.875 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.875 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.875 DEBUG:boto:Data: 2024-03-04T17:40:43.875 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.875 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.875 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.875 DEBUG:boto:Params: {} 2024-03-04T17:40:43.876 DEBUG:boto:Token: None 2024-03-04T17:40:43.876 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.876 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0a83cc007ba817b-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.880 DEBUG:boto:path=/ppfahn-16/obj14 2024-03-04T17:40:43.880 DEBUG:boto:auth_path=/ppfahn-16/obj14 2024-03-04T17:40:43.881 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.881 DEBUG:boto:Path: /ppfahn-16/obj14 2024-03-04T17:40:43.881 DEBUG:boto:Data: 2024-03-04T17:40:43.881 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.881 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.881 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.881 DEBUG:boto:Params: {} 2024-03-04T17:40:43.881 DEBUG:boto:Token: None 2024-03-04T17:40:43.881 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj14 2024-03-04T17:40:43.881 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HM2LOwczJG6m0MHjd7RwCGkMZng= 2024-03-04T17:40:43.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HM2LOwczJG6m0MHjd7RwCGkMZng='} 2024-03-04T17:40:43.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030a20ca2fb4fc54c-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.924 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.925 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.925 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.925 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.925 DEBUG:boto:Data: 2024-03-04T17:40:43.925 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.925 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.925 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.925 DEBUG:boto:Params: {} 2024-03-04T17:40:43.925 DEBUG:boto:Token: None 2024-03-04T17:40:43.925 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.926 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.929 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000260c5b9b52737620-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.930 DEBUG:boto:path=/ppfahn-16/obj15 2024-03-04T17:40:43.930 DEBUG:boto:auth_path=/ppfahn-16/obj15 2024-03-04T17:40:43.930 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.930 DEBUG:boto:Path: /ppfahn-16/obj15 2024-03-04T17:40:43.930 DEBUG:boto:Data: 2024-03-04T17:40:43.930 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.930 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.930 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.930 DEBUG:boto:Params: {} 2024-03-04T17:40:43.930 DEBUG:boto:Token: None 2024-03-04T17:40:43.930 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj15 2024-03-04T17:40:43.931 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hVgmCZtO9/3iTYhN1p1PUDSsyIA= 2024-03-04T17:40:43.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hVgmCZtO9/3iTYhN1p1PUDSsyIA='} 2024-03-04T17:40:43.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005d04d2e5db520e7e-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.942 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.942 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.942 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.942 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.942 DEBUG:boto:Data: 2024-03-04T17:40:43.942 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.942 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.942 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.942 DEBUG:boto:Params: {} 2024-03-04T17:40:43.942 DEBUG:boto:Token: None 2024-03-04T17:40:43.943 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.943 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d99646244b5863ba-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.947 DEBUG:boto:path=/ppfahn-16/obj16 2024-03-04T17:40:43.947 DEBUG:boto:auth_path=/ppfahn-16/obj16 2024-03-04T17:40:43.947 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.947 DEBUG:boto:Path: /ppfahn-16/obj16 2024-03-04T17:40:43.947 DEBUG:boto:Data: 2024-03-04T17:40:43.947 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.947 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.947 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.947 DEBUG:boto:Params: {} 2024-03-04T17:40:43.947 DEBUG:boto:Token: None 2024-03-04T17:40:43.947 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj16 2024-03-04T17:40:43.948 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kiPpSN7eTkyfX0gibWc9tDxBAQg= 2024-03-04T17:40:43.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kiPpSN7eTkyfX0gibWc9tDxBAQg='} 2024-03-04T17:40:43.992 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b15796409537333d-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.992 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:43.992 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:43.992 DEBUG:boto:Method: HEAD 2024-03-04T17:40:43.993 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:43.993 DEBUG:boto:Data: 2024-03-04T17:40:43.993 DEBUG:boto:Headers: {} 2024-03-04T17:40:43.993 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.993 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.993 DEBUG:boto:Params: {} 2024-03-04T17:40:43.993 DEBUG:boto:Token: None 2024-03-04T17:40:43.993 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/ 2024-03-04T17:40:43.993 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T17:40:43.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T17:40:43.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002179f53e15c0438e-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:43.997 DEBUG:boto:path=/ppfahn-16/obj17 2024-03-04T17:40:43.998 DEBUG:boto:auth_path=/ppfahn-16/obj17 2024-03-04T17:40:43.998 DEBUG:boto:Method: PUT 2024-03-04T17:40:43.998 DEBUG:boto:Path: /ppfahn-16/obj17 2024-03-04T17:40:43.998 DEBUG:boto:Data: 2024-03-04T17:40:43.998 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:43.998 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:43.998 DEBUG:boto:Port: 8000 2024-03-04T17:40:43.998 DEBUG:boto:Params: {} 2024-03-04T17:40:43.998 DEBUG:boto:Token: None 2024-03-04T17:40:43.998 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:43 GMT /ppfahn-16/obj17 2024-03-04T17:40:43.998 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CZgw0ag+p01pvqzbz+7emVawm5s= 2024-03-04T17:40:43.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CZgw0ag+p01pvqzbz+7emVawm5s='} 2024-03-04T17:40:44.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf5ae0bef14a1964-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.006 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.007 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.007 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.007 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.007 DEBUG:boto:Data: 2024-03-04T17:40:44.007 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.007 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.007 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.007 DEBUG:boto:Params: {} 2024-03-04T17:40:44.007 DEBUG:boto:Token: None 2024-03-04T17:40:44.007 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.007 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.011 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e858e624a794995-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.011 DEBUG:boto:path=/ppfahn-16/obj18 2024-03-04T17:40:44.011 DEBUG:boto:auth_path=/ppfahn-16/obj18 2024-03-04T17:40:44.011 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.012 DEBUG:boto:Path: /ppfahn-16/obj18 2024-03-04T17:40:44.012 DEBUG:boto:Data: 2024-03-04T17:40:44.012 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.012 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.012 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.012 DEBUG:boto:Params: {} 2024-03-04T17:40:44.012 DEBUG:boto:Token: None 2024-03-04T17:40:44.012 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj18 2024-03-04T17:40:44.012 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eaLAyg0bVqT3QsLlSHSoC3pBcgw= 2024-03-04T17:40:44.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eaLAyg0bVqT3QsLlSHSoC3pBcgw='} 2024-03-04T17:40:44.056 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8d8f5157424aa1a-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.056 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.056 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.057 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.057 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.057 DEBUG:boto:Data: 2024-03-04T17:40:44.057 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.057 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.057 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.057 DEBUG:boto:Params: {} 2024-03-04T17:40:44.057 DEBUG:boto:Token: None 2024-03-04T17:40:44.057 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.057 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000859adddbde4dc888-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.062 DEBUG:boto:path=/ppfahn-16/obj19 2024-03-04T17:40:44.062 DEBUG:boto:auth_path=/ppfahn-16/obj19 2024-03-04T17:40:44.062 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.062 DEBUG:boto:Path: /ppfahn-16/obj19 2024-03-04T17:40:44.062 DEBUG:boto:Data: 2024-03-04T17:40:44.062 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.062 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.062 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.062 DEBUG:boto:Params: {} 2024-03-04T17:40:44.063 DEBUG:boto:Token: None 2024-03-04T17:40:44.063 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj19 2024-03-04T17:40:44.063 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VS7Ti011Wuckv+ymtFAHjqUGhvg= 2024-03-04T17:40:44.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VS7Ti011Wuckv+ymtFAHjqUGhvg='} 2024-03-04T17:40:44.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021d31ecccfca329a-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.071 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.071 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.071 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.071 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.072 DEBUG:boto:Data: 2024-03-04T17:40:44.072 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.072 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.072 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.072 DEBUG:boto:Params: {} 2024-03-04T17:40:44.072 DEBUG:boto:Token: None 2024-03-04T17:40:44.072 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.072 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.076 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000980143f4962a44-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.076 DEBUG:boto:path=/ppfahn-16/obj20 2024-03-04T17:40:44.076 DEBUG:boto:auth_path=/ppfahn-16/obj20 2024-03-04T17:40:44.076 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.076 DEBUG:boto:Path: /ppfahn-16/obj20 2024-03-04T17:40:44.077 DEBUG:boto:Data: 2024-03-04T17:40:44.077 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.077 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.077 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.077 DEBUG:boto:Params: {} 2024-03-04T17:40:44.077 DEBUG:boto:Token: None 2024-03-04T17:40:44.077 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj20 2024-03-04T17:40:44.077 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+siqT5DcDhL4Q7CoQef8yldtGJ0= 2024-03-04T17:40:44.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+siqT5DcDhL4Q7CoQef8yldtGJ0='} 2024-03-04T17:40:44.120 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e9665b84d883873-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.121 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.121 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.121 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.121 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.121 DEBUG:boto:Data: 2024-03-04T17:40:44.121 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.121 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.121 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.121 DEBUG:boto:Params: {} 2024-03-04T17:40:44.122 DEBUG:boto:Token: None 2024-03-04T17:40:44.122 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.122 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a55d1de3f33906b7-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.126 DEBUG:boto:path=/ppfahn-16/obj21 2024-03-04T17:40:44.126 DEBUG:boto:auth_path=/ppfahn-16/obj21 2024-03-04T17:40:44.126 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.126 DEBUG:boto:Path: /ppfahn-16/obj21 2024-03-04T17:40:44.126 DEBUG:boto:Data: 2024-03-04T17:40:44.126 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.126 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.127 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.127 DEBUG:boto:Params: {} 2024-03-04T17:40:44.127 DEBUG:boto:Token: None 2024-03-04T17:40:44.127 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj21 2024-03-04T17:40:44.127 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:n5DnqcyIED91/hUcYoKaQ7R4FhI= 2024-03-04T17:40:44.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:n5DnqcyIED91/hUcYoKaQ7R4FhI='} 2024-03-04T17:40:44.139 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c113331aa2d29f5-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.139 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.139 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.139 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.139 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.139 DEBUG:boto:Data: 2024-03-04T17:40:44.140 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.140 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.140 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.140 DEBUG:boto:Params: {} 2024-03-04T17:40:44.140 DEBUG:boto:Token: None 2024-03-04T17:40:44.140 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.140 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001259bbe3b9165061-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.144 DEBUG:boto:path=/ppfahn-16/obj22 2024-03-04T17:40:44.144 DEBUG:boto:auth_path=/ppfahn-16/obj22 2024-03-04T17:40:44.145 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.145 DEBUG:boto:Path: /ppfahn-16/obj22 2024-03-04T17:40:44.145 DEBUG:boto:Data: 2024-03-04T17:40:44.145 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.145 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.145 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.145 DEBUG:boto:Params: {} 2024-03-04T17:40:44.145 DEBUG:boto:Token: None 2024-03-04T17:40:44.145 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj22 2024-03-04T17:40:44.145 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XuIyssg8rA9AIpzHpZJnhT49yZ0= 2024-03-04T17:40:44.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XuIyssg8rA9AIpzHpZJnhT49yZ0='} 2024-03-04T17:40:44.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4779adf9ed68d87-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.188 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.188 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.189 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.189 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.189 DEBUG:boto:Data: 2024-03-04T17:40:44.189 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.189 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.189 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.189 DEBUG:boto:Params: {} 2024-03-04T17:40:44.189 DEBUG:boto:Token: None 2024-03-04T17:40:44.189 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.189 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054fb263a952a5a2c-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.194 DEBUG:boto:path=/ppfahn-16/obj23 2024-03-04T17:40:44.194 DEBUG:boto:auth_path=/ppfahn-16/obj23 2024-03-04T17:40:44.194 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.194 DEBUG:boto:Path: /ppfahn-16/obj23 2024-03-04T17:40:44.194 DEBUG:boto:Data: 2024-03-04T17:40:44.194 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.194 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.194 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.194 DEBUG:boto:Params: {} 2024-03-04T17:40:44.195 DEBUG:boto:Token: None 2024-03-04T17:40:44.195 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj23 2024-03-04T17:40:44.195 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3dv/bwaqyaYPFA6CHcGnieqGGkY= 2024-03-04T17:40:44.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3dv/bwaqyaYPFA6CHcGnieqGGkY='} 2024-03-04T17:40:44.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e0d9dd7dfb54f78c-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.203 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.204 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.204 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.204 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.204 DEBUG:boto:Data: 2024-03-04T17:40:44.204 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.204 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.204 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.204 DEBUG:boto:Params: {} 2024-03-04T17:40:44.204 DEBUG:boto:Token: None 2024-03-04T17:40:44.204 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.204 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.208 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084bb20639fac9af3-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.208 DEBUG:boto:path=/ppfahn-16/obj24 2024-03-04T17:40:44.208 DEBUG:boto:auth_path=/ppfahn-16/obj24 2024-03-04T17:40:44.208 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.209 DEBUG:boto:Path: /ppfahn-16/obj24 2024-03-04T17:40:44.209 DEBUG:boto:Data: 2024-03-04T17:40:44.209 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.209 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.209 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.209 DEBUG:boto:Params: {} 2024-03-04T17:40:44.209 DEBUG:boto:Token: None 2024-03-04T17:40:44.209 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj24 2024-03-04T17:40:44.209 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dSUB0gVALfwS80dilGhmAHfflxU= 2024-03-04T17:40:44.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dSUB0gVALfwS80dilGhmAHfflxU='} 2024-03-04T17:40:44.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060e0e4e616258819-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.252 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.252 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.252 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.253 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.253 DEBUG:boto:Data: 2024-03-04T17:40:44.253 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.253 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.254 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.254 DEBUG:boto:Params: {} 2024-03-04T17:40:44.254 DEBUG:boto:Token: None 2024-03-04T17:40:44.254 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.254 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.258 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1db9d018ad68eae-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.258 DEBUG:boto:path=/ppfahn-16/obj25 2024-03-04T17:40:44.258 DEBUG:boto:auth_path=/ppfahn-16/obj25 2024-03-04T17:40:44.258 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.258 DEBUG:boto:Path: /ppfahn-16/obj25 2024-03-04T17:40:44.258 DEBUG:boto:Data: 2024-03-04T17:40:44.259 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.259 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.259 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.259 DEBUG:boto:Params: {} 2024-03-04T17:40:44.259 DEBUG:boto:Token: None 2024-03-04T17:40:44.259 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj25 2024-03-04T17:40:44.259 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S+nf2CarJY+NQ2RfQKFV3MhUFQY= 2024-03-04T17:40:44.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S+nf2CarJY+NQ2RfQKFV3MhUFQY='} 2024-03-04T17:40:44.267 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000779a95cc2d58b58a-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.267 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.268 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.268 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.268 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.268 DEBUG:boto:Data: 2024-03-04T17:40:44.268 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.268 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.268 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.268 DEBUG:boto:Params: {} 2024-03-04T17:40:44.268 DEBUG:boto:Token: None 2024-03-04T17:40:44.269 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.269 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebcf806dd8b72dd0-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.273 DEBUG:boto:path=/ppfahn-16/obj26 2024-03-04T17:40:44.273 DEBUG:boto:auth_path=/ppfahn-16/obj26 2024-03-04T17:40:44.273 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.273 DEBUG:boto:Path: /ppfahn-16/obj26 2024-03-04T17:40:44.273 DEBUG:boto:Data: 2024-03-04T17:40:44.273 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.273 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.273 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.273 DEBUG:boto:Params: {} 2024-03-04T17:40:44.273 DEBUG:boto:Token: None 2024-03-04T17:40:44.273 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj26 2024-03-04T17:40:44.274 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T5BNQVClvkd13Ru+HXdLq+pr/Ec= 2024-03-04T17:40:44.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T5BNQVClvkd13Ru+HXdLq+pr/Ec='} 2024-03-04T17:40:44.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000668befdd40331e4f-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.316 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.316 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.317 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.317 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.317 DEBUG:boto:Data: 2024-03-04T17:40:44.317 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.317 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.317 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.317 DEBUG:boto:Params: {} 2024-03-04T17:40:44.317 DEBUG:boto:Token: None 2024-03-04T17:40:44.317 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.317 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e1c07362bcc70cb-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.322 DEBUG:boto:path=/ppfahn-16/obj27 2024-03-04T17:40:44.322 DEBUG:boto:auth_path=/ppfahn-16/obj27 2024-03-04T17:40:44.322 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.322 DEBUG:boto:Path: /ppfahn-16/obj27 2024-03-04T17:40:44.323 DEBUG:boto:Data: 2024-03-04T17:40:44.323 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.323 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.323 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.323 DEBUG:boto:Params: {} 2024-03-04T17:40:44.323 DEBUG:boto:Token: None 2024-03-04T17:40:44.323 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj27 2024-03-04T17:40:44.323 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nbI4qVw0OR5c2Fieu0rjuZwjwG0= 2024-03-04T17:40:44.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nbI4qVw0OR5c2Fieu0rjuZwjwG0='} 2024-03-04T17:40:44.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1eb427ed10bfffe-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.332 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.332 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.333 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.333 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.333 DEBUG:boto:Data: 2024-03-04T17:40:44.333 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.333 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.333 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.333 DEBUG:boto:Params: {} 2024-03-04T17:40:44.333 DEBUG:boto:Token: None 2024-03-04T17:40:44.333 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.333 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.338 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074b57f179c81833f-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.338 DEBUG:boto:path=/ppfahn-16/obj28 2024-03-04T17:40:44.338 DEBUG:boto:auth_path=/ppfahn-16/obj28 2024-03-04T17:40:44.338 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.339 DEBUG:boto:Path: /ppfahn-16/obj28 2024-03-04T17:40:44.339 DEBUG:boto:Data: 2024-03-04T17:40:44.339 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.339 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.339 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.339 DEBUG:boto:Params: {} 2024-03-04T17:40:44.339 DEBUG:boto:Token: None 2024-03-04T17:40:44.339 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj28 2024-03-04T17:40:44.339 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LsqULEMt2kgpY/kacxHgPkW5oAY= 2024-03-04T17:40:44.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LsqULEMt2kgpY/kacxHgPkW5oAY='} 2024-03-04T17:40:44.384 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a06f8d62759ab614-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.384 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.385 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.385 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.385 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.385 DEBUG:boto:Data: 2024-03-04T17:40:44.385 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.385 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.385 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.385 DEBUG:boto:Params: {} 2024-03-04T17:40:44.385 DEBUG:boto:Token: None 2024-03-04T17:40:44.385 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.386 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000194c642ee7595e2f-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.390 DEBUG:boto:path=/ppfahn-16/obj29 2024-03-04T17:40:44.390 DEBUG:boto:auth_path=/ppfahn-16/obj29 2024-03-04T17:40:44.390 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.390 DEBUG:boto:Path: /ppfahn-16/obj29 2024-03-04T17:40:44.390 DEBUG:boto:Data: 2024-03-04T17:40:44.391 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.391 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.391 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.391 DEBUG:boto:Params: {} 2024-03-04T17:40:44.391 DEBUG:boto:Token: None 2024-03-04T17:40:44.391 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj29 2024-03-04T17:40:44.391 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SJ/+hdfKGqA/5vSBdpNrS+p6nIc= 2024-03-04T17:40:44.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SJ/+hdfKGqA/5vSBdpNrS+p6nIc='} 2024-03-04T17:40:44.400 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f8af6c91a768fe1-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.400 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.400 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.400 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.400 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.400 DEBUG:boto:Data: 2024-03-04T17:40:44.400 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.400 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.400 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.400 DEBUG:boto:Params: {} 2024-03-04T17:40:44.401 DEBUG:boto:Token: None 2024-03-04T17:40:44.401 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.401 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000592a51afee0a0eee-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.405 DEBUG:boto:path=/ppfahn-16/obj30 2024-03-04T17:40:44.405 DEBUG:boto:auth_path=/ppfahn-16/obj30 2024-03-04T17:40:44.405 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.405 DEBUG:boto:Path: /ppfahn-16/obj30 2024-03-04T17:40:44.405 DEBUG:boto:Data: 2024-03-04T17:40:44.405 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.406 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.406 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.406 DEBUG:boto:Params: {} 2024-03-04T17:40:44.406 DEBUG:boto:Token: None 2024-03-04T17:40:44.406 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj30 2024-03-04T17:40:44.406 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6ICKbvJCLUqrDzKNJINnTucUw84= 2024-03-04T17:40:44.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6ICKbvJCLUqrDzKNJINnTucUw84='} 2024-03-04T17:40:44.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d975bd9c1c1dd7d-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.448 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.448 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.448 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.449 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.449 DEBUG:boto:Data: 2024-03-04T17:40:44.449 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.449 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.449 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.449 DEBUG:boto:Params: {} 2024-03-04T17:40:44.449 DEBUG:boto:Token: None 2024-03-04T17:40:44.449 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.449 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.453 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087d84d8374330965-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.454 DEBUG:boto:path=/ppfahn-16/obj31 2024-03-04T17:40:44.454 DEBUG:boto:auth_path=/ppfahn-16/obj31 2024-03-04T17:40:44.454 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.454 DEBUG:boto:Path: /ppfahn-16/obj31 2024-03-04T17:40:44.454 DEBUG:boto:Data: 2024-03-04T17:40:44.454 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.454 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.454 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.454 DEBUG:boto:Params: {} 2024-03-04T17:40:44.454 DEBUG:boto:Token: None 2024-03-04T17:40:44.455 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj31 2024-03-04T17:40:44.455 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jCbltS9+5qIt8fBTkyxEpefZvoE= 2024-03-04T17:40:44.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jCbltS9+5qIt8fBTkyxEpefZvoE='} 2024-03-04T17:40:44.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000232d8754f76a44f3-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.463 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.463 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.463 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.463 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.463 DEBUG:boto:Data: 2024-03-04T17:40:44.464 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.464 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.464 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.464 DEBUG:boto:Params: {} 2024-03-04T17:40:44.464 DEBUG:boto:Token: None 2024-03-04T17:40:44.464 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.464 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040d2677de7c86bdc-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.469 DEBUG:boto:path=/ppfahn-16/obj32 2024-03-04T17:40:44.469 DEBUG:boto:auth_path=/ppfahn-16/obj32 2024-03-04T17:40:44.469 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.469 DEBUG:boto:Path: /ppfahn-16/obj32 2024-03-04T17:40:44.469 DEBUG:boto:Data: 2024-03-04T17:40:44.469 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.469 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.469 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.469 DEBUG:boto:Params: {} 2024-03-04T17:40:44.469 DEBUG:boto:Token: None 2024-03-04T17:40:44.470 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj32 2024-03-04T17:40:44.470 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ydlN8stGkpHawB3OthPLcTa9H9c= 2024-03-04T17:40:44.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ydlN8stGkpHawB3OthPLcTa9H9c='} 2024-03-04T17:40:44.516 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bbaf381cea5fdd16-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.516 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.516 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.517 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.517 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.517 DEBUG:boto:Data: 2024-03-04T17:40:44.517 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.517 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.517 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.517 DEBUG:boto:Params: {} 2024-03-04T17:40:44.517 DEBUG:boto:Token: None 2024-03-04T17:40:44.517 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.517 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b514afcfee18efd-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.524 DEBUG:boto:path=/ppfahn-16/obj33 2024-03-04T17:40:44.524 DEBUG:boto:auth_path=/ppfahn-16/obj33 2024-03-04T17:40:44.524 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.524 DEBUG:boto:Path: /ppfahn-16/obj33 2024-03-04T17:40:44.524 DEBUG:boto:Data: 2024-03-04T17:40:44.525 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.525 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.525 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.525 DEBUG:boto:Params: {} 2024-03-04T17:40:44.525 DEBUG:boto:Token: None 2024-03-04T17:40:44.525 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj33 2024-03-04T17:40:44.525 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uUj/qdprHlPRAHk1VvP6CTGJ0KU= 2024-03-04T17:40:44.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uUj/qdprHlPRAHk1VvP6CTGJ0KU='} 2024-03-04T17:40:44.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062fbfd7c0f8f679e-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.534 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.534 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.534 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.535 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.535 DEBUG:boto:Data: 2024-03-04T17:40:44.535 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.535 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.535 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.535 DEBUG:boto:Params: {} 2024-03-04T17:40:44.535 DEBUG:boto:Token: None 2024-03-04T17:40:44.535 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.535 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.540 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000966f14e7d3d128e9-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.540 DEBUG:boto:path=/ppfahn-16/obj34 2024-03-04T17:40:44.540 DEBUG:boto:auth_path=/ppfahn-16/obj34 2024-03-04T17:40:44.540 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.540 DEBUG:boto:Path: /ppfahn-16/obj34 2024-03-04T17:40:44.540 DEBUG:boto:Data: 2024-03-04T17:40:44.540 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.540 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.541 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.541 DEBUG:boto:Params: {} 2024-03-04T17:40:44.541 DEBUG:boto:Token: None 2024-03-04T17:40:44.541 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj34 2024-03-04T17:40:44.541 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RwCFGgm/6AY7XdwZTAFs2VILVgk= 2024-03-04T17:40:44.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RwCFGgm/6AY7XdwZTAFs2VILVgk='} 2024-03-04T17:40:44.588 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000aa7752c455daf4b-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.588 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.589 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.589 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.589 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.589 DEBUG:boto:Data: 2024-03-04T17:40:44.589 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.589 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.589 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.589 DEBUG:boto:Params: {} 2024-03-04T17:40:44.589 DEBUG:boto:Token: None 2024-03-04T17:40:44.589 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.590 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d90522445c28727-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.594 DEBUG:boto:path=/ppfahn-16/obj35 2024-03-04T17:40:44.594 DEBUG:boto:auth_path=/ppfahn-16/obj35 2024-03-04T17:40:44.594 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.594 DEBUG:boto:Path: /ppfahn-16/obj35 2024-03-04T17:40:44.594 DEBUG:boto:Data: 2024-03-04T17:40:44.594 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.595 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.595 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.595 DEBUG:boto:Params: {} 2024-03-04T17:40:44.595 DEBUG:boto:Token: None 2024-03-04T17:40:44.595 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj35 2024-03-04T17:40:44.595 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S3qW0nA2K267pB4oBoS5lYCCras= 2024-03-04T17:40:44.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S3qW0nA2K267pB4oBoS5lYCCras='} 2024-03-04T17:40:44.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c86aded11eecb97a-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.604 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.604 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.604 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.604 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.604 DEBUG:boto:Data: 2024-03-04T17:40:44.604 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.604 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.605 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.605 DEBUG:boto:Params: {} 2024-03-04T17:40:44.605 DEBUG:boto:Token: None 2024-03-04T17:40:44.605 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.605 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090c9e17ba8bfc041-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.609 DEBUG:boto:path=/ppfahn-16/obj36 2024-03-04T17:40:44.609 DEBUG:boto:auth_path=/ppfahn-16/obj36 2024-03-04T17:40:44.610 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.610 DEBUG:boto:Path: /ppfahn-16/obj36 2024-03-04T17:40:44.610 DEBUG:boto:Data: 2024-03-04T17:40:44.610 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.610 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.610 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.610 DEBUG:boto:Params: {} 2024-03-04T17:40:44.610 DEBUG:boto:Token: None 2024-03-04T17:40:44.610 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj36 2024-03-04T17:40:44.610 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:X73DsXkWcwu5tClBkoUIt5Yj2Cg= 2024-03-04T17:40:44.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:X73DsXkWcwu5tClBkoUIt5Yj2Cg='} 2024-03-04T17:40:44.652 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088765e8004952884-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.652 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.652 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.653 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.653 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.653 DEBUG:boto:Data: 2024-03-04T17:40:44.653 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.653 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.653 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.653 DEBUG:boto:Params: {} 2024-03-04T17:40:44.653 DEBUG:boto:Token: None 2024-03-04T17:40:44.653 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.653 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000520fc915cc77bd21-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.658 DEBUG:boto:path=/ppfahn-16/obj37 2024-03-04T17:40:44.658 DEBUG:boto:auth_path=/ppfahn-16/obj37 2024-03-04T17:40:44.659 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.659 DEBUG:boto:Path: /ppfahn-16/obj37 2024-03-04T17:40:44.659 DEBUG:boto:Data: 2024-03-04T17:40:44.659 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.659 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.659 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.659 DEBUG:boto:Params: {} 2024-03-04T17:40:44.659 DEBUG:boto:Token: None 2024-03-04T17:40:44.659 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj37 2024-03-04T17:40:44.659 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zYkL3s09o5vcbkC8LWq+ZYFoZSE= 2024-03-04T17:40:44.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zYkL3s09o5vcbkC8LWq+ZYFoZSE='} 2024-03-04T17:40:44.673 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2a5007c12f45c46-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.673 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.673 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.674 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.674 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.674 DEBUG:boto:Data: 2024-03-04T17:40:44.674 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.674 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.674 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.674 DEBUG:boto:Params: {} 2024-03-04T17:40:44.674 DEBUG:boto:Token: None 2024-03-04T17:40:44.674 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.674 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083e3ba9e28551e7f-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.682 DEBUG:boto:path=/ppfahn-16/obj38 2024-03-04T17:40:44.682 DEBUG:boto:auth_path=/ppfahn-16/obj38 2024-03-04T17:40:44.682 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.682 DEBUG:boto:Path: /ppfahn-16/obj38 2024-03-04T17:40:44.682 DEBUG:boto:Data: 2024-03-04T17:40:44.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.683 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.683 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.683 DEBUG:boto:Params: {} 2024-03-04T17:40:44.683 DEBUG:boto:Token: None 2024-03-04T17:40:44.683 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj38 2024-03-04T17:40:44.683 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/s4PQsf28AiVpQGO/ykeMyZwAog= 2024-03-04T17:40:44.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/s4PQsf28AiVpQGO/ykeMyZwAog='} 2024-03-04T17:40:44.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e421833a17da58e7-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.728 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.728 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.729 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.729 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.729 DEBUG:boto:Data: 2024-03-04T17:40:44.729 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.729 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.729 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.729 DEBUG:boto:Params: {} 2024-03-04T17:40:44.729 DEBUG:boto:Token: None 2024-03-04T17:40:44.729 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.729 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d72aadd0894b5cf6-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.734 DEBUG:boto:path=/ppfahn-16/obj39 2024-03-04T17:40:44.735 DEBUG:boto:auth_path=/ppfahn-16/obj39 2024-03-04T17:40:44.735 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.735 DEBUG:boto:Path: /ppfahn-16/obj39 2024-03-04T17:40:44.735 DEBUG:boto:Data: 2024-03-04T17:40:44.735 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.735 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.735 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.735 DEBUG:boto:Params: {} 2024-03-04T17:40:44.735 DEBUG:boto:Token: None 2024-03-04T17:40:44.735 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj39 2024-03-04T17:40:44.736 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1pBFBV4pS/Z8AZ7vNMCjRBZj0iE= 2024-03-04T17:40:44.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1pBFBV4pS/Z8AZ7vNMCjRBZj0iE='} 2024-03-04T17:40:44.744 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b6af08c2602d27b-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.744 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.744 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.744 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.744 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.744 DEBUG:boto:Data: 2024-03-04T17:40:44.744 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.744 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.744 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.744 DEBUG:boto:Params: {} 2024-03-04T17:40:44.745 DEBUG:boto:Token: None 2024-03-04T17:40:44.745 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.745 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.749 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3f503c205c2b591-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.749 DEBUG:boto:path=/ppfahn-16/obj40 2024-03-04T17:40:44.749 DEBUG:boto:auth_path=/ppfahn-16/obj40 2024-03-04T17:40:44.749 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.749 DEBUG:boto:Path: /ppfahn-16/obj40 2024-03-04T17:40:44.749 DEBUG:boto:Data: 2024-03-04T17:40:44.749 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.749 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.749 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.750 DEBUG:boto:Params: {} 2024-03-04T17:40:44.750 DEBUG:boto:Token: None 2024-03-04T17:40:44.750 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj40 2024-03-04T17:40:44.750 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:L2rJVIEhBux35JbaASqva0GYu4Y= 2024-03-04T17:40:44.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:L2rJVIEhBux35JbaASqva0GYu4Y='} 2024-03-04T17:40:44.796 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f231862d5a10d63-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.797 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.797 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.797 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.797 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.797 DEBUG:boto:Data: 2024-03-04T17:40:44.797 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.797 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.797 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.797 DEBUG:boto:Params: {} 2024-03-04T17:40:44.797 DEBUG:boto:Token: None 2024-03-04T17:40:44.798 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.798 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.802 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007fca701807e2bbe8-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.802 DEBUG:boto:path=/ppfahn-16/obj41 2024-03-04T17:40:44.802 DEBUG:boto:auth_path=/ppfahn-16/obj41 2024-03-04T17:40:44.802 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.802 DEBUG:boto:Path: /ppfahn-16/obj41 2024-03-04T17:40:44.802 DEBUG:boto:Data: 2024-03-04T17:40:44.803 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.803 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.803 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.803 DEBUG:boto:Params: {} 2024-03-04T17:40:44.803 DEBUG:boto:Token: None 2024-03-04T17:40:44.803 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj41 2024-03-04T17:40:44.803 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2NXoht6SMhjoNFvUXe7JYNk7G8M= 2024-03-04T17:40:44.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2NXoht6SMhjoNFvUXe7JYNk7G8M='} 2024-03-04T17:40:44.811 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f9b8800798e562f-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.811 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.812 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.812 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.812 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.812 DEBUG:boto:Data: 2024-03-04T17:40:44.812 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.812 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.812 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.812 DEBUG:boto:Params: {} 2024-03-04T17:40:44.812 DEBUG:boto:Token: None 2024-03-04T17:40:44.812 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.813 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.817 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e163a0a022020793-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.817 DEBUG:boto:path=/ppfahn-16/obj42 2024-03-04T17:40:44.817 DEBUG:boto:auth_path=/ppfahn-16/obj42 2024-03-04T17:40:44.817 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.817 DEBUG:boto:Path: /ppfahn-16/obj42 2024-03-04T17:40:44.817 DEBUG:boto:Data: 2024-03-04T17:40:44.817 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.818 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.818 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.818 DEBUG:boto:Params: {} 2024-03-04T17:40:44.818 DEBUG:boto:Token: None 2024-03-04T17:40:44.818 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj42 2024-03-04T17:40:44.818 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gETzSWJuh2WQfQZTsscxxDrS8EQ= 2024-03-04T17:40:44.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gETzSWJuh2WQfQZTsscxxDrS8EQ='} 2024-03-04T17:40:44.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2a77943b7794c0a-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.864 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.864 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.865 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.865 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.865 DEBUG:boto:Data: 2024-03-04T17:40:44.865 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.865 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.865 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.865 DEBUG:boto:Params: {} 2024-03-04T17:40:44.865 DEBUG:boto:Token: None 2024-03-04T17:40:44.865 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.865 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e45f66b2d53b548-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.870 DEBUG:boto:path=/ppfahn-16/obj43 2024-03-04T17:40:44.870 DEBUG:boto:auth_path=/ppfahn-16/obj43 2024-03-04T17:40:44.870 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.870 DEBUG:boto:Path: /ppfahn-16/obj43 2024-03-04T17:40:44.870 DEBUG:boto:Data: 2024-03-04T17:40:44.870 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.870 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.870 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.870 DEBUG:boto:Params: {} 2024-03-04T17:40:44.871 DEBUG:boto:Token: None 2024-03-04T17:40:44.871 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj43 2024-03-04T17:40:44.871 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vrs6qH3/EU7uxZwLffvkVOOjMGA= 2024-03-04T17:40:44.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vrs6qH3/EU7uxZwLffvkVOOjMGA='} 2024-03-04T17:40:44.879 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d951cac9ac89a11-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.879 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.879 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.879 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.879 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.879 DEBUG:boto:Data: 2024-03-04T17:40:44.879 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.879 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.879 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.879 DEBUG:boto:Params: {} 2024-03-04T17:40:44.879 DEBUG:boto:Token: None 2024-03-04T17:40:44.880 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.880 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.884 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006cd35ac99c8f129d-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.884 DEBUG:boto:path=/ppfahn-16/obj44 2024-03-04T17:40:44.884 DEBUG:boto:auth_path=/ppfahn-16/obj44 2024-03-04T17:40:44.884 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.884 DEBUG:boto:Path: /ppfahn-16/obj44 2024-03-04T17:40:44.884 DEBUG:boto:Data: 2024-03-04T17:40:44.884 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.885 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.885 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.885 DEBUG:boto:Params: {} 2024-03-04T17:40:44.885 DEBUG:boto:Token: None 2024-03-04T17:40:44.885 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj44 2024-03-04T17:40:44.885 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dBOoV0sUyWiD5G+VhQmoAf7lMVA= 2024-03-04T17:40:44.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dBOoV0sUyWiD5G+VhQmoAf7lMVA='} 2024-03-04T17:40:44.928 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007edfd1a234fa8adb-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.928 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.928 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.929 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.929 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.929 DEBUG:boto:Data: 2024-03-04T17:40:44.929 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.929 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.929 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.929 DEBUG:boto:Params: {} 2024-03-04T17:40:44.929 DEBUG:boto:Token: None 2024-03-04T17:40:44.929 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.929 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.933 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f59967247e4732e-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.934 DEBUG:boto:path=/ppfahn-16/obj45 2024-03-04T17:40:44.934 DEBUG:boto:auth_path=/ppfahn-16/obj45 2024-03-04T17:40:44.934 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.934 DEBUG:boto:Path: /ppfahn-16/obj45 2024-03-04T17:40:44.934 DEBUG:boto:Data: 2024-03-04T17:40:44.934 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.934 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.934 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.934 DEBUG:boto:Params: {} 2024-03-04T17:40:44.934 DEBUG:boto:Token: None 2024-03-04T17:40:44.935 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj45 2024-03-04T17:40:44.935 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oJElWmmgEtgX9+HLhxzvySK0qOs= 2024-03-04T17:40:44.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oJElWmmgEtgX9+HLhxzvySK0qOs='} 2024-03-04T17:40:44.944 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055ef1ee3b3eef8a6-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.944 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.944 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.944 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.944 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.944 DEBUG:boto:Data: 2024-03-04T17:40:44.944 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.944 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.945 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.945 DEBUG:boto:Params: {} 2024-03-04T17:40:44.945 DEBUG:boto:Token: None 2024-03-04T17:40:44.945 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.945 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:44.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d26daf4af46b7503-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.950 DEBUG:boto:path=/ppfahn-16/obj46 2024-03-04T17:40:44.950 DEBUG:boto:auth_path=/ppfahn-16/obj46 2024-03-04T17:40:44.950 DEBUG:boto:Method: PUT 2024-03-04T17:40:44.950 DEBUG:boto:Path: /ppfahn-16/obj46 2024-03-04T17:40:44.950 DEBUG:boto:Data: 2024-03-04T17:40:44.950 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:44.950 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.950 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.950 DEBUG:boto:Params: {} 2024-03-04T17:40:44.950 DEBUG:boto:Token: None 2024-03-04T17:40:44.950 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/obj46 2024-03-04T17:40:44.951 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cHKDrexfcbr5BT68bIAa8zsb2y8= 2024-03-04T17:40:44.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cHKDrexfcbr5BT68bIAa8zsb2y8='} 2024-03-04T17:40:44.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2c0bf932dc24d9f-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:44.996 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:44.996 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:44.997 DEBUG:boto:Method: HEAD 2024-03-04T17:40:44.997 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:44.997 DEBUG:boto:Data: 2024-03-04T17:40:44.997 DEBUG:boto:Headers: {} 2024-03-04T17:40:44.997 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:44.997 DEBUG:boto:Port: 8000 2024-03-04T17:40:44.997 DEBUG:boto:Params: {} 2024-03-04T17:40:44.997 DEBUG:boto:Token: None 2024-03-04T17:40:44.997 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:44 GMT /ppfahn-16/ 2024-03-04T17:40:44.998 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T17:40:44.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T17:40:45.001 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae6158d3b89aa7b1-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.002 DEBUG:boto:path=/ppfahn-16/obj47 2024-03-04T17:40:45.002 DEBUG:boto:auth_path=/ppfahn-16/obj47 2024-03-04T17:40:45.002 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.002 DEBUG:boto:Path: /ppfahn-16/obj47 2024-03-04T17:40:45.002 DEBUG:boto:Data: 2024-03-04T17:40:45.002 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.002 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:45.002 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.002 DEBUG:boto:Params: {} 2024-03-04T17:40:45.003 DEBUG:boto:Token: None 2024-03-04T17:40:45.003 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-16/obj47 2024-03-04T17:40:45.003 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jKVhfyGcO3pDQrOzTqNiYiCL8tw= 2024-03-04T17:40:45.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jKVhfyGcO3pDQrOzTqNiYiCL8tw='} 2024-03-04T17:40:45.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007313120985bfd186-0065e6079d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.011 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:45.011 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:45.012 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.012 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:45.012 DEBUG:boto:Data: 2024-03-04T17:40:45.012 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.012 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:45.012 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.012 DEBUG:boto:Params: {} 2024-03-04T17:40:45.012 DEBUG:boto:Token: None 2024-03-04T17:40:45.012 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-16/ 2024-03-04T17:40:45.012 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4saRfKQPUTEfX7GjBLAv4DeSJIw= 2024-03-04T17:40:45.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4saRfKQPUTEfX7GjBLAv4DeSJIw=', 'Content-Length': '0'} 2024-03-04T17:40:45.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078466d3a4e424606-0065e6079d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.016 DEBUG:boto:path=/ppfahn-16/obj48 2024-03-04T17:40:45.016 DEBUG:boto:auth_path=/ppfahn-16/obj48 2024-03-04T17:40:45.017 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.017 DEBUG:boto:Path: /ppfahn-16/obj48 2024-03-04T17:40:45.017 DEBUG:boto:Data: 2024-03-04T17:40:45.017 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.017 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:45.017 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.017 DEBUG:boto:Params: {} 2024-03-04T17:40:45.017 DEBUG:boto:Token: None 2024-03-04T17:40:45.017 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-16/obj48 2024-03-04T17:40:45.017 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1saF5tVTuc8a04rMDlDNhWyH8NY= 2024-03-04T17:40:45.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1saF5tVTuc8a04rMDlDNhWyH8NY='} 2024-03-04T17:40:45.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f8ea052fb462931-0065e6079d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.061 DEBUG:boto:path=/ppfahn-16/ 2024-03-04T17:40:45.061 DEBUG:boto:auth_path=/ppfahn-16/ 2024-03-04T17:40:45.061 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.061 DEBUG:boto:Path: /ppfahn-16/ 2024-03-04T17:40:45.061 DEBUG:boto:Data: 2024-03-04T17:40:45.061 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.061 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:45.061 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.061 DEBUG:boto:Params: {} 2024-03-04T17:40:45.061 DEBUG:boto:Token: None 2024-03-04T17:40:45.062 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-16/ 2024-03-04T17:40:45.062 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4saRfKQPUTEfX7GjBLAv4DeSJIw= 2024-03-04T17:40:45.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4saRfKQPUTEfX7GjBLAv4DeSJIw=', 'Content-Length': '0'} 2024-03-04T17:40:45.065 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000499e66335616f5d9-0065e6079d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.066 DEBUG:boto:path=/ppfahn-16/obj49 2024-03-04T17:40:45.066 DEBUG:boto:auth_path=/ppfahn-16/obj49 2024-03-04T17:40:45.066 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.066 DEBUG:boto:Path: /ppfahn-16/obj49 2024-03-04T17:40:45.066 DEBUG:boto:Data: 2024-03-04T17:40:45.066 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.066 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:40:45.066 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.066 DEBUG:boto:Params: {} 2024-03-04T17:40:45.066 DEBUG:boto:Token: None 2024-03-04T17:40:45.067 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-16/obj49 2024-03-04T17:40:45.067 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EZtTBRz59vkkN9NY01AKz6AiPPg= 2024-03-04T17:40:45.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EZtTBRz59vkkN9NY01AKz6AiPPg='} 2024-03-04T17:40:45.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3b3ecc4042d822c-0065e6079d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.075 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.075 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.075 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.075 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.075 DEBUG:boto:Data: 2024-03-04T17:40:45.076 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.076 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.076 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.076 DEBUG:boto:Params: {} 2024-03-04T17:40:45.076 DEBUG:boto:Token: None 2024-03-04T17:40:45.076 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.076 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.081 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5b56bfefcb80fff-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.081 DEBUG:boto:path=/ppfahn-17/obj1 2024-03-04T17:40:45.081 DEBUG:boto:auth_path=/ppfahn-17/obj1 2024-03-04T17:40:45.081 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.081 DEBUG:boto:Path: /ppfahn-17/obj1 2024-03-04T17:40:45.081 DEBUG:boto:Data: 2024-03-04T17:40:45.082 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.082 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.082 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.082 DEBUG:boto:Params: {} 2024-03-04T17:40:45.082 DEBUG:boto:Token: None 2024-03-04T17:40:45.082 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj1 2024-03-04T17:40:45.082 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yDaLHN9NVF86OeyyypZaMMnSCUM= 2024-03-04T17:40:45.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yDaLHN9NVF86OeyyypZaMMnSCUM='} 2024-03-04T17:40:45.124 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c691aadcf08dd70d-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.124 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.124 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.125 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.125 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.125 DEBUG:boto:Data: 2024-03-04T17:40:45.125 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.125 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.125 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.125 DEBUG:boto:Params: {} 2024-03-04T17:40:45.125 DEBUG:boto:Token: None 2024-03-04T17:40:45.125 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.125 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.129 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d5e0a1dbc5cb6db-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.130 DEBUG:boto:path=/ppfahn-17/obj2 2024-03-04T17:40:45.130 DEBUG:boto:auth_path=/ppfahn-17/obj2 2024-03-04T17:40:45.130 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.130 DEBUG:boto:Path: /ppfahn-17/obj2 2024-03-04T17:40:45.131 DEBUG:boto:Data: 2024-03-04T17:40:45.131 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.131 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.131 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.131 DEBUG:boto:Params: {} 2024-03-04T17:40:45.131 DEBUG:boto:Token: None 2024-03-04T17:40:45.131 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj2 2024-03-04T17:40:45.132 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LSmslRrwbnLBGx60LRqeI5/Yzcg= 2024-03-04T17:40:45.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LSmslRrwbnLBGx60LRqeI5/Yzcg='} 2024-03-04T17:40:45.144 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000641b903d8ac8d934-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.144 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.144 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.144 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.145 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.145 DEBUG:boto:Data: 2024-03-04T17:40:45.145 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.145 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.145 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.145 DEBUG:boto:Params: {} 2024-03-04T17:40:45.145 DEBUG:boto:Token: None 2024-03-04T17:40:45.146 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.146 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e34a4de6b95b3367-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.150 DEBUG:boto:path=/ppfahn-17/obj3 2024-03-04T17:40:45.151 DEBUG:boto:auth_path=/ppfahn-17/obj3 2024-03-04T17:40:45.151 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.151 DEBUG:boto:Path: /ppfahn-17/obj3 2024-03-04T17:40:45.151 DEBUG:boto:Data: 2024-03-04T17:40:45.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.151 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.151 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.152 DEBUG:boto:Params: {} 2024-03-04T17:40:45.152 DEBUG:boto:Token: None 2024-03-04T17:40:45.152 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj3 2024-03-04T17:40:45.152 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SkZwVHAG6CGafHFnUcTfIbX4eBA= 2024-03-04T17:40:45.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SkZwVHAG6CGafHFnUcTfIbX4eBA='} 2024-03-04T17:40:45.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fee932b4845f1eee-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.196 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.196 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.196 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.197 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.197 DEBUG:boto:Data: 2024-03-04T17:40:45.197 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.197 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.197 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.197 DEBUG:boto:Params: {} 2024-03-04T17:40:45.197 DEBUG:boto:Token: None 2024-03-04T17:40:45.197 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.197 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.201 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8eaa3b4459c1eaa-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.201 DEBUG:boto:path=/ppfahn-17/obj4 2024-03-04T17:40:45.201 DEBUG:boto:auth_path=/ppfahn-17/obj4 2024-03-04T17:40:45.202 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.202 DEBUG:boto:Path: /ppfahn-17/obj4 2024-03-04T17:40:45.202 DEBUG:boto:Data: 2024-03-04T17:40:45.202 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.202 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.202 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.202 DEBUG:boto:Params: {} 2024-03-04T17:40:45.202 DEBUG:boto:Token: None 2024-03-04T17:40:45.202 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj4 2024-03-04T17:40:45.202 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aGFsD0am4GBu2d6pq4KjI+7RJNo= 2024-03-04T17:40:45.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aGFsD0am4GBu2d6pq4KjI+7RJNo='} 2024-03-04T17:40:45.213 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000369ebfb905ffd1cc-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.214 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.214 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.214 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.214 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.214 DEBUG:boto:Data: 2024-03-04T17:40:45.214 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.214 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.214 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.214 DEBUG:boto:Params: {} 2024-03-04T17:40:45.214 DEBUG:boto:Token: None 2024-03-04T17:40:45.215 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.215 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003da358e067b354ea-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.219 DEBUG:boto:path=/ppfahn-17/obj5 2024-03-04T17:40:45.219 DEBUG:boto:auth_path=/ppfahn-17/obj5 2024-03-04T17:40:45.219 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.219 DEBUG:boto:Path: /ppfahn-17/obj5 2024-03-04T17:40:45.219 DEBUG:boto:Data: 2024-03-04T17:40:45.219 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.219 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.219 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.219 DEBUG:boto:Params: {} 2024-03-04T17:40:45.220 DEBUG:boto:Token: None 2024-03-04T17:40:45.220 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj5 2024-03-04T17:40:45.220 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mHTg3x99jvaD5pqr5e+YR9bNILM= 2024-03-04T17:40:45.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mHTg3x99jvaD5pqr5e+YR9bNILM='} 2024-03-04T17:40:45.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6e7a8824a34fa8f-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.264 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.264 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.265 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.265 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.265 DEBUG:boto:Data: 2024-03-04T17:40:45.265 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.265 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.265 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.265 DEBUG:boto:Params: {} 2024-03-04T17:40:45.265 DEBUG:boto:Token: None 2024-03-04T17:40:45.265 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.265 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.270 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000370ca0935fd2a0e6-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.270 DEBUG:boto:path=/ppfahn-17/obj6 2024-03-04T17:40:45.270 DEBUG:boto:auth_path=/ppfahn-17/obj6 2024-03-04T17:40:45.270 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.270 DEBUG:boto:Path: /ppfahn-17/obj6 2024-03-04T17:40:45.270 DEBUG:boto:Data: 2024-03-04T17:40:45.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.270 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.270 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.271 DEBUG:boto:Params: {} 2024-03-04T17:40:45.271 DEBUG:boto:Token: None 2024-03-04T17:40:45.271 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj6 2024-03-04T17:40:45.271 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RRD2rZGBdTtch233M4PbIlXn81s= 2024-03-04T17:40:45.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RRD2rZGBdTtch233M4PbIlXn81s='} 2024-03-04T17:40:45.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f38b8dd4d009344-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.279 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.279 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.279 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.280 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.280 DEBUG:boto:Data: 2024-03-04T17:40:45.280 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.280 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.280 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.280 DEBUG:boto:Params: {} 2024-03-04T17:40:45.280 DEBUG:boto:Token: None 2024-03-04T17:40:45.280 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.280 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002207dd53dc885ad6-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.284 DEBUG:boto:path=/ppfahn-17/obj7 2024-03-04T17:40:45.284 DEBUG:boto:auth_path=/ppfahn-17/obj7 2024-03-04T17:40:45.284 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.285 DEBUG:boto:Path: /ppfahn-17/obj7 2024-03-04T17:40:45.285 DEBUG:boto:Data: 2024-03-04T17:40:45.285 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.285 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.285 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.285 DEBUG:boto:Params: {} 2024-03-04T17:40:45.285 DEBUG:boto:Token: None 2024-03-04T17:40:45.285 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj7 2024-03-04T17:40:45.285 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TDNpslkjQNWD+EsXIjZjB55Iriw= 2024-03-04T17:40:45.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TDNpslkjQNWD+EsXIjZjB55Iriw='} 2024-03-04T17:40:45.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000714b5469cbfa6118-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.328 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.328 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.328 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.329 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.329 DEBUG:boto:Data: 2024-03-04T17:40:45.329 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.329 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.329 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.329 DEBUG:boto:Params: {} 2024-03-04T17:40:45.329 DEBUG:boto:Token: None 2024-03-04T17:40:45.329 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.329 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.333 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f0136c27560b35b-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.334 DEBUG:boto:path=/ppfahn-17/obj8 2024-03-04T17:40:45.334 DEBUG:boto:auth_path=/ppfahn-17/obj8 2024-03-04T17:40:45.334 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.334 DEBUG:boto:Path: /ppfahn-17/obj8 2024-03-04T17:40:45.334 DEBUG:boto:Data: 2024-03-04T17:40:45.334 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.334 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.334 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.334 DEBUG:boto:Params: {} 2024-03-04T17:40:45.334 DEBUG:boto:Token: None 2024-03-04T17:40:45.335 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj8 2024-03-04T17:40:45.335 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:32cUfduQM8/N3teYTZHcL676F/0= 2024-03-04T17:40:45.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:32cUfduQM8/N3teYTZHcL676F/0='} 2024-03-04T17:40:45.343 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cacf3606288e1040-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.343 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.344 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.344 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.344 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.344 DEBUG:boto:Data: 2024-03-04T17:40:45.344 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.344 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.344 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.344 DEBUG:boto:Params: {} 2024-03-04T17:40:45.344 DEBUG:boto:Token: None 2024-03-04T17:40:45.345 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.345 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.349 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d3d0921af527010-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.349 DEBUG:boto:path=/ppfahn-17/obj9 2024-03-04T17:40:45.349 DEBUG:boto:auth_path=/ppfahn-17/obj9 2024-03-04T17:40:45.349 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.350 DEBUG:boto:Path: /ppfahn-17/obj9 2024-03-04T17:40:45.350 DEBUG:boto:Data: 2024-03-04T17:40:45.350 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.350 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.350 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.350 DEBUG:boto:Params: {} 2024-03-04T17:40:45.350 DEBUG:boto:Token: None 2024-03-04T17:40:45.350 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj9 2024-03-04T17:40:45.350 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jhXwfSVKNRKCI0XCNFRLLkS68KA= 2024-03-04T17:40:45.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jhXwfSVKNRKCI0XCNFRLLkS68KA='} 2024-03-04T17:40:45.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000553843d86cfcc417-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.392 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.392 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.393 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.393 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.393 DEBUG:boto:Data: 2024-03-04T17:40:45.393 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.393 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.393 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.393 DEBUG:boto:Params: {} 2024-03-04T17:40:45.393 DEBUG:boto:Token: None 2024-03-04T17:40:45.393 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.393 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.397 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc6462eb93a6cec6-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.398 DEBUG:boto:path=/ppfahn-17/obj10 2024-03-04T17:40:45.398 DEBUG:boto:auth_path=/ppfahn-17/obj10 2024-03-04T17:40:45.398 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.398 DEBUG:boto:Path: /ppfahn-17/obj10 2024-03-04T17:40:45.398 DEBUG:boto:Data: 2024-03-04T17:40:45.398 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.398 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.398 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.398 DEBUG:boto:Params: {} 2024-03-04T17:40:45.398 DEBUG:boto:Token: None 2024-03-04T17:40:45.399 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj10 2024-03-04T17:40:45.399 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2KhvwDFz4QOQc6Dc5jGi2tU3spo= 2024-03-04T17:40:45.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2KhvwDFz4QOQc6Dc5jGi2tU3spo='} 2024-03-04T17:40:45.407 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc471d8dd0b1035c-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.407 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.408 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.408 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.408 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.408 DEBUG:boto:Data: 2024-03-04T17:40:45.408 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.408 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.408 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.408 DEBUG:boto:Params: {} 2024-03-04T17:40:45.408 DEBUG:boto:Token: None 2024-03-04T17:40:45.409 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.409 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000824e170ba060605-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.413 DEBUG:boto:path=/ppfahn-17/obj11 2024-03-04T17:40:45.413 DEBUG:boto:auth_path=/ppfahn-17/obj11 2024-03-04T17:40:45.413 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.413 DEBUG:boto:Path: /ppfahn-17/obj11 2024-03-04T17:40:45.413 DEBUG:boto:Data: 2024-03-04T17:40:45.413 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.413 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.413 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.414 DEBUG:boto:Params: {} 2024-03-04T17:40:45.414 DEBUG:boto:Token: None 2024-03-04T17:40:45.414 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj11 2024-03-04T17:40:45.414 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nQDh7BLfBmYWB0iA7/y4huVGg3Y= 2024-03-04T17:40:45.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nQDh7BLfBmYWB0iA7/y4huVGg3Y='} 2024-03-04T17:40:45.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068313b9e4fbec8c9-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.456 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.456 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.457 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.457 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.457 DEBUG:boto:Data: 2024-03-04T17:40:45.457 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.457 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.457 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.457 DEBUG:boto:Params: {} 2024-03-04T17:40:45.457 DEBUG:boto:Token: None 2024-03-04T17:40:45.457 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.457 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.461 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007eee396b18406604-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.462 DEBUG:boto:path=/ppfahn-17/obj12 2024-03-04T17:40:45.462 DEBUG:boto:auth_path=/ppfahn-17/obj12 2024-03-04T17:40:45.462 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.462 DEBUG:boto:Path: /ppfahn-17/obj12 2024-03-04T17:40:45.462 DEBUG:boto:Data: 2024-03-04T17:40:45.462 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.462 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.462 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.463 DEBUG:boto:Params: {} 2024-03-04T17:40:45.463 DEBUG:boto:Token: None 2024-03-04T17:40:45.463 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj12 2024-03-04T17:40:45.463 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bpMBzJF2aShgS8tXNyR+k2FZrIk= 2024-03-04T17:40:45.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bpMBzJF2aShgS8tXNyR+k2FZrIk='} 2024-03-04T17:40:45.471 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9f58edc49189b1d-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.471 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.471 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.471 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.472 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.472 DEBUG:boto:Data: 2024-03-04T17:40:45.472 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.472 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.472 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.472 DEBUG:boto:Params: {} 2024-03-04T17:40:45.472 DEBUG:boto:Token: None 2024-03-04T17:40:45.472 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.472 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.476 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cab407ce380c0bf4-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.476 DEBUG:boto:path=/ppfahn-17/obj13 2024-03-04T17:40:45.476 DEBUG:boto:auth_path=/ppfahn-17/obj13 2024-03-04T17:40:45.476 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.477 DEBUG:boto:Path: /ppfahn-17/obj13 2024-03-04T17:40:45.477 DEBUG:boto:Data: 2024-03-04T17:40:45.477 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.477 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.477 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.477 DEBUG:boto:Params: {} 2024-03-04T17:40:45.477 DEBUG:boto:Token: None 2024-03-04T17:40:45.477 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj13 2024-03-04T17:40:45.477 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5l38Nx3G/u/jTy85Q2ArhlFuJPA= 2024-03-04T17:40:45.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5l38Nx3G/u/jTy85Q2ArhlFuJPA='} 2024-03-04T17:40:45.520 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098c3a0d5c3dc6819-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.520 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.520 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.521 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.521 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.521 DEBUG:boto:Data: 2024-03-04T17:40:45.521 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.521 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.521 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.521 DEBUG:boto:Params: {} 2024-03-04T17:40:45.521 DEBUG:boto:Token: None 2024-03-04T17:40:45.521 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.521 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db953d5fd78f4f79-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.526 DEBUG:boto:path=/ppfahn-17/obj14 2024-03-04T17:40:45.526 DEBUG:boto:auth_path=/ppfahn-17/obj14 2024-03-04T17:40:45.526 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.526 DEBUG:boto:Path: /ppfahn-17/obj14 2024-03-04T17:40:45.526 DEBUG:boto:Data: 2024-03-04T17:40:45.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.526 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.526 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.527 DEBUG:boto:Params: {} 2024-03-04T17:40:45.527 DEBUG:boto:Token: None 2024-03-04T17:40:45.527 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj14 2024-03-04T17:40:45.527 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VhDsbFUjKxXfBsx3Mo3WInHMwY= 2024-03-04T17:40:45.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VhDsbFUjKxXfBsx3Mo3WInHMwY='} 2024-03-04T17:40:45.535 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3259b56875eef73-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.535 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.535 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.535 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.536 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.536 DEBUG:boto:Data: 2024-03-04T17:40:45.536 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.536 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.536 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.536 DEBUG:boto:Params: {} 2024-03-04T17:40:45.536 DEBUG:boto:Token: None 2024-03-04T17:40:45.536 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.536 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.541 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000395507de159b9e46-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.541 DEBUG:boto:path=/ppfahn-17/obj15 2024-03-04T17:40:45.541 DEBUG:boto:auth_path=/ppfahn-17/obj15 2024-03-04T17:40:45.541 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.541 DEBUG:boto:Path: /ppfahn-17/obj15 2024-03-04T17:40:45.541 DEBUG:boto:Data: 2024-03-04T17:40:45.541 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.541 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.541 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.542 DEBUG:boto:Params: {} 2024-03-04T17:40:45.542 DEBUG:boto:Token: None 2024-03-04T17:40:45.542 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj15 2024-03-04T17:40:45.542 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Cc6IsTmRLJd61TMTe2zOscRKrys= 2024-03-04T17:40:45.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Cc6IsTmRLJd61TMTe2zOscRKrys='} 2024-03-04T17:40:45.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000475518baaa41e52c-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.584 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.584 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.585 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.585 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.585 DEBUG:boto:Data: 2024-03-04T17:40:45.585 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.585 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.585 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.585 DEBUG:boto:Params: {} 2024-03-04T17:40:45.585 DEBUG:boto:Token: None 2024-03-04T17:40:45.585 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.585 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.590 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f33a5baa2bca48c3-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.590 DEBUG:boto:path=/ppfahn-17/obj16 2024-03-04T17:40:45.590 DEBUG:boto:auth_path=/ppfahn-17/obj16 2024-03-04T17:40:45.590 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.591 DEBUG:boto:Path: /ppfahn-17/obj16 2024-03-04T17:40:45.591 DEBUG:boto:Data: 2024-03-04T17:40:45.591 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.591 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.591 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.591 DEBUG:boto:Params: {} 2024-03-04T17:40:45.591 DEBUG:boto:Token: None 2024-03-04T17:40:45.591 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj16 2024-03-04T17:40:45.591 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MG3QM8vhvxmO+VMnpaEa/p/JNlg= 2024-03-04T17:40:45.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MG3QM8vhvxmO+VMnpaEa/p/JNlg='} 2024-03-04T17:40:45.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009440d29c86855a59-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.603 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.603 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.604 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.604 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.604 DEBUG:boto:Data: 2024-03-04T17:40:45.604 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.604 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.604 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.604 DEBUG:boto:Params: {} 2024-03-04T17:40:45.604 DEBUG:boto:Token: None 2024-03-04T17:40:45.604 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.604 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004dcb5b0a8c6006ed-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.610 DEBUG:boto:path=/ppfahn-17/obj17 2024-03-04T17:40:45.610 DEBUG:boto:auth_path=/ppfahn-17/obj17 2024-03-04T17:40:45.610 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.610 DEBUG:boto:Path: /ppfahn-17/obj17 2024-03-04T17:40:45.610 DEBUG:boto:Data: 2024-03-04T17:40:45.610 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.610 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.610 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.611 DEBUG:boto:Params: {} 2024-03-04T17:40:45.611 DEBUG:boto:Token: None 2024-03-04T17:40:45.611 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj17 2024-03-04T17:40:45.611 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:adKFllKXRZ5qCOL0eaFzg60hMgA= 2024-03-04T17:40:45.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:adKFllKXRZ5qCOL0eaFzg60hMgA='} 2024-03-04T17:40:45.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000032ab22713818c094-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.656 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.656 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.656 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.657 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.657 DEBUG:boto:Data: 2024-03-04T17:40:45.657 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.657 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.657 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.657 DEBUG:boto:Params: {} 2024-03-04T17:40:45.657 DEBUG:boto:Token: None 2024-03-04T17:40:45.657 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.657 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.661 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016a40ca7fde5f96f-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.662 DEBUG:boto:path=/ppfahn-17/obj18 2024-03-04T17:40:45.662 DEBUG:boto:auth_path=/ppfahn-17/obj18 2024-03-04T17:40:45.662 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.662 DEBUG:boto:Path: /ppfahn-17/obj18 2024-03-04T17:40:45.662 DEBUG:boto:Data: 2024-03-04T17:40:45.662 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.662 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.662 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.662 DEBUG:boto:Params: {} 2024-03-04T17:40:45.662 DEBUG:boto:Token: None 2024-03-04T17:40:45.663 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj18 2024-03-04T17:40:45.663 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ftJN7bclqZ2QLhisphT8nvEIBX0= 2024-03-04T17:40:45.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ftJN7bclqZ2QLhisphT8nvEIBX0='} 2024-03-04T17:40:45.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000350567b117c847-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.671 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.671 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.671 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.672 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.672 DEBUG:boto:Data: 2024-03-04T17:40:45.672 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.672 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.672 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.672 DEBUG:boto:Params: {} 2024-03-04T17:40:45.672 DEBUG:boto:Token: None 2024-03-04T17:40:45.672 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.672 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.677 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f215bd851a620ba-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.677 DEBUG:boto:path=/ppfahn-17/obj19 2024-03-04T17:40:45.677 DEBUG:boto:auth_path=/ppfahn-17/obj19 2024-03-04T17:40:45.677 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.677 DEBUG:boto:Path: /ppfahn-17/obj19 2024-03-04T17:40:45.678 DEBUG:boto:Data: 2024-03-04T17:40:45.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.678 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.678 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.678 DEBUG:boto:Params: {} 2024-03-04T17:40:45.678 DEBUG:boto:Token: None 2024-03-04T17:40:45.678 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj19 2024-03-04T17:40:45.678 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rWCbZh6lK6Xg5Tlj/MgcPJ3YuHA= 2024-03-04T17:40:45.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rWCbZh6lK6Xg5Tlj/MgcPJ3YuHA='} 2024-03-04T17:40:45.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f82c6d3078e22cc-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.728 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.729 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.729 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.729 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.729 DEBUG:boto:Data: 2024-03-04T17:40:45.729 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.729 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.729 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.729 DEBUG:boto:Params: {} 2024-03-04T17:40:45.729 DEBUG:boto:Token: None 2024-03-04T17:40:45.729 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.730 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061976106ac6a0bb9-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.734 DEBUG:boto:path=/ppfahn-17/obj20 2024-03-04T17:40:45.734 DEBUG:boto:auth_path=/ppfahn-17/obj20 2024-03-04T17:40:45.734 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.734 DEBUG:boto:Path: /ppfahn-17/obj20 2024-03-04T17:40:45.734 DEBUG:boto:Data: 2024-03-04T17:40:45.734 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.734 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.735 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.735 DEBUG:boto:Params: {} 2024-03-04T17:40:45.735 DEBUG:boto:Token: None 2024-03-04T17:40:45.735 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj20 2024-03-04T17:40:45.735 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wY89fp5GBX6ktR7Hp3IeFyW1Im8= 2024-03-04T17:40:45.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wY89fp5GBX6ktR7Hp3IeFyW1Im8='} 2024-03-04T17:40:45.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c0974a2316b9f5a-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.747 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.747 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.747 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.747 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.747 DEBUG:boto:Data: 2024-03-04T17:40:45.747 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.747 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.747 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.747 DEBUG:boto:Params: {} 2024-03-04T17:40:45.748 DEBUG:boto:Token: None 2024-03-04T17:40:45.748 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.748 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbb8cc7f54142fb1-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.752 DEBUG:boto:path=/ppfahn-17/obj21 2024-03-04T17:40:45.752 DEBUG:boto:auth_path=/ppfahn-17/obj21 2024-03-04T17:40:45.752 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.752 DEBUG:boto:Path: /ppfahn-17/obj21 2024-03-04T17:40:45.752 DEBUG:boto:Data: 2024-03-04T17:40:45.752 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.752 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.752 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.752 DEBUG:boto:Params: {} 2024-03-04T17:40:45.753 DEBUG:boto:Token: None 2024-03-04T17:40:45.753 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj21 2024-03-04T17:40:45.753 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l6wv1l406h1V1Vkrip9nHDacyiQ= 2024-03-04T17:40:45.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l6wv1l406h1V1Vkrip9nHDacyiQ='} 2024-03-04T17:40:45.800 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031ef01db3d0a9277-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.800 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.801 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.801 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.801 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.801 DEBUG:boto:Data: 2024-03-04T17:40:45.801 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.801 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.801 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.801 DEBUG:boto:Params: {} 2024-03-04T17:40:45.801 DEBUG:boto:Token: None 2024-03-04T17:40:45.801 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.802 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005cd37d8e8e63de26-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.806 DEBUG:boto:path=/ppfahn-17/obj22 2024-03-04T17:40:45.806 DEBUG:boto:auth_path=/ppfahn-17/obj22 2024-03-04T17:40:45.806 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.806 DEBUG:boto:Path: /ppfahn-17/obj22 2024-03-04T17:40:45.806 DEBUG:boto:Data: 2024-03-04T17:40:45.806 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.806 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.806 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.807 DEBUG:boto:Params: {} 2024-03-04T17:40:45.807 DEBUG:boto:Token: None 2024-03-04T17:40:45.807 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj22 2024-03-04T17:40:45.807 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xFuhqtKRRXtIMEKU2B562hoIjdY= 2024-03-04T17:40:45.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xFuhqtKRRXtIMEKU2B562hoIjdY='} 2024-03-04T17:40:45.815 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009aab2421c8f84205-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.815 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.815 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.815 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.815 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.815 DEBUG:boto:Data: 2024-03-04T17:40:45.815 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.816 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.816 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.816 DEBUG:boto:Params: {} 2024-03-04T17:40:45.816 DEBUG:boto:Token: None 2024-03-04T17:40:45.816 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.816 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1cda1ba5f36971a-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.820 DEBUG:boto:path=/ppfahn-17/obj23 2024-03-04T17:40:45.821 DEBUG:boto:auth_path=/ppfahn-17/obj23 2024-03-04T17:40:45.821 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.821 DEBUG:boto:Path: /ppfahn-17/obj23 2024-03-04T17:40:45.821 DEBUG:boto:Data: 2024-03-04T17:40:45.821 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.821 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.821 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.821 DEBUG:boto:Params: {} 2024-03-04T17:40:45.821 DEBUG:boto:Token: None 2024-03-04T17:40:45.821 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj23 2024-03-04T17:40:45.822 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qJr4lChYwcPmlvJ0RCCetCsSy5E= 2024-03-04T17:40:45.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qJr4lChYwcPmlvJ0RCCetCsSy5E='} 2024-03-04T17:40:45.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c473838de5b8c965-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.864 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.864 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.865 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.865 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.865 DEBUG:boto:Data: 2024-03-04T17:40:45.865 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.865 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.865 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.865 DEBUG:boto:Params: {} 2024-03-04T17:40:45.865 DEBUG:boto:Token: None 2024-03-04T17:40:45.865 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.865 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.870 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001fa9793e600869b1-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.870 DEBUG:boto:path=/ppfahn-17/obj24 2024-03-04T17:40:45.870 DEBUG:boto:auth_path=/ppfahn-17/obj24 2024-03-04T17:40:45.870 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.870 DEBUG:boto:Path: /ppfahn-17/obj24 2024-03-04T17:40:45.870 DEBUG:boto:Data: 2024-03-04T17:40:45.870 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.871 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.871 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.871 DEBUG:boto:Params: {} 2024-03-04T17:40:45.871 DEBUG:boto:Token: None 2024-03-04T17:40:45.871 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj24 2024-03-04T17:40:45.871 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ofKfuGZmfzQKU9h2ZHsMueKnESU= 2024-03-04T17:40:45.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ofKfuGZmfzQKU9h2ZHsMueKnESU='} 2024-03-04T17:40:45.879 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b89513b5a44a43f-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.879 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.879 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.879 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.879 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.880 DEBUG:boto:Data: 2024-03-04T17:40:45.880 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.880 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.880 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.880 DEBUG:boto:Params: {} 2024-03-04T17:40:45.880 DEBUG:boto:Token: None 2024-03-04T17:40:45.880 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.880 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.895 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000981b241e5b5673e1-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.895 DEBUG:boto:path=/ppfahn-17/obj25 2024-03-04T17:40:45.895 DEBUG:boto:auth_path=/ppfahn-17/obj25 2024-03-04T17:40:45.895 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.895 DEBUG:boto:Path: /ppfahn-17/obj25 2024-03-04T17:40:45.895 DEBUG:boto:Data: 2024-03-04T17:40:45.896 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.896 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.896 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.896 DEBUG:boto:Params: {} 2024-03-04T17:40:45.896 DEBUG:boto:Token: None 2024-03-04T17:40:45.896 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj25 2024-03-04T17:40:45.896 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3Cc04FwAcL+eFnDKqvkSZgauL50= 2024-03-04T17:40:45.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3Cc04FwAcL+eFnDKqvkSZgauL50='} 2024-03-04T17:40:45.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047bb45c010d4b9ac-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.940 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.941 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.941 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.941 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.941 DEBUG:boto:Data: 2024-03-04T17:40:45.941 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.941 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.941 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.941 DEBUG:boto:Params: {} 2024-03-04T17:40:45.941 DEBUG:boto:Token: None 2024-03-04T17:40:45.941 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.941 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.945 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c320ec9cc09bd360-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.946 DEBUG:boto:path=/ppfahn-17/obj26 2024-03-04T17:40:45.946 DEBUG:boto:auth_path=/ppfahn-17/obj26 2024-03-04T17:40:45.946 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.946 DEBUG:boto:Path: /ppfahn-17/obj26 2024-03-04T17:40:45.946 DEBUG:boto:Data: 2024-03-04T17:40:45.946 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.946 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.946 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.946 DEBUG:boto:Params: {} 2024-03-04T17:40:45.947 DEBUG:boto:Token: None 2024-03-04T17:40:45.947 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj26 2024-03-04T17:40:45.947 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/56QYhPAnmvKfxjkZpikxqsiqIA= 2024-03-04T17:40:45.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/56QYhPAnmvKfxjkZpikxqsiqIA='} 2024-03-04T17:40:45.955 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ee56401707c6055-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.955 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:45.955 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:45.955 DEBUG:boto:Method: HEAD 2024-03-04T17:40:45.955 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:45.956 DEBUG:boto:Data: 2024-03-04T17:40:45.956 DEBUG:boto:Headers: {} 2024-03-04T17:40:45.956 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.956 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.956 DEBUG:boto:Params: {} 2024-03-04T17:40:45.956 DEBUG:boto:Token: None 2024-03-04T17:40:45.956 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/ 2024-03-04T17:40:45.956 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T17:40:45.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T17:40:45.960 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eec9c8bc4810430f-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:45.960 DEBUG:boto:path=/ppfahn-17/obj27 2024-03-04T17:40:45.960 DEBUG:boto:auth_path=/ppfahn-17/obj27 2024-03-04T17:40:45.960 DEBUG:boto:Method: PUT 2024-03-04T17:40:45.960 DEBUG:boto:Path: /ppfahn-17/obj27 2024-03-04T17:40:45.961 DEBUG:boto:Data: 2024-03-04T17:40:45.961 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:45.961 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:45.961 DEBUG:boto:Port: 8000 2024-03-04T17:40:45.961 DEBUG:boto:Params: {} 2024-03-04T17:40:45.961 DEBUG:boto:Token: None 2024-03-04T17:40:45.961 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:45 GMT /ppfahn-17/obj27 2024-03-04T17:40:45.961 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lb4lbRs/vZokaRwNtxuSkIJFBEg= 2024-03-04T17:40:45.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lb4lbRs/vZokaRwNtxuSkIJFBEg='} 2024-03-04T17:40:46.008 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068aa4ee1e79a26ee-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.008 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.008 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.009 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.009 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.009 DEBUG:boto:Data: 2024-03-04T17:40:46.009 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.009 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.009 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.009 DEBUG:boto:Params: {} 2024-03-04T17:40:46.009 DEBUG:boto:Token: None 2024-03-04T17:40:46.009 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.010 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000819d21905f9ab41d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.014 DEBUG:boto:path=/ppfahn-17/obj28 2024-03-04T17:40:46.014 DEBUG:boto:auth_path=/ppfahn-17/obj28 2024-03-04T17:40:46.014 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.014 DEBUG:boto:Path: /ppfahn-17/obj28 2024-03-04T17:40:46.014 DEBUG:boto:Data: 2024-03-04T17:40:46.014 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.015 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.015 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.015 DEBUG:boto:Params: {} 2024-03-04T17:40:46.015 DEBUG:boto:Token: None 2024-03-04T17:40:46.015 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj28 2024-03-04T17:40:46.015 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:P/bSpO5QB+IyU1FbDSjF598ZqrA= 2024-03-04T17:40:46.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:P/bSpO5QB+IyU1FbDSjF598ZqrA='} 2024-03-04T17:40:46.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cc292a4014a0735-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.024 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.024 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.024 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.024 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.024 DEBUG:boto:Data: 2024-03-04T17:40:46.025 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.025 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.025 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.025 DEBUG:boto:Params: {} 2024-03-04T17:40:46.025 DEBUG:boto:Token: None 2024-03-04T17:40:46.025 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.025 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.032 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e91b402fd3bc9d78-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.033 DEBUG:boto:path=/ppfahn-17/obj29 2024-03-04T17:40:46.033 DEBUG:boto:auth_path=/ppfahn-17/obj29 2024-03-04T17:40:46.033 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.033 DEBUG:boto:Path: /ppfahn-17/obj29 2024-03-04T17:40:46.033 DEBUG:boto:Data: 2024-03-04T17:40:46.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.033 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.033 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.033 DEBUG:boto:Params: {} 2024-03-04T17:40:46.033 DEBUG:boto:Token: None 2024-03-04T17:40:46.034 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj29 2024-03-04T17:40:46.034 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9QDMa4H4+sIu37Mt3gjqwPOBy2c= 2024-03-04T17:40:46.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9QDMa4H4+sIu37Mt3gjqwPOBy2c='} 2024-03-04T17:40:46.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cbea81a5f22864b-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.076 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.076 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.077 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.077 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.077 DEBUG:boto:Data: 2024-03-04T17:40:46.077 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.077 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.077 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.077 DEBUG:boto:Params: {} 2024-03-04T17:40:46.077 DEBUG:boto:Token: None 2024-03-04T17:40:46.077 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.077 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.081 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000506b5060a927f67a-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.082 DEBUG:boto:path=/ppfahn-17/obj30 2024-03-04T17:40:46.082 DEBUG:boto:auth_path=/ppfahn-17/obj30 2024-03-04T17:40:46.082 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.082 DEBUG:boto:Path: /ppfahn-17/obj30 2024-03-04T17:40:46.082 DEBUG:boto:Data: 2024-03-04T17:40:46.082 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.082 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.082 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.082 DEBUG:boto:Params: {} 2024-03-04T17:40:46.082 DEBUG:boto:Token: None 2024-03-04T17:40:46.083 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj30 2024-03-04T17:40:46.083 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vmRz5ivJZSmL4j7+xdssSAkVzyk= 2024-03-04T17:40:46.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vmRz5ivJZSmL4j7+xdssSAkVzyk='} 2024-03-04T17:40:46.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003652d86d18b31186-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.091 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.091 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.091 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.091 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.092 DEBUG:boto:Data: 2024-03-04T17:40:46.092 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.092 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.092 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.092 DEBUG:boto:Params: {} 2024-03-04T17:40:46.092 DEBUG:boto:Token: None 2024-03-04T17:40:46.092 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.092 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.097 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000805106a68dd8829-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.097 DEBUG:boto:path=/ppfahn-17/obj31 2024-03-04T17:40:46.097 DEBUG:boto:auth_path=/ppfahn-17/obj31 2024-03-04T17:40:46.097 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.097 DEBUG:boto:Path: /ppfahn-17/obj31 2024-03-04T17:40:46.098 DEBUG:boto:Data: 2024-03-04T17:40:46.098 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.098 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.098 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.098 DEBUG:boto:Params: {} 2024-03-04T17:40:46.098 DEBUG:boto:Token: None 2024-03-04T17:40:46.098 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj31 2024-03-04T17:40:46.098 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MikwRb715H1rZixXy5CX+wVxbvk= 2024-03-04T17:40:46.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MikwRb715H1rZixXy5CX+wVxbvk='} 2024-03-04T17:40:46.144 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e792898aece2343-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.145 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.145 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.145 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.145 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.145 DEBUG:boto:Data: 2024-03-04T17:40:46.145 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.145 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.145 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.145 DEBUG:boto:Params: {} 2024-03-04T17:40:46.145 DEBUG:boto:Token: None 2024-03-04T17:40:46.146 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.146 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4336492e356181d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.150 DEBUG:boto:path=/ppfahn-17/obj32 2024-03-04T17:40:46.150 DEBUG:boto:auth_path=/ppfahn-17/obj32 2024-03-04T17:40:46.150 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.150 DEBUG:boto:Path: /ppfahn-17/obj32 2024-03-04T17:40:46.151 DEBUG:boto:Data: 2024-03-04T17:40:46.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.151 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.151 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.151 DEBUG:boto:Params: {} 2024-03-04T17:40:46.151 DEBUG:boto:Token: None 2024-03-04T17:40:46.151 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj32 2024-03-04T17:40:46.151 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qVUMhBkz8fKD1R1+DPok6bCvn/Q= 2024-03-04T17:40:46.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVUMhBkz8fKD1R1+DPok6bCvn/Q='} 2024-03-04T17:40:46.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa71728a726bce8f-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.160 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.160 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.161 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.161 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.161 DEBUG:boto:Data: 2024-03-04T17:40:46.161 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.161 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.161 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.161 DEBUG:boto:Params: {} 2024-03-04T17:40:46.161 DEBUG:boto:Token: None 2024-03-04T17:40:46.161 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.162 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f78c0791365d749d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.166 DEBUG:boto:path=/ppfahn-17/obj33 2024-03-04T17:40:46.166 DEBUG:boto:auth_path=/ppfahn-17/obj33 2024-03-04T17:40:46.166 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.166 DEBUG:boto:Path: /ppfahn-17/obj33 2024-03-04T17:40:46.166 DEBUG:boto:Data: 2024-03-04T17:40:46.166 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.166 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.166 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.166 DEBUG:boto:Params: {} 2024-03-04T17:40:46.167 DEBUG:boto:Token: None 2024-03-04T17:40:46.167 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj33 2024-03-04T17:40:46.167 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BebK0VpUAesGjabl17H5vnZTlRw= 2024-03-04T17:40:46.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BebK0VpUAesGjabl17H5vnZTlRw='} 2024-03-04T17:40:46.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b9585fb74c895cd9-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.212 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.212 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.213 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.213 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.213 DEBUG:boto:Data: 2024-03-04T17:40:46.213 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.213 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.213 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.213 DEBUG:boto:Params: {} 2024-03-04T17:40:46.213 DEBUG:boto:Token: None 2024-03-04T17:40:46.213 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.213 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.217 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e9b883d0fad4bbc-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.218 DEBUG:boto:path=/ppfahn-17/obj34 2024-03-04T17:40:46.218 DEBUG:boto:auth_path=/ppfahn-17/obj34 2024-03-04T17:40:46.218 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.218 DEBUG:boto:Path: /ppfahn-17/obj34 2024-03-04T17:40:46.218 DEBUG:boto:Data: 2024-03-04T17:40:46.218 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.218 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.218 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.218 DEBUG:boto:Params: {} 2024-03-04T17:40:46.219 DEBUG:boto:Token: None 2024-03-04T17:40:46.219 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj34 2024-03-04T17:40:46.219 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D8h4QSWhQVhGEbCqxD64QS8ZVck= 2024-03-04T17:40:46.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D8h4QSWhQVhGEbCqxD64QS8ZVck='} 2024-03-04T17:40:46.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000607601cec50adc60-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.227 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.228 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.228 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.228 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.228 DEBUG:boto:Data: 2024-03-04T17:40:46.228 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.228 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.228 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.228 DEBUG:boto:Params: {} 2024-03-04T17:40:46.228 DEBUG:boto:Token: None 2024-03-04T17:40:46.228 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.228 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d86ce4b0488629d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.232 DEBUG:boto:path=/ppfahn-17/obj35 2024-03-04T17:40:46.233 DEBUG:boto:auth_path=/ppfahn-17/obj35 2024-03-04T17:40:46.233 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.233 DEBUG:boto:Path: /ppfahn-17/obj35 2024-03-04T17:40:46.233 DEBUG:boto:Data: 2024-03-04T17:40:46.233 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.233 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.233 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.233 DEBUG:boto:Params: {} 2024-03-04T17:40:46.233 DEBUG:boto:Token: None 2024-03-04T17:40:46.234 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj35 2024-03-04T17:40:46.234 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qXx7mdgy9thi8HKDW3IVOXQnkWg= 2024-03-04T17:40:46.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qXx7mdgy9thi8HKDW3IVOXQnkWg='} 2024-03-04T17:40:46.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9898341ffee892c-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.276 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.276 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.277 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.277 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.277 DEBUG:boto:Data: 2024-03-04T17:40:46.277 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.277 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.277 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.277 DEBUG:boto:Params: {} 2024-03-04T17:40:46.277 DEBUG:boto:Token: None 2024-03-04T17:40:46.277 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.277 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.282 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de5a573780c2111b-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.282 DEBUG:boto:path=/ppfahn-17/obj36 2024-03-04T17:40:46.282 DEBUG:boto:auth_path=/ppfahn-17/obj36 2024-03-04T17:40:46.282 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.282 DEBUG:boto:Path: /ppfahn-17/obj36 2024-03-04T17:40:46.282 DEBUG:boto:Data: 2024-03-04T17:40:46.282 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.282 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.283 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.283 DEBUG:boto:Params: {} 2024-03-04T17:40:46.283 DEBUG:boto:Token: None 2024-03-04T17:40:46.283 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj36 2024-03-04T17:40:46.283 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oMujIWuffW5DXzRciETW+jxKn+0= 2024-03-04T17:40:46.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oMujIWuffW5DXzRciETW+jxKn+0='} 2024-03-04T17:40:46.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057fdd23bd1f4da32-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.291 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.292 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.292 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.292 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.292 DEBUG:boto:Data: 2024-03-04T17:40:46.292 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.292 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.292 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.292 DEBUG:boto:Params: {} 2024-03-04T17:40:46.292 DEBUG:boto:Token: None 2024-03-04T17:40:46.293 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.293 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.297 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d456efd45dde7b38-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.297 DEBUG:boto:path=/ppfahn-17/obj37 2024-03-04T17:40:46.297 DEBUG:boto:auth_path=/ppfahn-17/obj37 2024-03-04T17:40:46.297 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.297 DEBUG:boto:Path: /ppfahn-17/obj37 2024-03-04T17:40:46.297 DEBUG:boto:Data: 2024-03-04T17:40:46.297 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.298 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.298 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.298 DEBUG:boto:Params: {} 2024-03-04T17:40:46.298 DEBUG:boto:Token: None 2024-03-04T17:40:46.298 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj37 2024-03-04T17:40:46.298 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CCapDQ7R5YuC3D22A7Zt3EoAkkE= 2024-03-04T17:40:46.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CCapDQ7R5YuC3D22A7Zt3EoAkkE='} 2024-03-04T17:40:46.344 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5afdf9884447322-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.344 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.344 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.345 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.345 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.345 DEBUG:boto:Data: 2024-03-04T17:40:46.345 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.345 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.345 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.345 DEBUG:boto:Params: {} 2024-03-04T17:40:46.345 DEBUG:boto:Token: None 2024-03-04T17:40:46.345 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.345 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.349 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000decc226cdca32f87-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.350 DEBUG:boto:path=/ppfahn-17/obj38 2024-03-04T17:40:46.350 DEBUG:boto:auth_path=/ppfahn-17/obj38 2024-03-04T17:40:46.350 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.350 DEBUG:boto:Path: /ppfahn-17/obj38 2024-03-04T17:40:46.350 DEBUG:boto:Data: 2024-03-04T17:40:46.350 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.350 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.350 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.350 DEBUG:boto:Params: {} 2024-03-04T17:40:46.351 DEBUG:boto:Token: None 2024-03-04T17:40:46.351 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj38 2024-03-04T17:40:46.351 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2PPCERf5TiOLFCQqKsY7swLppZM= 2024-03-04T17:40:46.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2PPCERf5TiOLFCQqKsY7swLppZM='} 2024-03-04T17:40:46.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005639fdb399978f6d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.359 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.359 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.360 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.360 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.360 DEBUG:boto:Data: 2024-03-04T17:40:46.360 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.360 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.360 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.360 DEBUG:boto:Params: {} 2024-03-04T17:40:46.360 DEBUG:boto:Token: None 2024-03-04T17:40:46.360 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.361 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.365 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fdbd2235214a3be-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.365 DEBUG:boto:path=/ppfahn-17/obj39 2024-03-04T17:40:46.365 DEBUG:boto:auth_path=/ppfahn-17/obj39 2024-03-04T17:40:46.365 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.365 DEBUG:boto:Path: /ppfahn-17/obj39 2024-03-04T17:40:46.366 DEBUG:boto:Data: 2024-03-04T17:40:46.366 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.366 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.366 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.366 DEBUG:boto:Params: {} 2024-03-04T17:40:46.366 DEBUG:boto:Token: None 2024-03-04T17:40:46.366 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj39 2024-03-04T17:40:46.366 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p+pu7W3miBh2BIsEwrdf8qNIB0s= 2024-03-04T17:40:46.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p+pu7W3miBh2BIsEwrdf8qNIB0s='} 2024-03-04T17:40:46.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c1e4e0122adaf6d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.408 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.409 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.409 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.409 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.409 DEBUG:boto:Data: 2024-03-04T17:40:46.409 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.409 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.409 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.409 DEBUG:boto:Params: {} 2024-03-04T17:40:46.409 DEBUG:boto:Token: None 2024-03-04T17:40:46.410 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.410 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.414 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b06dadc81fa455f8-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.414 DEBUG:boto:path=/ppfahn-17/obj40 2024-03-04T17:40:46.414 DEBUG:boto:auth_path=/ppfahn-17/obj40 2024-03-04T17:40:46.414 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.414 DEBUG:boto:Path: /ppfahn-17/obj40 2024-03-04T17:40:46.414 DEBUG:boto:Data: 2024-03-04T17:40:46.414 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.415 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.415 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.415 DEBUG:boto:Params: {} 2024-03-04T17:40:46.415 DEBUG:boto:Token: None 2024-03-04T17:40:46.415 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj40 2024-03-04T17:40:46.415 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T+Orqd0GYxd4JomYHABUyozPpyo= 2024-03-04T17:40:46.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T+Orqd0GYxd4JomYHABUyozPpyo='} 2024-03-04T17:40:46.424 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e47a17a30566fa9c-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.424 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.424 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.424 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.424 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.424 DEBUG:boto:Data: 2024-03-04T17:40:46.424 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.425 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.425 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.425 DEBUG:boto:Params: {} 2024-03-04T17:40:46.425 DEBUG:boto:Token: None 2024-03-04T17:40:46.425 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.425 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.429 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005232ebdd516be170-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.430 DEBUG:boto:path=/ppfahn-17/obj41 2024-03-04T17:40:46.430 DEBUG:boto:auth_path=/ppfahn-17/obj41 2024-03-04T17:40:46.430 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.430 DEBUG:boto:Path: /ppfahn-17/obj41 2024-03-04T17:40:46.430 DEBUG:boto:Data: 2024-03-04T17:40:46.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.430 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.430 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.430 DEBUG:boto:Params: {} 2024-03-04T17:40:46.431 DEBUG:boto:Token: None 2024-03-04T17:40:46.431 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj41 2024-03-04T17:40:46.431 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8rMN8WbSJkBIMmifYmFa4omD8ig= 2024-03-04T17:40:46.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8rMN8WbSJkBIMmifYmFa4omD8ig='} 2024-03-04T17:40:46.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000aa8663fc100596f-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.476 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.477 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.477 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.477 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.477 DEBUG:boto:Data: 2024-03-04T17:40:46.477 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.477 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.477 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.477 DEBUG:boto:Params: {} 2024-03-04T17:40:46.477 DEBUG:boto:Token: None 2024-03-04T17:40:46.477 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.477 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000283f6621939756c3-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.482 DEBUG:boto:path=/ppfahn-17/obj42 2024-03-04T17:40:46.482 DEBUG:boto:auth_path=/ppfahn-17/obj42 2024-03-04T17:40:46.482 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.482 DEBUG:boto:Path: /ppfahn-17/obj42 2024-03-04T17:40:46.482 DEBUG:boto:Data: 2024-03-04T17:40:46.482 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.482 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.483 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.483 DEBUG:boto:Params: {} 2024-03-04T17:40:46.483 DEBUG:boto:Token: None 2024-03-04T17:40:46.483 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj42 2024-03-04T17:40:46.483 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+st7WxInpo7TXAGR4vRWhxceaUE= 2024-03-04T17:40:46.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+st7WxInpo7TXAGR4vRWhxceaUE='} 2024-03-04T17:40:46.491 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3b402e08f847be9-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.491 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.491 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.491 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.492 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.492 DEBUG:boto:Data: 2024-03-04T17:40:46.492 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.492 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.492 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.492 DEBUG:boto:Params: {} 2024-03-04T17:40:46.492 DEBUG:boto:Token: None 2024-03-04T17:40:46.492 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.492 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab239f6932e18c15-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.496 DEBUG:boto:path=/ppfahn-17/obj43 2024-03-04T17:40:46.496 DEBUG:boto:auth_path=/ppfahn-17/obj43 2024-03-04T17:40:46.497 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.497 DEBUG:boto:Path: /ppfahn-17/obj43 2024-03-04T17:40:46.497 DEBUG:boto:Data: 2024-03-04T17:40:46.497 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.497 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.497 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.497 DEBUG:boto:Params: {} 2024-03-04T17:40:46.497 DEBUG:boto:Token: None 2024-03-04T17:40:46.497 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj43 2024-03-04T17:40:46.497 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:71WCl/b9KA7XM6dL+0TrE59i3yQ= 2024-03-04T17:40:46.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:71WCl/b9KA7XM6dL+0TrE59i3yQ='} 2024-03-04T17:40:46.540 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081292f3b69152814-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.540 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.540 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.541 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.541 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.541 DEBUG:boto:Data: 2024-03-04T17:40:46.541 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.541 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.541 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.541 DEBUG:boto:Params: {} 2024-03-04T17:40:46.541 DEBUG:boto:Token: None 2024-03-04T17:40:46.541 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.541 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033d6ea807620431d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.545 DEBUG:boto:path=/ppfahn-17/obj44 2024-03-04T17:40:46.546 DEBUG:boto:auth_path=/ppfahn-17/obj44 2024-03-04T17:40:46.546 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.546 DEBUG:boto:Path: /ppfahn-17/obj44 2024-03-04T17:40:46.546 DEBUG:boto:Data: 2024-03-04T17:40:46.546 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.546 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.546 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.546 DEBUG:boto:Params: {} 2024-03-04T17:40:46.546 DEBUG:boto:Token: None 2024-03-04T17:40:46.546 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj44 2024-03-04T17:40:46.547 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CogpU7Py9ZE9IS0BR/aN3TNJfWk= 2024-03-04T17:40:46.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CogpU7Py9ZE9IS0BR/aN3TNJfWk='} 2024-03-04T17:40:46.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000369350ae28df5938-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.558 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.558 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.558 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.558 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.558 DEBUG:boto:Data: 2024-03-04T17:40:46.559 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.559 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.559 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.559 DEBUG:boto:Params: {} 2024-03-04T17:40:46.559 DEBUG:boto:Token: None 2024-03-04T17:40:46.559 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.559 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055c13632a896e0a7-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.564 DEBUG:boto:path=/ppfahn-17/obj45 2024-03-04T17:40:46.564 DEBUG:boto:auth_path=/ppfahn-17/obj45 2024-03-04T17:40:46.564 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.564 DEBUG:boto:Path: /ppfahn-17/obj45 2024-03-04T17:40:46.564 DEBUG:boto:Data: 2024-03-04T17:40:46.564 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.564 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.565 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.565 DEBUG:boto:Params: {} 2024-03-04T17:40:46.565 DEBUG:boto:Token: None 2024-03-04T17:40:46.565 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj45 2024-03-04T17:40:46.565 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:N+ezKFWyC1G/Jz19uyUgMgX0ikg= 2024-03-04T17:40:46.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:N+ezKFWyC1G/Jz19uyUgMgX0ikg='} 2024-03-04T17:40:46.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a38953808b6f9994-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.612 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.612 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.612 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.613 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.613 DEBUG:boto:Data: 2024-03-04T17:40:46.613 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.613 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.613 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.613 DEBUG:boto:Params: {} 2024-03-04T17:40:46.613 DEBUG:boto:Token: None 2024-03-04T17:40:46.613 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.613 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3a00d1452f91175-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.617 DEBUG:boto:path=/ppfahn-17/obj46 2024-03-04T17:40:46.618 DEBUG:boto:auth_path=/ppfahn-17/obj46 2024-03-04T17:40:46.618 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.618 DEBUG:boto:Path: /ppfahn-17/obj46 2024-03-04T17:40:46.618 DEBUG:boto:Data: 2024-03-04T17:40:46.618 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.618 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.618 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.618 DEBUG:boto:Params: {} 2024-03-04T17:40:46.618 DEBUG:boto:Token: None 2024-03-04T17:40:46.619 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj46 2024-03-04T17:40:46.619 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qRXLPLxbFyJphpxDBrrGeb+7Yfg= 2024-03-04T17:40:46.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qRXLPLxbFyJphpxDBrrGeb+7Yfg='} 2024-03-04T17:40:46.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c96630a68272a2b3-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.627 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.628 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.628 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.628 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.628 DEBUG:boto:Data: 2024-03-04T17:40:46.628 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.628 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.628 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.628 DEBUG:boto:Params: {} 2024-03-04T17:40:46.628 DEBUG:boto:Token: None 2024-03-04T17:40:46.629 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.629 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000973ad9895456a4e9-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.633 DEBUG:boto:path=/ppfahn-17/obj47 2024-03-04T17:40:46.633 DEBUG:boto:auth_path=/ppfahn-17/obj47 2024-03-04T17:40:46.633 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.633 DEBUG:boto:Path: /ppfahn-17/obj47 2024-03-04T17:40:46.633 DEBUG:boto:Data: 2024-03-04T17:40:46.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.634 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.634 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.634 DEBUG:boto:Params: {} 2024-03-04T17:40:46.634 DEBUG:boto:Token: None 2024-03-04T17:40:46.634 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj47 2024-03-04T17:40:46.634 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:szAHp1unAp3AdPyxu5zGqxfkDoE= 2024-03-04T17:40:46.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:szAHp1unAp3AdPyxu5zGqxfkDoE='} 2024-03-04T17:40:46.676 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031b0810b2f0ea471-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.676 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.676 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.677 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.677 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.677 DEBUG:boto:Data: 2024-03-04T17:40:46.677 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.677 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.677 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.677 DEBUG:boto:Params: {} 2024-03-04T17:40:46.677 DEBUG:boto:Token: None 2024-03-04T17:40:46.677 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.677 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b62aa6767b99cc0-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.682 DEBUG:boto:path=/ppfahn-17/obj48 2024-03-04T17:40:46.682 DEBUG:boto:auth_path=/ppfahn-17/obj48 2024-03-04T17:40:46.683 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.683 DEBUG:boto:Path: /ppfahn-17/obj48 2024-03-04T17:40:46.683 DEBUG:boto:Data: 2024-03-04T17:40:46.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.683 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.683 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.683 DEBUG:boto:Params: {} 2024-03-04T17:40:46.683 DEBUG:boto:Token: None 2024-03-04T17:40:46.683 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj48 2024-03-04T17:40:46.684 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m1zKkweucbcEs/ygMm0rpLb00ws= 2024-03-04T17:40:46.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m1zKkweucbcEs/ygMm0rpLb00ws='} 2024-03-04T17:40:46.692 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9b05bd30bfae35c-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.692 DEBUG:boto:path=/ppfahn-17/ 2024-03-04T17:40:46.692 DEBUG:boto:auth_path=/ppfahn-17/ 2024-03-04T17:40:46.692 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.692 DEBUG:boto:Path: /ppfahn-17/ 2024-03-04T17:40:46.693 DEBUG:boto:Data: 2024-03-04T17:40:46.693 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.693 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.693 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.693 DEBUG:boto:Params: {} 2024-03-04T17:40:46.693 DEBUG:boto:Token: None 2024-03-04T17:40:46.693 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/ 2024-03-04T17:40:46.693 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T17:40:46.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T17:40:46.697 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f57c772f26fb4e86-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.697 DEBUG:boto:path=/ppfahn-17/obj49 2024-03-04T17:40:46.698 DEBUG:boto:auth_path=/ppfahn-17/obj49 2024-03-04T17:40:46.698 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.698 DEBUG:boto:Path: /ppfahn-17/obj49 2024-03-04T17:40:46.698 DEBUG:boto:Data: 2024-03-04T17:40:46.698 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.698 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:40:46.698 DEBUG:boto:Port: 8000 2024-03-04T17:40:46.698 DEBUG:boto:Params: {} 2024-03-04T17:40:46.698 DEBUG:boto:Token: None 2024-03-04T17:40:46.699 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-17/obj49 2024-03-04T17:40:46.699 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ALkrktUupXASaUtKYNpG0F7p4QE= 2024-03-04T17:40:46.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ALkrktUupXASaUtKYNpG0F7p4QE='} 2024-03-04T17:40:46.744 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000921874273e842b51-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.744 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:46.744 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:46.744 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.745 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:46.745 DEBUG:boto:Data: 2024-03-04T17:40:46.745 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.745 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.745 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.745 DEBUG:boto:Params: {} 2024-03-04T17:40:46.745 DEBUG:boto:Token: None 2024-03-04T17:40:46.745 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/ 2024-03-04T17:40:46.745 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T17:40:46.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T17:40:46.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a72310fb41dcbd0-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.750 DEBUG:boto:path=/ppfahn-18/obj1 2024-03-04T17:40:46.750 DEBUG:boto:auth_path=/ppfahn-18/obj1 2024-03-04T17:40:46.750 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.750 DEBUG:boto:Path: /ppfahn-18/obj1 2024-03-04T17:40:46.751 DEBUG:boto:Data: 2024-03-04T17:40:46.751 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.751 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.751 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.751 DEBUG:boto:Params: {} 2024-03-04T17:40:46.751 DEBUG:boto:Token: None 2024-03-04T17:40:46.751 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/obj1 2024-03-04T17:40:46.751 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tXXvBfu8xCaHPRuA0kLcEoID3n0= 2024-03-04T17:40:46.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tXXvBfu8xCaHPRuA0kLcEoID3n0='} 2024-03-04T17:40:46.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce81c5b799b371da-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.765 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:46.765 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:46.765 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.765 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:46.765 DEBUG:boto:Data: 2024-03-04T17:40:46.766 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.766 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.766 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.766 DEBUG:boto:Params: {} 2024-03-04T17:40:46.766 DEBUG:boto:Token: None 2024-03-04T17:40:46.766 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/ 2024-03-04T17:40:46.766 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T17:40:46.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T17:40:46.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e75a0cac7487aece-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.770 DEBUG:boto:path=/ppfahn-18/obj2 2024-03-04T17:40:46.770 DEBUG:boto:auth_path=/ppfahn-18/obj2 2024-03-04T17:40:46.771 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.771 DEBUG:boto:Path: /ppfahn-18/obj2 2024-03-04T17:40:46.771 DEBUG:boto:Data: 2024-03-04T17:40:46.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.771 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.771 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.771 DEBUG:boto:Params: {} 2024-03-04T17:40:46.771 DEBUG:boto:Token: None 2024-03-04T17:40:46.771 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/obj2 2024-03-04T17:40:46.771 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eHZF7PjUcl5Cljs6VenFAAlyMeM= 2024-03-04T17:40:46.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eHZF7PjUcl5Cljs6VenFAAlyMeM='} 2024-03-04T17:40:46.816 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2e4f46c9f4595ac-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.816 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:46.816 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:46.817 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.817 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:46.817 DEBUG:boto:Data: 2024-03-04T17:40:46.817 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.817 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.817 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.817 DEBUG:boto:Params: {} 2024-03-04T17:40:46.817 DEBUG:boto:Token: None 2024-03-04T17:40:46.817 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/ 2024-03-04T17:40:46.817 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T17:40:46.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T17:40:46.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d56b51579bc111c-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.822 DEBUG:boto:path=/ppfahn-18/obj3 2024-03-04T17:40:46.822 DEBUG:boto:auth_path=/ppfahn-18/obj3 2024-03-04T17:40:46.822 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.822 DEBUG:boto:Path: /ppfahn-18/obj3 2024-03-04T17:40:46.822 DEBUG:boto:Data: 2024-03-04T17:40:46.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.822 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.822 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.822 DEBUG:boto:Params: {} 2024-03-04T17:40:46.823 DEBUG:boto:Token: None 2024-03-04T17:40:46.823 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/obj3 2024-03-04T17:40:46.823 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QbpJ4/kruIpmTbxdVcxxIsvrIhQ= 2024-03-04T17:40:46.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QbpJ4/kruIpmTbxdVcxxIsvrIhQ='} 2024-03-04T17:40:46.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f7da9cba83843fc-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.834 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:46.834 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:46.834 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.835 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:46.835 DEBUG:boto:Data: 2024-03-04T17:40:46.835 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.835 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.835 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.835 DEBUG:boto:Params: {} 2024-03-04T17:40:46.835 DEBUG:boto:Token: None 2024-03-04T17:40:46.835 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/ 2024-03-04T17:40:46.835 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T17:40:46.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T17:40:46.839 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9833eea2925af4c-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.840 DEBUG:boto:path=/ppfahn-18/obj4 2024-03-04T17:40:46.840 DEBUG:boto:auth_path=/ppfahn-18/obj4 2024-03-04T17:40:46.840 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.840 DEBUG:boto:Path: /ppfahn-18/obj4 2024-03-04T17:40:46.840 DEBUG:boto:Data: 2024-03-04T17:40:46.840 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.840 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.840 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.841 DEBUG:boto:Params: {} 2024-03-04T17:40:46.841 DEBUG:boto:Token: None 2024-03-04T17:40:46.841 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/obj4 2024-03-04T17:40:46.841 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NtXF9AOi9P86UBPvnSdkuKdn3Dg= 2024-03-04T17:40:46.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NtXF9AOi9P86UBPvnSdkuKdn3Dg='} 2024-03-04T17:40:46.884 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f16955d2af61501b-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.884 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:46.884 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:46.885 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.885 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:46.885 DEBUG:boto:Data: 2024-03-04T17:40:46.885 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.885 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.885 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.885 DEBUG:boto:Params: {} 2024-03-04T17:40:46.885 DEBUG:boto:Token: None 2024-03-04T17:40:46.885 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/ 2024-03-04T17:40:46.885 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T17:40:46.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T17:40:46.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020379bdec4d68e28-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.890 DEBUG:boto:path=/ppfahn-18/obj5 2024-03-04T17:40:46.890 DEBUG:boto:auth_path=/ppfahn-18/obj5 2024-03-04T17:40:46.890 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.890 DEBUG:boto:Path: /ppfahn-18/obj5 2024-03-04T17:40:46.890 DEBUG:boto:Data: 2024-03-04T17:40:46.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.890 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.890 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.890 DEBUG:boto:Params: {} 2024-03-04T17:40:46.890 DEBUG:boto:Token: None 2024-03-04T17:40:46.890 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/obj5 2024-03-04T17:40:46.891 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:n3sqCDrLv7P5eNNNzc1iFCSWQqw= 2024-03-04T17:40:46.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:n3sqCDrLv7P5eNNNzc1iFCSWQqw='} 2024-03-04T17:40:46.899 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d3cebec60580091-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.899 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:46.899 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:46.899 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.899 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:46.900 DEBUG:boto:Data: 2024-03-04T17:40:46.900 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.900 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.900 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.900 DEBUG:boto:Params: {} 2024-03-04T17:40:46.900 DEBUG:boto:Token: None 2024-03-04T17:40:46.900 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/ 2024-03-04T17:40:46.900 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T17:40:46.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T17:40:46.904 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4fffd1f508a170c-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.904 DEBUG:boto:path=/ppfahn-18/obj6 2024-03-04T17:40:46.905 DEBUG:boto:auth_path=/ppfahn-18/obj6 2024-03-04T17:40:46.905 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.905 DEBUG:boto:Path: /ppfahn-18/obj6 2024-03-04T17:40:46.905 DEBUG:boto:Data: 2024-03-04T17:40:46.905 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.905 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.905 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.905 DEBUG:boto:Params: {} 2024-03-04T17:40:46.905 DEBUG:boto:Token: None 2024-03-04T17:40:46.905 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/obj6 2024-03-04T17:40:46.905 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XRNWc0Y33u90Fc/qyAxJfdgaOi0= 2024-03-04T17:40:46.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XRNWc0Y33u90Fc/qyAxJfdgaOi0='} 2024-03-04T17:40:46.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073db94e65aacbd36-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.948 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:46.948 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:46.949 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.949 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:46.949 DEBUG:boto:Data: 2024-03-04T17:40:46.949 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.949 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.949 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.949 DEBUG:boto:Params: {} 2024-03-04T17:40:46.949 DEBUG:boto:Token: None 2024-03-04T17:40:46.949 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/ 2024-03-04T17:40:46.949 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T17:40:46.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T17:40:46.954 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ceddd92152404cdd-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.954 DEBUG:boto:path=/ppfahn-18/obj7 2024-03-04T17:40:46.954 DEBUG:boto:auth_path=/ppfahn-18/obj7 2024-03-04T17:40:46.954 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.954 DEBUG:boto:Path: /ppfahn-18/obj7 2024-03-04T17:40:46.954 DEBUG:boto:Data: 2024-03-04T17:40:46.954 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.954 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.955 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.955 DEBUG:boto:Params: {} 2024-03-04T17:40:46.955 DEBUG:boto:Token: None 2024-03-04T17:40:46.955 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/obj7 2024-03-04T17:40:46.955 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nSvK9PMe02/tc0ktDkhmoIWMWrw= 2024-03-04T17:40:46.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nSvK9PMe02/tc0ktDkhmoIWMWrw='} 2024-03-04T17:40:46.963 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097ab9f90aa45ba08-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.964 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:46.964 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:46.964 DEBUG:boto:Method: HEAD 2024-03-04T17:40:46.964 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:46.964 DEBUG:boto:Data: 2024-03-04T17:40:46.964 DEBUG:boto:Headers: {} 2024-03-04T17:40:46.964 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.964 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.964 DEBUG:boto:Params: {} 2024-03-04T17:40:46.965 DEBUG:boto:Token: None 2024-03-04T17:40:46.965 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/ 2024-03-04T17:40:46.965 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T17:40:46.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T17:40:46.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000684b644cba205a2-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:46.969 DEBUG:boto:path=/ppfahn-18/obj8 2024-03-04T17:40:46.969 DEBUG:boto:auth_path=/ppfahn-18/obj8 2024-03-04T17:40:46.969 DEBUG:boto:Method: PUT 2024-03-04T17:40:46.969 DEBUG:boto:Path: /ppfahn-18/obj8 2024-03-04T17:40:46.970 DEBUG:boto:Data: 2024-03-04T17:40:46.970 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:46.970 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:46.970 DEBUG:boto:Port: 8001 2024-03-04T17:40:46.970 DEBUG:boto:Params: {} 2024-03-04T17:40:46.970 DEBUG:boto:Token: None 2024-03-04T17:40:46.970 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:46 GMT /ppfahn-18/obj8 2024-03-04T17:40:46.970 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7VnksVOus/Ex5YKJLKSjmIq80ko= 2024-03-04T17:40:46.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7VnksVOus/Ex5YKJLKSjmIq80ko='} 2024-03-04T17:40:47.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f941e02042bf3cb4-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.017 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.017 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.017 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.017 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.017 DEBUG:boto:Data: 2024-03-04T17:40:47.017 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.017 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.018 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.018 DEBUG:boto:Params: {} 2024-03-04T17:40:47.018 DEBUG:boto:Token: None 2024-03-04T17:40:47.018 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.018 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.026 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad92d99651cda83c-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.026 DEBUG:boto:path=/ppfahn-18/obj9 2024-03-04T17:40:47.026 DEBUG:boto:auth_path=/ppfahn-18/obj9 2024-03-04T17:40:47.026 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.026 DEBUG:boto:Path: /ppfahn-18/obj9 2024-03-04T17:40:47.027 DEBUG:boto:Data: 2024-03-04T17:40:47.027 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.027 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.027 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.027 DEBUG:boto:Params: {} 2024-03-04T17:40:47.027 DEBUG:boto:Token: None 2024-03-04T17:40:47.027 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj9 2024-03-04T17:40:47.027 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RySp7YRzKOe5z3gQbVfY9Ie9In8= 2024-03-04T17:40:47.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RySp7YRzKOe5z3gQbVfY9Ie9In8='} 2024-03-04T17:40:47.041 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9c4d784ed332dc7-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.041 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.041 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.041 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.041 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.041 DEBUG:boto:Data: 2024-03-04T17:40:47.042 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.042 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.042 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.042 DEBUG:boto:Params: {} 2024-03-04T17:40:47.042 DEBUG:boto:Token: None 2024-03-04T17:40:47.042 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.042 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.047 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc8c8ffa8e523036-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.047 DEBUG:boto:path=/ppfahn-18/obj10 2024-03-04T17:40:47.047 DEBUG:boto:auth_path=/ppfahn-18/obj10 2024-03-04T17:40:47.047 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.048 DEBUG:boto:Path: /ppfahn-18/obj10 2024-03-04T17:40:47.048 DEBUG:boto:Data: 2024-03-04T17:40:47.048 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.048 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.048 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.048 DEBUG:boto:Params: {} 2024-03-04T17:40:47.048 DEBUG:boto:Token: None 2024-03-04T17:40:47.048 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj10 2024-03-04T17:40:47.048 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3Z6X5J6KVpezda4r60yLTOaO67U= 2024-03-04T17:40:47.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3Z6X5J6KVpezda4r60yLTOaO67U='} 2024-03-04T17:40:47.092 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000318e43f21fe53014-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.092 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.092 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.093 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.093 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.093 DEBUG:boto:Data: 2024-03-04T17:40:47.093 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.093 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.093 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.093 DEBUG:boto:Params: {} 2024-03-04T17:40:47.093 DEBUG:boto:Token: None 2024-03-04T17:40:47.093 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.093 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dda5c040e34e49a1-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.117 DEBUG:boto:path=/ppfahn-18/obj11 2024-03-04T17:40:47.117 DEBUG:boto:auth_path=/ppfahn-18/obj11 2024-03-04T17:40:47.117 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.117 DEBUG:boto:Path: /ppfahn-18/obj11 2024-03-04T17:40:47.117 DEBUG:boto:Data: 2024-03-04T17:40:47.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.118 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.118 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.118 DEBUG:boto:Params: {} 2024-03-04T17:40:47.118 DEBUG:boto:Token: None 2024-03-04T17:40:47.118 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj11 2024-03-04T17:40:47.118 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S7uSTe+XfvOM2l0OCVsky9s3ISg= 2024-03-04T17:40:47.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S7uSTe+XfvOM2l0OCVsky9s3ISg='} 2024-03-04T17:40:47.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f92c9661d7dbb1b-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.134 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.134 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.134 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.134 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.135 DEBUG:boto:Data: 2024-03-04T17:40:47.135 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.135 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.135 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.135 DEBUG:boto:Params: {} 2024-03-04T17:40:47.135 DEBUG:boto:Token: None 2024-03-04T17:40:47.135 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.136 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ada3ccd10332982-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.138 DEBUG:boto:path=/ppfahn-18/obj12 2024-03-04T17:40:47.139 DEBUG:boto:auth_path=/ppfahn-18/obj12 2024-03-04T17:40:47.139 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.139 DEBUG:boto:Path: /ppfahn-18/obj12 2024-03-04T17:40:47.139 DEBUG:boto:Data: 2024-03-04T17:40:47.139 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.139 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.139 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.139 DEBUG:boto:Params: {} 2024-03-04T17:40:47.139 DEBUG:boto:Token: None 2024-03-04T17:40:47.139 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj12 2024-03-04T17:40:47.140 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BD+MTykFXSqp8LtTezj/wJL1yEU= 2024-03-04T17:40:47.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BD+MTykFXSqp8LtTezj/wJL1yEU='} 2024-03-04T17:40:47.184 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ce397ddf082381b-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.184 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.185 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.185 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.185 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.185 DEBUG:boto:Data: 2024-03-04T17:40:47.185 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.185 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.185 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.185 DEBUG:boto:Params: {} 2024-03-04T17:40:47.185 DEBUG:boto:Token: None 2024-03-04T17:40:47.185 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.186 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.193 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfb0f63c972b8883-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.193 DEBUG:boto:path=/ppfahn-18/obj13 2024-03-04T17:40:47.193 DEBUG:boto:auth_path=/ppfahn-18/obj13 2024-03-04T17:40:47.193 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.193 DEBUG:boto:Path: /ppfahn-18/obj13 2024-03-04T17:40:47.194 DEBUG:boto:Data: 2024-03-04T17:40:47.194 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.194 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.194 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.194 DEBUG:boto:Params: {} 2024-03-04T17:40:47.194 DEBUG:boto:Token: None 2024-03-04T17:40:47.194 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj13 2024-03-04T17:40:47.194 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pLzw0cNJPkOs2aNWt+PyyOiGJu8= 2024-03-04T17:40:47.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pLzw0cNJPkOs2aNWt+PyyOiGJu8='} 2024-03-04T17:40:47.201 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015fdfe4f548e89b0-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.201 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.201 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.202 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.202 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.202 DEBUG:boto:Data: 2024-03-04T17:40:47.202 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.202 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.202 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.202 DEBUG:boto:Params: {} 2024-03-04T17:40:47.202 DEBUG:boto:Token: None 2024-03-04T17:40:47.202 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.203 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.205 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080a82f0c40b30cfa-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.205 DEBUG:boto:path=/ppfahn-18/obj14 2024-03-04T17:40:47.205 DEBUG:boto:auth_path=/ppfahn-18/obj14 2024-03-04T17:40:47.206 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.206 DEBUG:boto:Path: /ppfahn-18/obj14 2024-03-04T17:40:47.206 DEBUG:boto:Data: 2024-03-04T17:40:47.206 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.206 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.206 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.206 DEBUG:boto:Params: {} 2024-03-04T17:40:47.206 DEBUG:boto:Token: None 2024-03-04T17:40:47.206 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj14 2024-03-04T17:40:47.206 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CPPeB/1RzwpgUrLK3X5jOwgqhm0= 2024-03-04T17:40:47.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CPPeB/1RzwpgUrLK3X5jOwgqhm0='} 2024-03-04T17:40:47.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d65629b60c47340-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.248 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.248 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.248 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.249 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.249 DEBUG:boto:Data: 2024-03-04T17:40:47.249 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.249 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.249 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.249 DEBUG:boto:Params: {} 2024-03-04T17:40:47.249 DEBUG:boto:Token: None 2024-03-04T17:40:47.249 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.249 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c992ccbbe336c198-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.252 DEBUG:boto:path=/ppfahn-18/obj15 2024-03-04T17:40:47.253 DEBUG:boto:auth_path=/ppfahn-18/obj15 2024-03-04T17:40:47.253 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.253 DEBUG:boto:Path: /ppfahn-18/obj15 2024-03-04T17:40:47.253 DEBUG:boto:Data: 2024-03-04T17:40:47.253 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.253 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.253 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.253 DEBUG:boto:Params: {} 2024-03-04T17:40:47.253 DEBUG:boto:Token: None 2024-03-04T17:40:47.253 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj15 2024-03-04T17:40:47.254 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZKtnkI4Vy/5hNP2L1WlIU1FXIxw= 2024-03-04T17:40:47.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZKtnkI4Vy/5hNP2L1WlIU1FXIxw='} 2024-03-04T17:40:47.260 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074f23abec2167869-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.260 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.260 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.260 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.260 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.260 DEBUG:boto:Data: 2024-03-04T17:40:47.260 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.260 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.261 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.261 DEBUG:boto:Params: {} 2024-03-04T17:40:47.261 DEBUG:boto:Token: None 2024-03-04T17:40:47.261 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.261 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc9e16202b0879ae-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.264 DEBUG:boto:path=/ppfahn-18/obj16 2024-03-04T17:40:47.264 DEBUG:boto:auth_path=/ppfahn-18/obj16 2024-03-04T17:40:47.264 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.264 DEBUG:boto:Path: /ppfahn-18/obj16 2024-03-04T17:40:47.264 DEBUG:boto:Data: 2024-03-04T17:40:47.264 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.264 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.264 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.264 DEBUG:boto:Params: {} 2024-03-04T17:40:47.265 DEBUG:boto:Token: None 2024-03-04T17:40:47.265 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj16 2024-03-04T17:40:47.265 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ekUhpFOqUF/04ijQyWjO9pnjgTU= 2024-03-04T17:40:47.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ekUhpFOqUF/04ijQyWjO9pnjgTU='} 2024-03-04T17:40:47.308 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdcc450764619a8a-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.309 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.309 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.309 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.309 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.309 DEBUG:boto:Data: 2024-03-04T17:40:47.309 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.309 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.309 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.309 DEBUG:boto:Params: {} 2024-03-04T17:40:47.309 DEBUG:boto:Token: None 2024-03-04T17:40:47.310 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.310 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.312 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b26da1df2d88abab-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.312 DEBUG:boto:path=/ppfahn-18/obj17 2024-03-04T17:40:47.313 DEBUG:boto:auth_path=/ppfahn-18/obj17 2024-03-04T17:40:47.313 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.313 DEBUG:boto:Path: /ppfahn-18/obj17 2024-03-04T17:40:47.313 DEBUG:boto:Data: 2024-03-04T17:40:47.313 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.313 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.313 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.313 DEBUG:boto:Params: {} 2024-03-04T17:40:47.313 DEBUG:boto:Token: None 2024-03-04T17:40:47.313 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj17 2024-03-04T17:40:47.314 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S3ktK/Ofahw7lDp3YwaNMAiCL/8= 2024-03-04T17:40:47.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S3ktK/Ofahw7lDp3YwaNMAiCL/8='} 2024-03-04T17:40:47.319 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081c49a539dcfa749-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.320 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.320 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.320 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.320 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.320 DEBUG:boto:Data: 2024-03-04T17:40:47.320 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.320 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.320 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.320 DEBUG:boto:Params: {} 2024-03-04T17:40:47.321 DEBUG:boto:Token: None 2024-03-04T17:40:47.321 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.321 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d549a776d7ffb8aa-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.324 DEBUG:boto:path=/ppfahn-18/obj18 2024-03-04T17:40:47.324 DEBUG:boto:auth_path=/ppfahn-18/obj18 2024-03-04T17:40:47.325 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.325 DEBUG:boto:Path: /ppfahn-18/obj18 2024-03-04T17:40:47.325 DEBUG:boto:Data: 2024-03-04T17:40:47.325 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.325 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.325 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.325 DEBUG:boto:Params: {} 2024-03-04T17:40:47.325 DEBUG:boto:Token: None 2024-03-04T17:40:47.325 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj18 2024-03-04T17:40:47.325 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y/QcqX8UBgGeVF/Ya9mZeoUixS0= 2024-03-04T17:40:47.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y/QcqX8UBgGeVF/Ya9mZeoUixS0='} 2024-03-04T17:40:47.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a3486b5541ee207-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.368 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.369 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.369 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.369 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.369 DEBUG:boto:Data: 2024-03-04T17:40:47.369 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.369 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.369 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.369 DEBUG:boto:Params: {} 2024-03-04T17:40:47.369 DEBUG:boto:Token: None 2024-03-04T17:40:47.369 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.370 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3aa9a361c3ceaad-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.374 DEBUG:boto:path=/ppfahn-18/obj19 2024-03-04T17:40:47.374 DEBUG:boto:auth_path=/ppfahn-18/obj19 2024-03-04T17:40:47.374 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.374 DEBUG:boto:Path: /ppfahn-18/obj19 2024-03-04T17:40:47.375 DEBUG:boto:Data: 2024-03-04T17:40:47.375 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.375 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.375 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.375 DEBUG:boto:Params: {} 2024-03-04T17:40:47.375 DEBUG:boto:Token: None 2024-03-04T17:40:47.375 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj19 2024-03-04T17:40:47.375 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tQxlhblPDnvYrBOTkX4WjLWgry8= 2024-03-04T17:40:47.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tQxlhblPDnvYrBOTkX4WjLWgry8='} 2024-03-04T17:40:47.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd5457ce8353704b-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.382 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.382 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.382 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.382 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.383 DEBUG:boto:Data: 2024-03-04T17:40:47.383 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.383 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.383 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.383 DEBUG:boto:Params: {} 2024-03-04T17:40:47.383 DEBUG:boto:Token: None 2024-03-04T17:40:47.383 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.383 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d70a7ee907fd576a-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.388 DEBUG:boto:path=/ppfahn-18/obj20 2024-03-04T17:40:47.388 DEBUG:boto:auth_path=/ppfahn-18/obj20 2024-03-04T17:40:47.388 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.388 DEBUG:boto:Path: /ppfahn-18/obj20 2024-03-04T17:40:47.388 DEBUG:boto:Data: 2024-03-04T17:40:47.388 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.388 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.388 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.388 DEBUG:boto:Params: {} 2024-03-04T17:40:47.389 DEBUG:boto:Token: None 2024-03-04T17:40:47.389 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj20 2024-03-04T17:40:47.389 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NOE+dMQOq34WhzZjBlxQWLl8ESo= 2024-03-04T17:40:47.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NOE+dMQOq34WhzZjBlxQWLl8ESo='} 2024-03-04T17:40:47.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009673c75ebc7fa454-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.436 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.437 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.437 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.437 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.437 DEBUG:boto:Data: 2024-03-04T17:40:47.437 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.437 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.437 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.437 DEBUG:boto:Params: {} 2024-03-04T17:40:47.437 DEBUG:boto:Token: None 2024-03-04T17:40:47.437 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.438 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c975c0b5fdaa35be-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.442 DEBUG:boto:path=/ppfahn-18/obj21 2024-03-04T17:40:47.442 DEBUG:boto:auth_path=/ppfahn-18/obj21 2024-03-04T17:40:47.442 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.442 DEBUG:boto:Path: /ppfahn-18/obj21 2024-03-04T17:40:47.443 DEBUG:boto:Data: 2024-03-04T17:40:47.443 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.443 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.443 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.443 DEBUG:boto:Params: {} 2024-03-04T17:40:47.443 DEBUG:boto:Token: None 2024-03-04T17:40:47.443 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj21 2024-03-04T17:40:47.443 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TyjniXjn4XEx/BvPzKRYnClebPc= 2024-03-04T17:40:47.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TyjniXjn4XEx/BvPzKRYnClebPc='} 2024-03-04T17:40:47.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1005eaaba8f4f1e-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.455 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.455 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.455 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.455 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.455 DEBUG:boto:Data: 2024-03-04T17:40:47.455 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.455 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.455 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.455 DEBUG:boto:Params: {} 2024-03-04T17:40:47.456 DEBUG:boto:Token: None 2024-03-04T17:40:47.456 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.456 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad9e7d6bca01447f-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.460 DEBUG:boto:path=/ppfahn-18/obj22 2024-03-04T17:40:47.460 DEBUG:boto:auth_path=/ppfahn-18/obj22 2024-03-04T17:40:47.460 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.460 DEBUG:boto:Path: /ppfahn-18/obj22 2024-03-04T17:40:47.460 DEBUG:boto:Data: 2024-03-04T17:40:47.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.460 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.460 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.461 DEBUG:boto:Params: {} 2024-03-04T17:40:47.461 DEBUG:boto:Token: None 2024-03-04T17:40:47.461 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj22 2024-03-04T17:40:47.461 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XC3OehRPM+TEUkejR9803cm9InY= 2024-03-04T17:40:47.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XC3OehRPM+TEUkejR9803cm9InY='} 2024-03-04T17:40:47.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cf0a7c7236d0c82-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.504 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.504 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.505 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.505 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.505 DEBUG:boto:Data: 2024-03-04T17:40:47.505 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.505 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.505 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.505 DEBUG:boto:Params: {} 2024-03-04T17:40:47.505 DEBUG:boto:Token: None 2024-03-04T17:40:47.505 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.506 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa4b5779478cefd7-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.510 DEBUG:boto:path=/ppfahn-18/obj23 2024-03-04T17:40:47.510 DEBUG:boto:auth_path=/ppfahn-18/obj23 2024-03-04T17:40:47.510 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.510 DEBUG:boto:Path: /ppfahn-18/obj23 2024-03-04T17:40:47.510 DEBUG:boto:Data: 2024-03-04T17:40:47.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.510 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.510 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.510 DEBUG:boto:Params: {} 2024-03-04T17:40:47.511 DEBUG:boto:Token: None 2024-03-04T17:40:47.511 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj23 2024-03-04T17:40:47.511 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m2BXxST4rhX4nEqZnBDlXXIfIFI= 2024-03-04T17:40:47.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m2BXxST4rhX4nEqZnBDlXXIfIFI='} 2024-03-04T17:40:47.519 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000acf8fbf8e8ebff97-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.519 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.519 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.519 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.520 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.520 DEBUG:boto:Data: 2024-03-04T17:40:47.520 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.520 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.520 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.520 DEBUG:boto:Params: {} 2024-03-04T17:40:47.520 DEBUG:boto:Token: None 2024-03-04T17:40:47.520 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.520 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a62081c2a7dfefc-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.524 DEBUG:boto:path=/ppfahn-18/obj24 2024-03-04T17:40:47.524 DEBUG:boto:auth_path=/ppfahn-18/obj24 2024-03-04T17:40:47.524 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.525 DEBUG:boto:Path: /ppfahn-18/obj24 2024-03-04T17:40:47.525 DEBUG:boto:Data: 2024-03-04T17:40:47.525 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.525 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.525 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.525 DEBUG:boto:Params: {} 2024-03-04T17:40:47.525 DEBUG:boto:Token: None 2024-03-04T17:40:47.525 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj24 2024-03-04T17:40:47.525 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uLena+Q5ba/vzt8OqrTfmINjhKM= 2024-03-04T17:40:47.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uLena+Q5ba/vzt8OqrTfmINjhKM='} 2024-03-04T17:40:47.568 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c55cbebc826cf2eb-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.568 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.568 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.569 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.569 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.569 DEBUG:boto:Data: 2024-03-04T17:40:47.569 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.569 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.569 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.569 DEBUG:boto:Params: {} 2024-03-04T17:40:47.569 DEBUG:boto:Token: None 2024-03-04T17:40:47.569 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.569 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba451b87b5135991-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.573 DEBUG:boto:path=/ppfahn-18/obj25 2024-03-04T17:40:47.573 DEBUG:boto:auth_path=/ppfahn-18/obj25 2024-03-04T17:40:47.574 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.574 DEBUG:boto:Path: /ppfahn-18/obj25 2024-03-04T17:40:47.574 DEBUG:boto:Data: 2024-03-04T17:40:47.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.574 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.574 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.574 DEBUG:boto:Params: {} 2024-03-04T17:40:47.574 DEBUG:boto:Token: None 2024-03-04T17:40:47.574 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj25 2024-03-04T17:40:47.574 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VPTjRg/bD/90Uo0BC4CzhPe3oS8= 2024-03-04T17:40:47.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VPTjRg/bD/90Uo0BC4CzhPe3oS8='} 2024-03-04T17:40:47.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000529dbdab99a8f169-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.583 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.583 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.583 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.583 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.583 DEBUG:boto:Data: 2024-03-04T17:40:47.583 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.583 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.583 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.583 DEBUG:boto:Params: {} 2024-03-04T17:40:47.583 DEBUG:boto:Token: None 2024-03-04T17:40:47.584 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.584 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f7e947e00941a6f-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.588 DEBUG:boto:path=/ppfahn-18/obj26 2024-03-04T17:40:47.588 DEBUG:boto:auth_path=/ppfahn-18/obj26 2024-03-04T17:40:47.588 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.588 DEBUG:boto:Path: /ppfahn-18/obj26 2024-03-04T17:40:47.588 DEBUG:boto:Data: 2024-03-04T17:40:47.588 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.589 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.589 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.589 DEBUG:boto:Params: {} 2024-03-04T17:40:47.589 DEBUG:boto:Token: None 2024-03-04T17:40:47.589 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj26 2024-03-04T17:40:47.589 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JrMZQx8skL0IQs9+So6PGQxxKWY= 2024-03-04T17:40:47.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JrMZQx8skL0IQs9+So6PGQxxKWY='} 2024-03-04T17:40:47.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da8294b37bda4702-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.632 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.632 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.633 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.633 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.633 DEBUG:boto:Data: 2024-03-04T17:40:47.633 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.633 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.633 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.633 DEBUG:boto:Params: {} 2024-03-04T17:40:47.633 DEBUG:boto:Token: None 2024-03-04T17:40:47.634 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.634 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.638 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009018a56643c8b64-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.638 DEBUG:boto:path=/ppfahn-18/obj27 2024-03-04T17:40:47.638 DEBUG:boto:auth_path=/ppfahn-18/obj27 2024-03-04T17:40:47.638 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.638 DEBUG:boto:Path: /ppfahn-18/obj27 2024-03-04T17:40:47.638 DEBUG:boto:Data: 2024-03-04T17:40:47.638 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.639 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.639 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.639 DEBUG:boto:Params: {} 2024-03-04T17:40:47.639 DEBUG:boto:Token: None 2024-03-04T17:40:47.639 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj27 2024-03-04T17:40:47.639 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s9raxzbn77r1nM4gIjYCK1qqPYg= 2024-03-04T17:40:47.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s9raxzbn77r1nM4gIjYCK1qqPYg='} 2024-03-04T17:40:47.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011aaa603ee485ced-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.647 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.647 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.648 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.648 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.648 DEBUG:boto:Data: 2024-03-04T17:40:47.648 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.648 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.648 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.648 DEBUG:boto:Params: {} 2024-03-04T17:40:47.648 DEBUG:boto:Token: None 2024-03-04T17:40:47.648 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.648 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b57d00166d30515a-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.652 DEBUG:boto:path=/ppfahn-18/obj28 2024-03-04T17:40:47.653 DEBUG:boto:auth_path=/ppfahn-18/obj28 2024-03-04T17:40:47.653 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.653 DEBUG:boto:Path: /ppfahn-18/obj28 2024-03-04T17:40:47.653 DEBUG:boto:Data: 2024-03-04T17:40:47.653 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.653 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.653 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.653 DEBUG:boto:Params: {} 2024-03-04T17:40:47.653 DEBUG:boto:Token: None 2024-03-04T17:40:47.653 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj28 2024-03-04T17:40:47.654 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:clzqdq4W+vu3fVfr/Pq134g0zfs= 2024-03-04T17:40:47.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:clzqdq4W+vu3fVfr/Pq134g0zfs='} 2024-03-04T17:40:47.696 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000238ba022b950e81-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.696 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.696 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.696 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.697 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.697 DEBUG:boto:Data: 2024-03-04T17:40:47.697 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.697 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.697 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.697 DEBUG:boto:Params: {} 2024-03-04T17:40:47.697 DEBUG:boto:Token: None 2024-03-04T17:40:47.697 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.697 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005de9df8d53016e54-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.702 DEBUG:boto:path=/ppfahn-18/obj29 2024-03-04T17:40:47.702 DEBUG:boto:auth_path=/ppfahn-18/obj29 2024-03-04T17:40:47.702 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.702 DEBUG:boto:Path: /ppfahn-18/obj29 2024-03-04T17:40:47.702 DEBUG:boto:Data: 2024-03-04T17:40:47.702 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.702 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.703 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.703 DEBUG:boto:Params: {} 2024-03-04T17:40:47.703 DEBUG:boto:Token: None 2024-03-04T17:40:47.703 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj29 2024-03-04T17:40:47.703 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E8vH3ChebXa/ZaeXS/OWVZFft7Y= 2024-03-04T17:40:47.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E8vH3ChebXa/ZaeXS/OWVZFft7Y='} 2024-03-04T17:40:47.712 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbcccefd5722ffb9-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.712 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.712 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.712 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.712 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.712 DEBUG:boto:Data: 2024-03-04T17:40:47.712 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.713 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.713 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.713 DEBUG:boto:Params: {} 2024-03-04T17:40:47.713 DEBUG:boto:Token: None 2024-03-04T17:40:47.713 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.713 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.717 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f3bc29a1bddbf09-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.717 DEBUG:boto:path=/ppfahn-18/obj30 2024-03-04T17:40:47.717 DEBUG:boto:auth_path=/ppfahn-18/obj30 2024-03-04T17:40:47.718 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.718 DEBUG:boto:Path: /ppfahn-18/obj30 2024-03-04T17:40:47.718 DEBUG:boto:Data: 2024-03-04T17:40:47.718 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.718 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.718 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.718 DEBUG:boto:Params: {} 2024-03-04T17:40:47.718 DEBUG:boto:Token: None 2024-03-04T17:40:47.718 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj30 2024-03-04T17:40:47.718 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kXDFKkOvmAKbnnGegFAC2LKvhho= 2024-03-04T17:40:47.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kXDFKkOvmAKbnnGegFAC2LKvhho='} 2024-03-04T17:40:47.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d434083ef746095d-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.764 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.764 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.764 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.765 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.765 DEBUG:boto:Data: 2024-03-04T17:40:47.765 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.765 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.765 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.765 DEBUG:boto:Params: {} 2024-03-04T17:40:47.765 DEBUG:boto:Token: None 2024-03-04T17:40:47.765 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.765 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009df4e8957bd22947-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.770 DEBUG:boto:path=/ppfahn-18/obj31 2024-03-04T17:40:47.770 DEBUG:boto:auth_path=/ppfahn-18/obj31 2024-03-04T17:40:47.770 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.770 DEBUG:boto:Path: /ppfahn-18/obj31 2024-03-04T17:40:47.770 DEBUG:boto:Data: 2024-03-04T17:40:47.770 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.771 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.771 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.771 DEBUG:boto:Params: {} 2024-03-04T17:40:47.771 DEBUG:boto:Token: None 2024-03-04T17:40:47.771 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj31 2024-03-04T17:40:47.771 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fWpPIic20yYavqVs65Hn+tjaYpc= 2024-03-04T17:40:47.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fWpPIic20yYavqVs65Hn+tjaYpc='} 2024-03-04T17:40:47.779 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000542169fceae67706-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.779 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.779 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.780 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.780 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.780 DEBUG:boto:Data: 2024-03-04T17:40:47.780 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.780 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.780 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.781 DEBUG:boto:Params: {} 2024-03-04T17:40:47.781 DEBUG:boto:Token: None 2024-03-04T17:40:47.781 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.781 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ad5d6fe0ed6795f-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.786 DEBUG:boto:path=/ppfahn-18/obj32 2024-03-04T17:40:47.786 DEBUG:boto:auth_path=/ppfahn-18/obj32 2024-03-04T17:40:47.786 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.786 DEBUG:boto:Path: /ppfahn-18/obj32 2024-03-04T17:40:47.786 DEBUG:boto:Data: 2024-03-04T17:40:47.786 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.786 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.786 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.786 DEBUG:boto:Params: {} 2024-03-04T17:40:47.787 DEBUG:boto:Token: None 2024-03-04T17:40:47.787 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj32 2024-03-04T17:40:47.787 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UcPERRwoI9WY8cWh2FsnMzfeWgQ= 2024-03-04T17:40:47.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UcPERRwoI9WY8cWh2FsnMzfeWgQ='} 2024-03-04T17:40:47.832 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003bb38c4ca2b2b5cb-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.832 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.832 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.832 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.833 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.833 DEBUG:boto:Data: 2024-03-04T17:40:47.833 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.833 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.833 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.833 DEBUG:boto:Params: {} 2024-03-04T17:40:47.833 DEBUG:boto:Token: None 2024-03-04T17:40:47.833 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.833 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b049c66ab5f4ebb-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.838 DEBUG:boto:path=/ppfahn-18/obj33 2024-03-04T17:40:47.838 DEBUG:boto:auth_path=/ppfahn-18/obj33 2024-03-04T17:40:47.838 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.838 DEBUG:boto:Path: /ppfahn-18/obj33 2024-03-04T17:40:47.838 DEBUG:boto:Data: 2024-03-04T17:40:47.838 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.838 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.838 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.838 DEBUG:boto:Params: {} 2024-03-04T17:40:47.839 DEBUG:boto:Token: None 2024-03-04T17:40:47.839 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj33 2024-03-04T17:40:47.839 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nhRRuPaUFFZqCtM5qkVSaThUvxM= 2024-03-04T17:40:47.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nhRRuPaUFFZqCtM5qkVSaThUvxM='} 2024-03-04T17:40:47.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e73dc262f25698f2-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.846 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.847 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.847 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.847 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.847 DEBUG:boto:Data: 2024-03-04T17:40:47.847 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.847 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.847 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.847 DEBUG:boto:Params: {} 2024-03-04T17:40:47.847 DEBUG:boto:Token: None 2024-03-04T17:40:47.847 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.848 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.851 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bc75abe0b11e040-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.852 DEBUG:boto:path=/ppfahn-18/obj34 2024-03-04T17:40:47.852 DEBUG:boto:auth_path=/ppfahn-18/obj34 2024-03-04T17:40:47.852 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.852 DEBUG:boto:Path: /ppfahn-18/obj34 2024-03-04T17:40:47.852 DEBUG:boto:Data: 2024-03-04T17:40:47.852 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.852 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.852 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.853 DEBUG:boto:Params: {} 2024-03-04T17:40:47.853 DEBUG:boto:Token: None 2024-03-04T17:40:47.853 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj34 2024-03-04T17:40:47.853 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VnZcd1IcyAH9U9jmoqr9SmCt6c4= 2024-03-04T17:40:47.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VnZcd1IcyAH9U9jmoqr9SmCt6c4='} 2024-03-04T17:40:47.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5435487d9bace6e-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.896 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.896 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.896 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.897 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.897 DEBUG:boto:Data: 2024-03-04T17:40:47.897 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.897 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.897 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.897 DEBUG:boto:Params: {} 2024-03-04T17:40:47.897 DEBUG:boto:Token: None 2024-03-04T17:40:47.897 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.897 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.901 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029fcf73a15a2d321-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.901 DEBUG:boto:path=/ppfahn-18/obj35 2024-03-04T17:40:47.902 DEBUG:boto:auth_path=/ppfahn-18/obj35 2024-03-04T17:40:47.902 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.902 DEBUG:boto:Path: /ppfahn-18/obj35 2024-03-04T17:40:47.902 DEBUG:boto:Data: 2024-03-04T17:40:47.902 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.902 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.902 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.902 DEBUG:boto:Params: {} 2024-03-04T17:40:47.903 DEBUG:boto:Token: None 2024-03-04T17:40:47.903 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj35 2024-03-04T17:40:47.903 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nsGNAIf9u1B3Cxukzrev7ST9pvU= 2024-03-04T17:40:47.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nsGNAIf9u1B3Cxukzrev7ST9pvU='} 2024-03-04T17:40:47.912 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b5c25bceff2ecef-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.912 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.912 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.912 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.913 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.913 DEBUG:boto:Data: 2024-03-04T17:40:47.913 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.913 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.913 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.913 DEBUG:boto:Params: {} 2024-03-04T17:40:47.913 DEBUG:boto:Token: None 2024-03-04T17:40:47.913 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.914 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047228b693d5456e9-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.918 DEBUG:boto:path=/ppfahn-18/obj36 2024-03-04T17:40:47.918 DEBUG:boto:auth_path=/ppfahn-18/obj36 2024-03-04T17:40:47.918 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.918 DEBUG:boto:Path: /ppfahn-18/obj36 2024-03-04T17:40:47.918 DEBUG:boto:Data: 2024-03-04T17:40:47.918 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.918 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.919 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.919 DEBUG:boto:Params: {} 2024-03-04T17:40:47.919 DEBUG:boto:Token: None 2024-03-04T17:40:47.919 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj36 2024-03-04T17:40:47.919 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2SE+4ckzwMwvEuzXAhozEVp+0LM= 2024-03-04T17:40:47.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2SE+4ckzwMwvEuzXAhozEVp+0LM='} 2024-03-04T17:40:47.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9e45acdb67cc77a-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.964 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.964 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.965 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.965 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.965 DEBUG:boto:Data: 2024-03-04T17:40:47.965 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.965 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.965 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.965 DEBUG:boto:Params: {} 2024-03-04T17:40:47.965 DEBUG:boto:Token: None 2024-03-04T17:40:47.965 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.965 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010f450c1ebf2f11e-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.970 DEBUG:boto:path=/ppfahn-18/obj37 2024-03-04T17:40:47.970 DEBUG:boto:auth_path=/ppfahn-18/obj37 2024-03-04T17:40:47.970 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.970 DEBUG:boto:Path: /ppfahn-18/obj37 2024-03-04T17:40:47.970 DEBUG:boto:Data: 2024-03-04T17:40:47.970 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.971 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.971 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.971 DEBUG:boto:Params: {} 2024-03-04T17:40:47.971 DEBUG:boto:Token: None 2024-03-04T17:40:47.971 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj37 2024-03-04T17:40:47.971 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RarFwb+ENEl0Yc0a3AZ3+0+WnxE= 2024-03-04T17:40:47.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RarFwb+ENEl0Yc0a3AZ3+0+WnxE='} 2024-03-04T17:40:47.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d295ef0998ac5dbd-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.979 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:47.979 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:47.980 DEBUG:boto:Method: HEAD 2024-03-04T17:40:47.980 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:47.980 DEBUG:boto:Data: 2024-03-04T17:40:47.980 DEBUG:boto:Headers: {} 2024-03-04T17:40:47.980 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.980 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.980 DEBUG:boto:Params: {} 2024-03-04T17:40:47.980 DEBUG:boto:Token: None 2024-03-04T17:40:47.980 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/ 2024-03-04T17:40:47.980 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T17:40:47.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T17:40:47.984 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d4c4e9aa4498083-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:47.985 DEBUG:boto:path=/ppfahn-18/obj38 2024-03-04T17:40:47.985 DEBUG:boto:auth_path=/ppfahn-18/obj38 2024-03-04T17:40:47.985 DEBUG:boto:Method: PUT 2024-03-04T17:40:47.985 DEBUG:boto:Path: /ppfahn-18/obj38 2024-03-04T17:40:47.985 DEBUG:boto:Data: 2024-03-04T17:40:47.985 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:47.985 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:47.985 DEBUG:boto:Port: 8001 2024-03-04T17:40:47.985 DEBUG:boto:Params: {} 2024-03-04T17:40:47.985 DEBUG:boto:Token: None 2024-03-04T17:40:47.986 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:47 GMT /ppfahn-18/obj38 2024-03-04T17:40:47.986 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/8IViy8HT8/bKsilVsPdPGBXhS4= 2024-03-04T17:40:47.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/8IViy8HT8/bKsilVsPdPGBXhS4='} 2024-03-04T17:40:48.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a64cd804e4c37691-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.032 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:48.032 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:48.032 DEBUG:boto:Method: HEAD 2024-03-04T17:40:48.033 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:48.033 DEBUG:boto:Data: 2024-03-04T17:40:48.033 DEBUG:boto:Headers: {} 2024-03-04T17:40:48.033 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.033 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.033 DEBUG:boto:Params: {} 2024-03-04T17:40:48.033 DEBUG:boto:Token: None 2024-03-04T17:40:48.033 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/ 2024-03-04T17:40:48.033 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T17:40:48.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T17:40:48.037 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d4ae951ad0c7557-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.038 DEBUG:boto:path=/ppfahn-18/obj39 2024-03-04T17:40:48.038 DEBUG:boto:auth_path=/ppfahn-18/obj39 2024-03-04T17:40:48.038 DEBUG:boto:Method: PUT 2024-03-04T17:40:48.038 DEBUG:boto:Path: /ppfahn-18/obj39 2024-03-04T17:40:48.038 DEBUG:boto:Data: 2024-03-04T17:40:48.038 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:48.038 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.038 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.038 DEBUG:boto:Params: {} 2024-03-04T17:40:48.039 DEBUG:boto:Token: None 2024-03-04T17:40:48.039 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/obj39 2024-03-04T17:40:48.039 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:C+xPRV2IGT7rT/HKMA5Bqd/cdkQ= 2024-03-04T17:40:48.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:C+xPRV2IGT7rT/HKMA5Bqd/cdkQ='} 2024-03-04T17:40:48.047 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cd4aeb34ae1d09d-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.048 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:48.048 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:48.048 DEBUG:boto:Method: HEAD 2024-03-04T17:40:48.048 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:48.048 DEBUG:boto:Data: 2024-03-04T17:40:48.048 DEBUG:boto:Headers: {} 2024-03-04T17:40:48.048 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.048 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.048 DEBUG:boto:Params: {} 2024-03-04T17:40:48.048 DEBUG:boto:Token: None 2024-03-04T17:40:48.049 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/ 2024-03-04T17:40:48.049 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T17:40:48.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T17:40:48.055 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006baf7f28eece738d-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.055 DEBUG:boto:path=/ppfahn-18/obj40 2024-03-04T17:40:48.055 DEBUG:boto:auth_path=/ppfahn-18/obj40 2024-03-04T17:40:48.055 DEBUG:boto:Method: PUT 2024-03-04T17:40:48.055 DEBUG:boto:Path: /ppfahn-18/obj40 2024-03-04T17:40:48.055 DEBUG:boto:Data: 2024-03-04T17:40:48.056 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:48.056 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.056 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.056 DEBUG:boto:Params: {} 2024-03-04T17:40:48.056 DEBUG:boto:Token: None 2024-03-04T17:40:48.056 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/obj40 2024-03-04T17:40:48.056 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MXMIZcmN+H+QH8rIlqx7GIM2Afs= 2024-03-04T17:40:48.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MXMIZcmN+H+QH8rIlqx7GIM2Afs='} 2024-03-04T17:40:48.100 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a131d22b16f5f74-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.100 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:48.100 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:48.100 DEBUG:boto:Method: HEAD 2024-03-04T17:40:48.101 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:48.101 DEBUG:boto:Data: 2024-03-04T17:40:48.101 DEBUG:boto:Headers: {} 2024-03-04T17:40:48.101 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.101 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.101 DEBUG:boto:Params: {} 2024-03-04T17:40:48.101 DEBUG:boto:Token: None 2024-03-04T17:40:48.101 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/ 2024-03-04T17:40:48.101 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T17:40:48.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T17:40:48.106 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc5424a2ff14ce7b-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.106 DEBUG:boto:path=/ppfahn-18/obj41 2024-03-04T17:40:48.106 DEBUG:boto:auth_path=/ppfahn-18/obj41 2024-03-04T17:40:48.106 DEBUG:boto:Method: PUT 2024-03-04T17:40:48.106 DEBUG:boto:Path: /ppfahn-18/obj41 2024-03-04T17:40:48.106 DEBUG:boto:Data: 2024-03-04T17:40:48.106 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:48.107 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.107 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.107 DEBUG:boto:Params: {} 2024-03-04T17:40:48.107 DEBUG:boto:Token: None 2024-03-04T17:40:48.107 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/obj41 2024-03-04T17:40:48.107 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+t7tIFV5NGI1Hhe6+ECqyM7JZoU= 2024-03-04T17:40:48.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+t7tIFV5NGI1Hhe6+ECqyM7JZoU='} 2024-03-04T17:40:48.116 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c75eb7e11982ce9-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.116 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:48.116 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:48.116 DEBUG:boto:Method: HEAD 2024-03-04T17:40:48.117 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:48.117 DEBUG:boto:Data: 2024-03-04T17:40:48.118 DEBUG:boto:Headers: {} 2024-03-04T17:40:48.118 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.118 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.118 DEBUG:boto:Params: {} 2024-03-04T17:40:48.118 DEBUG:boto:Token: None 2024-03-04T17:40:48.119 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/ 2024-03-04T17:40:48.119 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T17:40:48.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T17:40:48.123 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d2b8834f54a46f4-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.123 DEBUG:boto:path=/ppfahn-18/obj42 2024-03-04T17:40:48.124 DEBUG:boto:auth_path=/ppfahn-18/obj42 2024-03-04T17:40:48.124 DEBUG:boto:Method: PUT 2024-03-04T17:40:48.124 DEBUG:boto:Path: /ppfahn-18/obj42 2024-03-04T17:40:48.124 DEBUG:boto:Data: 2024-03-04T17:40:48.124 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:48.124 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.124 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.124 DEBUG:boto:Params: {} 2024-03-04T17:40:48.124 DEBUG:boto:Token: None 2024-03-04T17:40:48.124 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/obj42 2024-03-04T17:40:48.125 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eFAn9RJrC2oNSl3u3Qhs0fExcls= 2024-03-04T17:40:48.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eFAn9RJrC2oNSl3u3Qhs0fExcls='} 2024-03-04T17:40:48.168 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6d9c88d4c9830be-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.168 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:48.169 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:48.169 DEBUG:boto:Method: HEAD 2024-03-04T17:40:48.169 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:48.169 DEBUG:boto:Data: 2024-03-04T17:40:48.169 DEBUG:boto:Headers: {} 2024-03-04T17:40:48.169 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.169 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.169 DEBUG:boto:Params: {} 2024-03-04T17:40:48.169 DEBUG:boto:Token: None 2024-03-04T17:40:48.169 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/ 2024-03-04T17:40:48.170 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T17:40:48.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T17:40:48.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0b6259f29eaf597-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.174 DEBUG:boto:path=/ppfahn-18/obj43 2024-03-04T17:40:48.174 DEBUG:boto:auth_path=/ppfahn-18/obj43 2024-03-04T17:40:48.174 DEBUG:boto:Method: PUT 2024-03-04T17:40:48.174 DEBUG:boto:Path: /ppfahn-18/obj43 2024-03-04T17:40:48.174 DEBUG:boto:Data: 2024-03-04T17:40:48.174 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:48.175 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.175 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.175 DEBUG:boto:Params: {} 2024-03-04T17:40:48.175 DEBUG:boto:Token: None 2024-03-04T17:40:48.175 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/obj43 2024-03-04T17:40:48.175 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yxMj2iPqWdgj/I1P/2l8yw+5T9c= 2024-03-04T17:40:48.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yxMj2iPqWdgj/I1P/2l8yw+5T9c='} 2024-03-04T17:40:48.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c72e38993a560405-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.183 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:48.183 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:48.183 DEBUG:boto:Method: HEAD 2024-03-04T17:40:48.184 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:48.184 DEBUG:boto:Data: 2024-03-04T17:40:48.184 DEBUG:boto:Headers: {} 2024-03-04T17:40:48.184 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.184 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.184 DEBUG:boto:Params: {} 2024-03-04T17:40:48.184 DEBUG:boto:Token: None 2024-03-04T17:40:48.184 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/ 2024-03-04T17:40:48.184 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T17:40:48.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T17:40:48.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011f0092d0415020e-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.190 DEBUG:boto:path=/ppfahn-18/obj44 2024-03-04T17:40:48.190 DEBUG:boto:auth_path=/ppfahn-18/obj44 2024-03-04T17:40:48.190 DEBUG:boto:Method: PUT 2024-03-04T17:40:48.190 DEBUG:boto:Path: /ppfahn-18/obj44 2024-03-04T17:40:48.190 DEBUG:boto:Data: 2024-03-04T17:40:48.190 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:48.190 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.190 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.190 DEBUG:boto:Params: {} 2024-03-04T17:40:48.190 DEBUG:boto:Token: None 2024-03-04T17:40:48.191 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/obj44 2024-03-04T17:40:48.191 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:N0C7V9T6qYdkftFJrxj7I7eymv4= 2024-03-04T17:40:48.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:N0C7V9T6qYdkftFJrxj7I7eymv4='} 2024-03-04T17:40:48.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046bf58e5ddce7200-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.236 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:48.236 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:48.237 DEBUG:boto:Method: HEAD 2024-03-04T17:40:48.237 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:48.237 DEBUG:boto:Data: 2024-03-04T17:40:48.237 DEBUG:boto:Headers: {} 2024-03-04T17:40:48.237 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.237 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.237 DEBUG:boto:Params: {} 2024-03-04T17:40:48.237 DEBUG:boto:Token: None 2024-03-04T17:40:48.237 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/ 2024-03-04T17:40:48.237 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T17:40:48.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T17:40:48.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002120198180554935-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.242 DEBUG:boto:path=/ppfahn-18/obj45 2024-03-04T17:40:48.243 DEBUG:boto:auth_path=/ppfahn-18/obj45 2024-03-04T17:40:48.243 DEBUG:boto:Method: PUT 2024-03-04T17:40:48.243 DEBUG:boto:Path: /ppfahn-18/obj45 2024-03-04T17:40:48.243 DEBUG:boto:Data: 2024-03-04T17:40:48.243 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:48.243 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.243 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.243 DEBUG:boto:Params: {} 2024-03-04T17:40:48.243 DEBUG:boto:Token: None 2024-03-04T17:40:48.244 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/obj45 2024-03-04T17:40:48.244 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xIYkjH30yELEE+6r30pyFu1Omw8= 2024-03-04T17:40:48.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xIYkjH30yELEE+6r30pyFu1Omw8='} 2024-03-04T17:40:48.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000159c3e8b332bcea4-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.252 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:48.252 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:48.252 DEBUG:boto:Method: HEAD 2024-03-04T17:40:48.252 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:48.252 DEBUG:boto:Data: 2024-03-04T17:40:48.252 DEBUG:boto:Headers: {} 2024-03-04T17:40:48.253 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.253 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.253 DEBUG:boto:Params: {} 2024-03-04T17:40:48.253 DEBUG:boto:Token: None 2024-03-04T17:40:48.253 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/ 2024-03-04T17:40:48.253 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T17:40:48.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T17:40:48.258 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdabc9fbbc072696-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.258 DEBUG:boto:path=/ppfahn-18/obj46 2024-03-04T17:40:48.258 DEBUG:boto:auth_path=/ppfahn-18/obj46 2024-03-04T17:40:48.259 DEBUG:boto:Method: PUT 2024-03-04T17:40:48.259 DEBUG:boto:Path: /ppfahn-18/obj46 2024-03-04T17:40:48.259 DEBUG:boto:Data: 2024-03-04T17:40:48.259 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:48.259 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.259 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.259 DEBUG:boto:Params: {} 2024-03-04T17:40:48.259 DEBUG:boto:Token: None 2024-03-04T17:40:48.259 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/obj46 2024-03-04T17:40:48.259 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HP7rhD1GDQu7pT7yNIu497FM5EU= 2024-03-04T17:40:48.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HP7rhD1GDQu7pT7yNIu497FM5EU='} 2024-03-04T17:40:48.304 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084ab3141f3b356b4-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.304 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:48.304 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:48.304 DEBUG:boto:Method: HEAD 2024-03-04T17:40:48.305 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:48.305 DEBUG:boto:Data: 2024-03-04T17:40:48.305 DEBUG:boto:Headers: {} 2024-03-04T17:40:48.305 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.305 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.305 DEBUG:boto:Params: {} 2024-03-04T17:40:48.305 DEBUG:boto:Token: None 2024-03-04T17:40:48.305 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/ 2024-03-04T17:40:48.305 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T17:40:48.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T17:40:48.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093a3b1b57143a1f2-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.310 DEBUG:boto:path=/ppfahn-18/obj47 2024-03-04T17:40:48.310 DEBUG:boto:auth_path=/ppfahn-18/obj47 2024-03-04T17:40:48.310 DEBUG:boto:Method: PUT 2024-03-04T17:40:48.310 DEBUG:boto:Path: /ppfahn-18/obj47 2024-03-04T17:40:48.310 DEBUG:boto:Data: 2024-03-04T17:40:48.310 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:48.311 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.311 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.311 DEBUG:boto:Params: {} 2024-03-04T17:40:48.311 DEBUG:boto:Token: None 2024-03-04T17:40:48.311 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/obj47 2024-03-04T17:40:48.311 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F321Zeirb4Qh1sZSv09Gl4V270o= 2024-03-04T17:40:48.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F321Zeirb4Qh1sZSv09Gl4V270o='} 2024-03-04T17:40:48.319 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000defd011a6cb65660-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.320 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:48.320 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:48.320 DEBUG:boto:Method: HEAD 2024-03-04T17:40:48.320 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:48.320 DEBUG:boto:Data: 2024-03-04T17:40:48.320 DEBUG:boto:Headers: {} 2024-03-04T17:40:48.320 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.320 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.320 DEBUG:boto:Params: {} 2024-03-04T17:40:48.320 DEBUG:boto:Token: None 2024-03-04T17:40:48.321 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/ 2024-03-04T17:40:48.321 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T17:40:48.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T17:40:48.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a09371a9dddcfb2-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.325 DEBUG:boto:path=/ppfahn-18/obj48 2024-03-04T17:40:48.325 DEBUG:boto:auth_path=/ppfahn-18/obj48 2024-03-04T17:40:48.326 DEBUG:boto:Method: PUT 2024-03-04T17:40:48.326 DEBUG:boto:Path: /ppfahn-18/obj48 2024-03-04T17:40:48.326 DEBUG:boto:Data: 2024-03-04T17:40:48.326 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:48.326 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.326 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.326 DEBUG:boto:Params: {} 2024-03-04T17:40:48.326 DEBUG:boto:Token: None 2024-03-04T17:40:48.326 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/obj48 2024-03-04T17:40:48.326 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rdJVEKBdVKD5QJl+Xc3BUtAobIA= 2024-03-04T17:40:48.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rdJVEKBdVKD5QJl+Xc3BUtAobIA='} 2024-03-04T17:40:48.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eabc819ad0d34912-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.368 DEBUG:boto:path=/ppfahn-18/ 2024-03-04T17:40:48.368 DEBUG:boto:auth_path=/ppfahn-18/ 2024-03-04T17:40:48.369 DEBUG:boto:Method: HEAD 2024-03-04T17:40:48.369 DEBUG:boto:Path: /ppfahn-18/ 2024-03-04T17:40:48.369 DEBUG:boto:Data: 2024-03-04T17:40:48.369 DEBUG:boto:Headers: {} 2024-03-04T17:40:48.369 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.369 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.369 DEBUG:boto:Params: {} 2024-03-04T17:40:48.369 DEBUG:boto:Token: None 2024-03-04T17:40:48.369 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/ 2024-03-04T17:40:48.369 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T17:40:48.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T17:40:48.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdb376f94bc5741b-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.374 DEBUG:boto:path=/ppfahn-18/obj49 2024-03-04T17:40:48.374 DEBUG:boto:auth_path=/ppfahn-18/obj49 2024-03-04T17:40:48.374 DEBUG:boto:Method: PUT 2024-03-04T17:40:48.374 DEBUG:boto:Path: /ppfahn-18/obj49 2024-03-04T17:40:48.375 DEBUG:boto:Data: 2024-03-04T17:40:48.375 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:40:48.375 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:40:48.375 DEBUG:boto:Port: 8001 2024-03-04T17:40:48.375 DEBUG:boto:Params: {} 2024-03-04T17:40:48.375 DEBUG:boto:Token: None 2024-03-04T17:40:48.375 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:40:48 GMT /ppfahn-18/obj49 2024-03-04T17:40:48.375 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/9VsweVnoUAw6iEnur47e9kogfU= 2024-03-04T17:40:48.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/9VsweVnoUAw6iEnur47e9kogfU='} 2024-03-04T17:40:48.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b51d14cfbc113b17-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:40:48.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:48.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:48.384 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:48.425 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:48.425 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:40:48.503 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709574043.443936_99.1","last_update":"2024-03-04T17:40:43.443936Z"}] 2024-03-04T17:40:48.503 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574043.443936_99.1'} 2024-03-04T17:40:48.503 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:40:48.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:48.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:48.504 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:48.547 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:48.547 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:48.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:40:48.598+0000 7f55d50db980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:48.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:40:48.602+0000 7f55d50db980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:48.624 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574043.443936_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:40:43.443936Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:48Z"} 2024-03-04T17:40:48.625 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574043.443936_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:40:43.443936Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:48Z"} 2024-03-04T17:40:48.625 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574043.443936_99.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:40:43.443936Z', 'realm_epoch': 2}}] 2024-03-04T17:40:48.625 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574043.443936_99.1'} 2024-03-04T17:40:48.625 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574043.443936_99.1'} 2024-03-04T17:40:48.625 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:40:48.625 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:40:48.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:48.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:48.626 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:48.667 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:48.667 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:48.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:48.723+0000 7fbbba3e4980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:48.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:40:48.723+0000 7fbbba3e4980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:48.748 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574043.443936_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:40:43.443936Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:48Z"} 2024-03-04T17:40:48.748 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574043.443936_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:40:43.443936Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:48Z"} 2024-03-04T17:40:48.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574043.443936_99.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:40:43.443936Z', 'realm_epoch': 2}}] 2024-03-04T17:40:48.749 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574043.443936_99.1'} 2024-03-04T17:40:48.749 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574043.443936_99.1'} 2024-03-04T17:40:48.749 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:40:48.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:40:48.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:40:48.749 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:40:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:48.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:40:48.862+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:40:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:40:53.862+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:40:58.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:40:58.862+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:41:03.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:41:03.862+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:41:08.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:41:08.866+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:41:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:41:13.866+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:41:18.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:41:18.866+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:41:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:41:23.866+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:41:28.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:41:28.870+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:41:33.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:41:33.870+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:41:38.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:41:38.869+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:41:43.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:41:43.873+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:41:48.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:41:48.873+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:41:53.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:41:53.873+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:41:58.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:41:58.877+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:42:03.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:42:03.877+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:42:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:42:08.877+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:42:13.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:42:13.877+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:42:18.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:42:18.881+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:42:23.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:42:23.881+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:42:24.394 INFO:tasks.rgw.c2.client.0.smithi153.stdout:2024-03-04T17:42:24.393+0000 7fa53962a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:42:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:42:28.881+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:42:33.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:42:33.885+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:42:38.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:42:38.885+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:42:43.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:42:43.885+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:42:48.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:42:48.885+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:42:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:42:53.889+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:42:58.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:42:58.888+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:43:03.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:43:03.892+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:43:08.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:43:08.892+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:43:13.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:43:13.892+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:43:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:43:18.892+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:43:23.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:43:23.892+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:43:28.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:43:28.896+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:43:33.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:43:33.896+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:43:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:43:38.900+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:43:43.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:43:43.900+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:43:48.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:43:48.900+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:43:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:43:53.900+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:43:58.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:43:58.904+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:44:03.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:44:03.904+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:44:08.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:44:08.904+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:44:13.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:44:13.908+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:44:18.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:44:18.908+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:44:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:44:23.911+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:44:28.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:44:28.907+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:44:33.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:44:33.911+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:44:38.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:44:38.911+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:44:43.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:44:43.911+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:44:48.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:44:48.915+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:44:53.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:44:53.919+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:44:58.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:44:58.915+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:45:03.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:03.919+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:45:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:08.923+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:45:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:13.919+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:45:18.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:18.923+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:45:23.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:23.927+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:45:28.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:28.927+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:45:33.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:33.931+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:45:38.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:38.931+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T17:45:43.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:43.930+0000 7f5debcbc980 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-03-04T17:45:43.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:43.930+0000 7f5debcbc980 0 bucket sync checkpoint failed: (110) Connection timed out 2024-03-04T17:45:43.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:43.930+0000 7f5debcbc980 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-03-04T17:45:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-03-04T17:45:43.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_object_delete ... FAIL 2024-03-04T17:45:43.979 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-19 2024-03-04T17:45:43.979 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:45:43.979 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:45:43.980 DEBUG:boto:Method: PUT 2024-03-04T17:45:43.980 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:45:43.980 DEBUG:boto:Data: 2024-03-04T17:45:43.980 DEBUG:boto:Headers: {} 2024-03-04T17:45:43.980 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:45:43.980 DEBUG:boto:Port: 8000 2024-03-04T17:45:43.980 DEBUG:boto:Params: {} 2024-03-04T17:45:43.980 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:45:43.981 DEBUG:boto:Token: None 2024-03-04T17:45:43.981 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:45:43 GMT /ppfahn-19/ 2024-03-04T17:45:43.981 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kMAxUXKPD/AO+GZpJ+2UMoh/x1w= 2024-03-04T17:45:43.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:45:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kMAxUXKPD/AO+GZpJ+2UMoh/x1w=', 'Content-Length': '0'} 2024-03-04T17:45:44.017 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a667a1b1a1b549af-0065e608c7-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:45:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:45:44.018 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-20 2024-03-04T17:45:44.018 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:45:44.018 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:45:44.018 DEBUG:boto:Method: PUT 2024-03-04T17:45:44.018 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:45:44.018 DEBUG:boto:Data: 2024-03-04T17:45:44.018 DEBUG:boto:Headers: {} 2024-03-04T17:45:44.018 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:45:44.018 DEBUG:boto:Port: 8000 2024-03-04T17:45:44.018 DEBUG:boto:Params: {} 2024-03-04T17:45:44.019 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:45:44.019 DEBUG:boto:Token: None 2024-03-04T17:45:44.019 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:45:44 GMT /ppfahn-20/ 2024-03-04T17:45:44.019 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mrkgyvAvYkkjsuhWrOED6gl93ko= 2024-03-04T17:45:44.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:45:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mrkgyvAvYkkjsuhWrOED6gl93ko=', 'Content-Length': '0'} 2024-03-04T17:45:44.085 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a15e6573f3362098-0065e608c8-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:45:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:45:44.085 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-21 2024-03-04T17:45:44.085 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:45:44.085 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:45:44.085 DEBUG:boto:Method: PUT 2024-03-04T17:45:44.085 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:45:44.086 DEBUG:boto:Data: 2024-03-04T17:45:44.086 DEBUG:boto:Headers: {} 2024-03-04T17:45:44.086 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:45:44.086 DEBUG:boto:Port: 8001 2024-03-04T17:45:44.086 DEBUG:boto:Params: {} 2024-03-04T17:45:44.086 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T17:45:44.086 DEBUG:boto:Token: None 2024-03-04T17:45:44.086 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:45:44 GMT /ppfahn-21/ 2024-03-04T17:45:44.086 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WdMKAmcu+c372SIufALz51KkAUI= 2024-03-04T17:45:44.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:45:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WdMKAmcu+c372SIufALz51KkAUI=', 'Content-Length': '0'} 2024-03-04T17:45:44.144 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006b11fd12a7b554ea-0065e608c8-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:45:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:45:44.144 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:45:44.145 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:45:44.145 DEBUG:boto:path=/ppfahn-19/?versioning 2024-03-04T17:45:44.145 DEBUG:boto:auth_path=/ppfahn-19/?versioning 2024-03-04T17:45:44.145 DEBUG:boto:Method: PUT 2024-03-04T17:45:44.145 DEBUG:boto:Path: /ppfahn-19/?versioning 2024-03-04T17:45:44.145 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:45:44.145 DEBUG:boto:Headers: {} 2024-03-04T17:45:44.145 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:45:44.145 DEBUG:boto:Port: 8000 2024-03-04T17:45:44.145 DEBUG:boto:Params: {} 2024-03-04T17:45:44.146 DEBUG:boto:Token: None 2024-03-04T17:45:44.146 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:45:44 GMT /ppfahn-19/?versioning 2024-03-04T17:45:44.146 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:t36eqwr8rNAHYYyza8sYDpfDTd0= 2024-03-04T17:45:44.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:45:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:t36eqwr8rNAHYYyza8sYDpfDTd0=', 'Content-Length': '228'} 2024-03-04T17:45:44.165 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000063160697e9bf1c2b-0065e608c8-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:45:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:45:44.166 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:45:44.166 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:45:44.166 DEBUG:boto:path=/ppfahn-20/?versioning 2024-03-04T17:45:44.166 DEBUG:boto:auth_path=/ppfahn-20/?versioning 2024-03-04T17:45:44.166 DEBUG:boto:Method: PUT 2024-03-04T17:45:44.166 DEBUG:boto:Path: /ppfahn-20/?versioning 2024-03-04T17:45:44.166 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:45:44.166 DEBUG:boto:Headers: {} 2024-03-04T17:45:44.166 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:45:44.166 DEBUG:boto:Port: 8000 2024-03-04T17:45:44.167 DEBUG:boto:Params: {} 2024-03-04T17:45:44.167 DEBUG:boto:Token: None 2024-03-04T17:45:44.167 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:45:44 GMT /ppfahn-20/?versioning 2024-03-04T17:45:44.167 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:q6fISkDjwU3nZi4tNGIrdnBszZg= 2024-03-04T17:45:44.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:45:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:q6fISkDjwU3nZi4tNGIrdnBszZg=', 'Content-Length': '228'} 2024-03-04T17:45:44.179 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a48882fcea7fc871-0065e608c8-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:45:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:45:44.179 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:45:44.179 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:45:44.180 DEBUG:boto:path=/ppfahn-21/?versioning 2024-03-04T17:45:44.180 DEBUG:boto:auth_path=/ppfahn-21/?versioning 2024-03-04T17:45:44.180 DEBUG:boto:Method: PUT 2024-03-04T17:45:44.180 DEBUG:boto:Path: /ppfahn-21/?versioning 2024-03-04T17:45:44.180 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:45:44.180 DEBUG:boto:Headers: {} 2024-03-04T17:45:44.180 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:45:44.180 DEBUG:boto:Port: 8001 2024-03-04T17:45:44.180 DEBUG:boto:Params: {} 2024-03-04T17:45:44.180 DEBUG:boto:Token: None 2024-03-04T17:45:44.181 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:45:44 GMT /ppfahn-21/?versioning 2024-03-04T17:45:44.181 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UhFFRsyxl8ypE8bIq8PWmjh5NNo= 2024-03-04T17:45:44.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:45:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UhFFRsyxl8ypE8bIq8PWmjh5NNo=', 'Content-Length': '228'} 2024-03-04T17:45:44.192 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000053bb77c6a27c8f5c-0065e608c8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:45:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:45:44.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:44.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:44.193 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:44.231 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:44.231 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:44.289 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:45:44.313 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709574344.195836_145.1","last_update":"2024-03-04T17:45:44.195836Z"}] 2024-03-04T17:45:44.314 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574344.195836_145.1'} 2024-03-04T17:45:44.314 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:45:44.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:44.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:44.314 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:44.361 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:44.361 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:44.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:44.422+0000 7faf9bab8980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:44.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:44.422+0000 7faf9bab8980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:44.443 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574344.119136_137.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:45:44.119136Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:44Z"} 2024-03-04T17:45:44.443 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574344.119136_137.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:45:44.119136Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:44Z"} 2024-03-04T17:45:44.444 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574344.119136_137.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:45:44.119136Z', 'realm_epoch': 2}}] 2024-03-04T17:45:44.444 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574344.195836_145.1'} 2024-03-04T17:45:44.444 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574344.119136_137.1'} 2024-03-04T17:45:44.444 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709574344.195836_145.1 target=1_1709574344.119136_137.1 2024-03-04T17:45:49.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:49.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:49.445 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:49.489 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:49.489 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:49.546+0000 7fdbc65b0980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:49.546+0000 7fdbc65b0980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:49.568 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574344.195836_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:45:44.195836Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:49Z"} 2024-03-04T17:45:49.569 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574344.195836_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:45:44.195836Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:49Z"} 2024-03-04T17:45:49.569 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574344.195836_145.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:45:44.195836Z', 'realm_epoch': 2}}] 2024-03-04T17:45:49.569 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574344.195836_145.1'} 2024-03-04T17:45:49.569 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574344.195836_145.1'} 2024-03-04T17:45:49.569 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:45:49.569 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:45:49.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:49.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:49.570 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:49.613 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:49.613 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:49.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:45:49.680+0000 7f1ab95a5980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:49.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:45:49.684+0000 7f1ab95a5980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:49.700 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574344.003896_128.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:45:44.003896Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:49Z"} 2024-03-04T17:45:49.700 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574344.003896_128.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:45:44.003896Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:49Z"} 2024-03-04T17:45:49.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574344.003896_128.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:45:44.003896Z', 'realm_epoch': 2}}] 2024-03-04T17:45:49.700 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574344.195836_145.1'} 2024-03-04T17:45:49.700 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574344.003896_128.1'} 2024-03-04T17:45:49.701 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1709574344.195836_145.1 target=1_1709574344.003896_128.1 2024-03-04T17:45:54.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:54.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:54.702 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:54.745 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:54.745 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:54.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:45:54.804+0000 7fc62515a980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:54.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:45:54.804+0000 7fc62515a980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:54.822 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574344.195836_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:45:44.195836Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:54Z"} 2024-03-04T17:45:54.823 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574344.195836_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:45:44.195836Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:54Z"} 2024-03-04T17:45:54.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574344.195836_145.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:45:44.195836Z', 'realm_epoch': 2}}] 2024-03-04T17:45:54.823 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574344.195836_145.1'} 2024-03-04T17:45:54.823 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574344.195836_145.1'} 2024-03-04T17:45:54.823 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:45:54.823 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:45:54.824 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:45:54.824 DEBUG:boto:Method: HEAD 2024-03-04T17:45:54.824 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:45:54.824 DEBUG:boto:Data: 2024-03-04T17:45:54.824 DEBUG:boto:Headers: {} 2024-03-04T17:45:54.824 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:45:54.824 DEBUG:boto:Port: 8000 2024-03-04T17:45:54.824 DEBUG:boto:Params: {} 2024-03-04T17:45:54.824 DEBUG:boto:Token: None 2024-03-04T17:45:54.825 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:45:54 GMT /ppfahn-19/ 2024-03-04T17:45:54.825 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H6SFNhV3gUhnQSc0+UblHsPbdGU= 2024-03-04T17:45:54.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:45:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H6SFNhV3gUhnQSc0+UblHsPbdGU=', 'Content-Length': '0'} 2024-03-04T17:45:54.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e2ac7efbcae0c61-0065e608d2-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:45:54.828 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:45:54.828 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:45:54.828 DEBUG:boto:Method: PUT 2024-03-04T17:45:54.828 DEBUG:boto:Path: /ppfahn-19/dummy 2024-03-04T17:45:54.828 DEBUG:boto:Data: 2024-03-04T17:45:54.828 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T17:45:54.829 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:45:54.829 DEBUG:boto:Port: 8000 2024-03-04T17:45:54.829 DEBUG:boto:Params: {} 2024-03-04T17:45:54.829 DEBUG:boto:Token: None 2024-03-04T17:45:54.829 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 17:45:54 GMT /ppfahn-19/dummy 2024-03-04T17:45:54.829 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l0a1cI2sp+1KTljz5GBeNcIHPCM= 2024-03-04T17:45:54.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:45:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l0a1cI2sp+1KTljz5GBeNcIHPCM='} 2024-03-04T17:45:54.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-amz-request-id', 'tx000006684b05a089ab96a-0065e608d2-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:45:54.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:45:54.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:45:54.842 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:45:54.893 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:54.893 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:54.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:54.958+0000 7f8a8f208980 1 waiting to reach incremental sync.. 2024-03-04T17:45:59.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:45:59.958+0000 7f8a8f208980 1 waiting to reach incremental sync.. 2024-03-04T17:46:04.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:46:04.962+0000 7f8a8f208980 1 bucket sync caught up with source: 2024-03-04T17:46:04.960 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000002.15.11, , , , , ] 2024-03-04T17:46:04.960 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000002.15.11, , , , , ] 2024-03-04T17:46:04.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:46:04.962+0000 7f8a8f208980 0 bucket checkpoint complete 2024-03-04T17:46:05.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:05.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:05.004 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:46:05.055 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:05.056 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:05.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:05.120+0000 7ff252159980 1 waiting to reach incremental sync.. 2024-03-04T17:46:10.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:10.124+0000 7ff252159980 1 bucket sync caught up with source: 2024-03-04T17:46:10.120 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000002.15.11, , , , , ] 2024-03-04T17:46:10.120 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000002.15.11, , , , , ] 2024-03-04T17:46:10.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:10.124+0000 7ff252159980 0 bucket checkpoint complete 2024-03-04T17:46:10.160 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:46:10.160 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:46:10.160 DEBUG:boto:Method: HEAD 2024-03-04T17:46:10.161 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:46:10.161 DEBUG:boto:Data: 2024-03-04T17:46:10.161 DEBUG:boto:Headers: {} 2024-03-04T17:46:10.161 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:10.161 DEBUG:boto:Port: 8000 2024-03-04T17:46:10.161 DEBUG:boto:Params: {} 2024-03-04T17:46:10.161 DEBUG:boto:Token: None 2024-03-04T17:46:10.161 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:46:10 GMT /ppfahn-20/ 2024-03-04T17:46:10.161 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PZOe11wWyYn4h1PIvRfxSJ4Tbi8= 2024-03-04T17:46:10.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PZOe11wWyYn4h1PIvRfxSJ4Tbi8=', 'Content-Length': '0'} 2024-03-04T17:46:10.166 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fdf176797770423c-0065e608e2-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:10.166 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:46:10.166 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:46:10.167 DEBUG:boto:Method: PUT 2024-03-04T17:46:10.167 DEBUG:boto:Path: /ppfahn-20/dummy 2024-03-04T17:46:10.167 DEBUG:boto:Data: 2024-03-04T17:46:10.167 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T17:46:10.167 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:10.167 DEBUG:boto:Port: 8000 2024-03-04T17:46:10.167 DEBUG:boto:Params: {} 2024-03-04T17:46:10.167 DEBUG:boto:Token: None 2024-03-04T17:46:10.167 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 17:46:10 GMT /ppfahn-20/dummy 2024-03-04T17:46:10.167 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lXVEBp0+ENildwxI4jUipFFnTpA= 2024-03-04T17:46:10.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:10 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lXVEBp0+ENildwxI4jUipFFnTpA='} 2024-03-04T17:46:10.195 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-amz-request-id', 'tx00000cb41d8dbf304a492-0065e608e2-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:10.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:10.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:10.196 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:46:10.245 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:10.245 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:10.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:10.316+0000 7f26ba29b980 1 waiting to reach incremental sync.. 2024-03-04T17:46:15.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:15.316+0000 7f26ba29b980 1 waiting to reach incremental sync.. 2024-03-04T17:46:20.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:20.316+0000 7f26ba29b980 1 waiting to reach incremental sync.. 2024-03-04T17:46:25.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:25.319+0000 7f26ba29b980 1 bucket sync caught up with source: 2024-03-04T17:46:25.316 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000002.46.11, , , , , ] 2024-03-04T17:46:25.316 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000002.46.11, , , , , ] 2024-03-04T17:46:25.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:25.319+0000 7f26ba29b980 0 bucket checkpoint complete 2024-03-04T17:46:25.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:25.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:25.361 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:46:25.411 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:25.411 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:25.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:25.479+0000 7f31159e1980 1 waiting to reach incremental sync.. 2024-03-04T17:46:30.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:30.483+0000 7f31159e1980 1 bucket sync caught up with source: 2024-03-04T17:46:30.479 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000003.48.10, , , , , ] 2024-03-04T17:46:30.479 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000002.46.11, , , , , ] 2024-03-04T17:46:30.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:30.483+0000 7f31159e1980 0 bucket checkpoint complete 2024-03-04T17:46:30.519 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:46:30.519 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:46:30.519 DEBUG:boto:Method: HEAD 2024-03-04T17:46:30.519 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:46:30.519 DEBUG:boto:Data: 2024-03-04T17:46:30.519 DEBUG:boto:Headers: {} 2024-03-04T17:46:30.519 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:30.519 DEBUG:boto:Port: 8001 2024-03-04T17:46:30.519 DEBUG:boto:Params: {} 2024-03-04T17:46:30.520 DEBUG:boto:Token: None 2024-03-04T17:46:30.520 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:46:30 GMT /ppfahn-21/ 2024-03-04T17:46:30.520 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nLNHwcQLIVNr2IrT7z6OGJupYBA= 2024-03-04T17:46:30.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nLNHwcQLIVNr2IrT7z6OGJupYBA=', 'Content-Length': '0'} 2024-03-04T17:46:30.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000414ec8b0b1799e1a-0065e608f6-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:30.524 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:46:30.525 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:46:30.525 DEBUG:boto:Method: PUT 2024-03-04T17:46:30.525 DEBUG:boto:Path: /ppfahn-21/dummy 2024-03-04T17:46:30.525 DEBUG:boto:Data: 2024-03-04T17:46:30.525 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T17:46:30.525 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:30.525 DEBUG:boto:Port: 8001 2024-03-04T17:46:30.525 DEBUG:boto:Params: {} 2024-03-04T17:46:30.525 DEBUG:boto:Token: None 2024-03-04T17:46:30.525 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 17:46:30 GMT /ppfahn-21/dummy 2024-03-04T17:46:30.526 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jg2J7j6+XTr7TlxL66lLagavlfw= 2024-03-04T17:46:30.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jg2J7j6+XTr7TlxL66lLagavlfw='} 2024-03-04T17:46:30.543 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-amz-request-id', 'tx000009cf971aa85478757-0065e608f6-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:30.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:30.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:30.544 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:46:30.592 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:30.592 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:30.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:30.655+0000 7f26f623e980 1 waiting to reach incremental sync.. 2024-03-04T17:46:35.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:35.655+0000 7f26f623e980 1 waiting to reach incremental sync.. 2024-03-04T17:46:40.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:40.659+0000 7f26f623e980 1 waiting to reach incremental sync.. 2024-03-04T17:46:45.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:45.659+0000 7f26f623e980 1 bucket sync caught up with source: 2024-03-04T17:46:45.659 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000002.63.11, , , , , ] 2024-03-04T17:46:45.659 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000002.63.11, , , , , ] 2024-03-04T17:46:45.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:45.659+0000 7f26f623e980 0 bucket checkpoint complete 2024-03-04T17:46:45.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:45.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:45.697 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:46:45.747 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:45.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:46:45.810+0000 7f8767a25980 1 bucket sync caught up with source: 2024-03-04T17:46:45.810 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000002.63.11, , , , , ] 2024-03-04T17:46:45.810 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000002.63.11, , , , , ] 2024-03-04T17:46:45.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:46:45.810+0000 7f8767a25980 0 bucket checkpoint complete 2024-03-04T17:46:45.833 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:46:45.833 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:46:45.833 DEBUG:boto:Method: HEAD 2024-03-04T17:46:45.833 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:46:45.833 DEBUG:boto:Data: 2024-03-04T17:46:45.833 DEBUG:boto:Headers: {} 2024-03-04T17:46:45.833 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:45.834 DEBUG:boto:Port: 8000 2024-03-04T17:46:45.834 DEBUG:boto:Params: {} 2024-03-04T17:46:45.834 DEBUG:boto:Token: None 2024-03-04T17:46:45.834 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:46:45 GMT /ppfahn-19/ 2024-03-04T17:46:45.834 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kHsrBkJSaY50q4XsquiFgZA2arg= 2024-03-04T17:46:45.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kHsrBkJSaY50q4XsquiFgZA2arg=', 'Content-Length': '0'} 2024-03-04T17:46:45.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003338103cfe84a077-0065e60905-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:45.839 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.839 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.839 DEBUG:boto:Method: PUT 2024-03-04T17:46:45.839 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.839 DEBUG:boto:Data: 2024-03-04T17:46:45.839 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:45.839 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:45.839 DEBUG:boto:Port: 8000 2024-03-04T17:46:45.839 DEBUG:boto:Params: {} 2024-03-04T17:46:45.840 DEBUG:boto:Token: None 2024-03-04T17:46:45.840 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:46:45 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.840 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IpNxAlPr9CedvClJSA1FjXEcBp4= 2024-03-04T17:46:45.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IpNxAlPr9CedvClJSA1FjXEcBp4='} 2024-03-04T17:46:45.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-amz-request-id', 'tx0000010e27c0c46f29415-0065e60905-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:45.864 DEBUG:rgw_multi.tests:version1 id=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:46:45.864 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.864 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.864 DEBUG:boto:Method: PUT 2024-03-04T17:46:45.864 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.864 DEBUG:boto:Data: 2024-03-04T17:46:45.865 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:45.865 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:45.865 DEBUG:boto:Port: 8000 2024-03-04T17:46:45.865 DEBUG:boto:Params: {} 2024-03-04T17:46:45.865 DEBUG:boto:Token: None 2024-03-04T17:46:45.865 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 17:46:45 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.865 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fuDLZGq6AkkjPr8D0Grr52Yuh1k= 2024-03-04T17:46:45.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fuDLZGq6AkkjPr8D0Grr52Yuh1k='} 2024-03-04T17:46:45.887 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hU-HHkEgaS4k8Z--6rpL96JUAFiPcoa'), ('x-amz-request-id', 'tx000001d957b4e8b4f71d5-0065e60905-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:45.887 DEBUG:rgw_multi.tests:version2 id=hU-HHkEgaS4k8Z--6rpL96JUAFiPcoa 2024-03-04T17:46:45.887 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.887 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.887 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=hU-HHkEgaS4k8Z--6rpL96JUAFiPcoa 2024-03-04T17:46:45.887 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=hU-HHkEgaS4k8Z--6rpL96JUAFiPcoa 2024-03-04T17:46:45.888 DEBUG:boto:Method: DELETE 2024-03-04T17:46:45.888 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=hU-HHkEgaS4k8Z--6rpL96JUAFiPcoa 2024-03-04T17:46:45.888 DEBUG:boto:Data: 2024-03-04T17:46:45.888 DEBUG:boto:Headers: {} 2024-03-04T17:46:45.888 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:45.888 DEBUG:boto:Port: 8000 2024-03-04T17:46:45.888 DEBUG:boto:Params: {} 2024-03-04T17:46:45.888 DEBUG:boto:Token: None 2024-03-04T17:46:45.888 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:45 GMT /ppfahn-19/obj-test-zone1?versionId=hU-HHkEgaS4k8Z--6rpL96JUAFiPcoa 2024-03-04T17:46:45.889 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pmkHoHk3fec3g7/ih82+TNON/g0= 2024-03-04T17:46:45.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pmkHoHk3fec3g7/ih82+TNON/g0=', 'Content-Length': '0'} 2024-03-04T17:46:45.910 DEBUG:boto:Response headers: [('x-amz-version-id', 'hU-HHkEgaS4k8Z--6rpL96JUAFiPcoa'), ('x-amz-request-id', 'tx0000071dde22774df52f5-0065e60905-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:45.911 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.911 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.911 DEBUG:boto:Method: PUT 2024-03-04T17:46:45.911 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.911 DEBUG:boto:Data: 2024-03-04T17:46:45.911 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:45.911 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:45.911 DEBUG:boto:Port: 8000 2024-03-04T17:46:45.912 DEBUG:boto:Params: {} 2024-03-04T17:46:45.912 DEBUG:boto:Token: None 2024-03-04T17:46:45.912 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 17:46:45 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.912 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gA5Bpm2tguxdTO7oWDPNIZa8Uh0= 2024-03-04T17:46:45.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gA5Bpm2tguxdTO7oWDPNIZa8Uh0='} 2024-03-04T17:46:45.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'wGsox7Rg8-uDvcKcU11XDKdhTWrn4A7'), ('x-amz-request-id', 'tx00000a6a50e4da20a8306-0065e60905-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:45.960 DEBUG:rgw_multi.tests:version3 id=wGsox7Rg8-uDvcKcU11XDKdhTWrn4A7 2024-03-04T17:46:45.961 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.961 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:46:45.961 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=wGsox7Rg8-uDvcKcU11XDKdhTWrn4A7 2024-03-04T17:46:45.961 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=wGsox7Rg8-uDvcKcU11XDKdhTWrn4A7 2024-03-04T17:46:45.961 DEBUG:boto:Method: DELETE 2024-03-04T17:46:45.961 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=wGsox7Rg8-uDvcKcU11XDKdhTWrn4A7 2024-03-04T17:46:45.961 DEBUG:boto:Data: 2024-03-04T17:46:45.961 DEBUG:boto:Headers: {} 2024-03-04T17:46:45.961 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:45.961 DEBUG:boto:Port: 8000 2024-03-04T17:46:45.961 DEBUG:boto:Params: {} 2024-03-04T17:46:45.962 DEBUG:boto:Token: None 2024-03-04T17:46:45.962 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:45 GMT /ppfahn-19/obj-test-zone1?versionId=wGsox7Rg8-uDvcKcU11XDKdhTWrn4A7 2024-03-04T17:46:45.962 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F3dlduNqIhRxF9WdnKZrEFy/l00= 2024-03-04T17:46:45.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F3dlduNqIhRxF9WdnKZrEFy/l00=', 'Content-Length': '0'} 2024-03-04T17:46:45.984 DEBUG:boto:Response headers: [('x-amz-version-id', 'wGsox7Rg8-uDvcKcU11XDKdhTWrn4A7'), ('x-amz-request-id', 'tx00000d6a300b7f47fc9d6-0065e60905-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:45.985 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:46:45.985 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:46:45.985 DEBUG:boto:Method: HEAD 2024-03-04T17:46:45.985 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:46:45.985 DEBUG:boto:Data: 2024-03-04T17:46:45.985 DEBUG:boto:Headers: {} 2024-03-04T17:46:45.985 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:45.985 DEBUG:boto:Port: 8000 2024-03-04T17:46:45.985 DEBUG:boto:Params: {} 2024-03-04T17:46:45.985 DEBUG:boto:Token: None 2024-03-04T17:46:45.986 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:46:45 GMT /ppfahn-19/ 2024-03-04T17:46:45.986 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kHsrBkJSaY50q4XsquiFgZA2arg= 2024-03-04T17:46:45.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kHsrBkJSaY50q4XsquiFgZA2arg=', 'Content-Length': '0'} 2024-03-04T17:46:45.990 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037bcd1143b92a9e5-0065e60905-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:45.990 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:46:45.990 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:46:45.990 DEBUG:boto:Method: PUT 2024-03-04T17:46:45.991 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:46:45.991 DEBUG:boto:Data: 2024-03-04T17:46:45.991 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:45.991 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:45.991 DEBUG:boto:Port: 8000 2024-03-04T17:46:45.991 DEBUG:boto:Params: {} 2024-03-04T17:46:45.991 DEBUG:boto:Token: None 2024-03-04T17:46:45.991 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:46:45 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:46:45.991 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nSsD5Rv0XEgqvDRX1/2C5VFOY/o= 2024-03-04T17:46:45.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nSsD5Rv0XEgqvDRX1/2C5VFOY/o='} 2024-03-04T17:46:46.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-amz-request-id', 'tx00000cc21b0173edc0d85-0065e60905-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.016 DEBUG:rgw_multi.tests:version1 id=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:46:46.016 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:46:46.017 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:46:46.017 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.017 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:46:46.017 DEBUG:boto:Data: 2024-03-04T17:46:46.017 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.017 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.017 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.017 DEBUG:boto:Params: {} 2024-03-04T17:46:46.017 DEBUG:boto:Token: None 2024-03-04T17:46:46.017 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:46:46.018 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DXfZdDYVgLiFxYB9jvwT1UpQnL4= 2024-03-04T17:46:46.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DXfZdDYVgLiFxYB9jvwT1UpQnL4='} 2024-03-04T17:46:46.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zpSXDR7yuaMP6mRYu1t8wjSVkpLJhMc'), ('x-amz-request-id', 'tx000005cae1f2ae4496917-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.060 DEBUG:rgw_multi.tests:version2 id=zpSXDR7yuaMP6mRYu1t8wjSVkpLJhMc 2024-03-04T17:46:46.060 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:46:46.060 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:46:46.061 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=zpSXDR7yuaMP6mRYu1t8wjSVkpLJhMc 2024-03-04T17:46:46.061 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=zpSXDR7yuaMP6mRYu1t8wjSVkpLJhMc 2024-03-04T17:46:46.061 DEBUG:boto:Method: DELETE 2024-03-04T17:46:46.061 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=zpSXDR7yuaMP6mRYu1t8wjSVkpLJhMc 2024-03-04T17:46:46.061 DEBUG:boto:Data: 2024-03-04T17:46:46.061 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.061 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.061 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.061 DEBUG:boto:Params: {} 2024-03-04T17:46:46.061 DEBUG:boto:Token: None 2024-03-04T17:46:46.061 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-19/obj-test-zone2?versionId=zpSXDR7yuaMP6mRYu1t8wjSVkpLJhMc 2024-03-04T17:46:46.062 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IHWPfvfv1oefuonBnTd64ksCvG8= 2024-03-04T17:46:46.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IHWPfvfv1oefuonBnTd64ksCvG8=', 'Content-Length': '0'} 2024-03-04T17:46:46.087 DEBUG:boto:Response headers: [('x-amz-version-id', 'zpSXDR7yuaMP6mRYu1t8wjSVkpLJhMc'), ('x-amz-request-id', 'tx00000330775029108d1dc-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.087 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:46:46.087 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:46:46.087 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.087 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:46:46.087 DEBUG:boto:Data: 2024-03-04T17:46:46.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.087 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.088 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.088 DEBUG:boto:Params: {} 2024-03-04T17:46:46.088 DEBUG:boto:Token: None 2024-03-04T17:46:46.088 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:46:46.088 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:34RRIRhfsn7VZSIP8ShTlO+w8Aw= 2024-03-04T17:46:46.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:34RRIRhfsn7VZSIP8ShTlO+w8Aw='} 2024-03-04T17:46:46.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WayljgVsMfxUTUhm52.uvPtUIgzCqwQ'), ('x-amz-request-id', 'tx00000d882a266eee11558-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.109 DEBUG:rgw_multi.tests:version3 id=WayljgVsMfxUTUhm52.uvPtUIgzCqwQ 2024-03-04T17:46:46.110 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:46:46.110 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:46:46.110 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=WayljgVsMfxUTUhm52.uvPtUIgzCqwQ 2024-03-04T17:46:46.110 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=WayljgVsMfxUTUhm52.uvPtUIgzCqwQ 2024-03-04T17:46:46.110 DEBUG:boto:Method: DELETE 2024-03-04T17:46:46.110 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=WayljgVsMfxUTUhm52.uvPtUIgzCqwQ 2024-03-04T17:46:46.110 DEBUG:boto:Data: 2024-03-04T17:46:46.110 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.111 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.111 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.111 DEBUG:boto:Params: {} 2024-03-04T17:46:46.111 DEBUG:boto:Token: None 2024-03-04T17:46:46.111 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-19/obj-test-zone2?versionId=WayljgVsMfxUTUhm52.uvPtUIgzCqwQ 2024-03-04T17:46:46.111 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eK5JK0TCSi669AuZPF5S2ciadmg= 2024-03-04T17:46:46.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eK5JK0TCSi669AuZPF5S2ciadmg=', 'Content-Length': '0'} 2024-03-04T17:46:46.137 DEBUG:boto:Response headers: [('x-amz-version-id', 'WayljgVsMfxUTUhm52.uvPtUIgzCqwQ'), ('x-amz-request-id', 'tx00000d37f50ea4938ccb7-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.137 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:46:46.137 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:46:46.138 DEBUG:boto:Method: HEAD 2024-03-04T17:46:46.138 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:46:46.138 DEBUG:boto:Data: 2024-03-04T17:46:46.138 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.138 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:46.138 DEBUG:boto:Port: 8001 2024-03-04T17:46:46.138 DEBUG:boto:Params: {} 2024-03-04T17:46:46.138 DEBUG:boto:Token: None 2024-03-04T17:46:46.138 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-19/ 2024-03-04T17:46:46.138 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QoqjNm6UmyUdz6Z5llDXlZDzclU= 2024-03-04T17:46:46.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QoqjNm6UmyUdz6Z5llDXlZDzclU=', 'Content-Length': '0'} 2024-03-04T17:46:46.142 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b782c870f8628d16-0065e60906-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.143 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.143 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.143 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.143 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.143 DEBUG:boto:Data: 2024-03-04T17:46:46.143 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.143 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:46.143 DEBUG:boto:Port: 8001 2024-03-04T17:46:46.143 DEBUG:boto:Params: {} 2024-03-04T17:46:46.144 DEBUG:boto:Token: None 2024-03-04T17:46:46.144 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.144 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F3pJfVlrDdDDjPPH91IeOY6NkT0= 2024-03-04T17:46:46.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F3pJfVlrDdDDjPPH91IeOY6NkT0='} 2024-03-04T17:46:46.165 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-amz-request-id', 'tx0000005c0da5c597c7a24-0065e60906-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.165 DEBUG:rgw_multi.tests:version1 id=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:46:46.166 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.166 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.166 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.166 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.166 DEBUG:boto:Data: 2024-03-04T17:46:46.166 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.166 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:46.166 DEBUG:boto:Port: 8001 2024-03-04T17:46:46.166 DEBUG:boto:Params: {} 2024-03-04T17:46:46.167 DEBUG:boto:Token: None 2024-03-04T17:46:46.167 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.167 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SLaKe3DRXw/O1IRetZcmeCJ8lpw= 2024-03-04T17:46:46.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SLaKe3DRXw/O1IRetZcmeCJ8lpw='} 2024-03-04T17:46:46.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'HH-w345cfelHANGvkWATGRn0GitcVVS'), ('x-amz-request-id', 'tx0000019cedb50d0063489-0065e60906-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.212 DEBUG:rgw_multi.tests:version2 id=HH-w345cfelHANGvkWATGRn0GitcVVS 2024-03-04T17:46:46.213 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.213 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.213 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=HH-w345cfelHANGvkWATGRn0GitcVVS 2024-03-04T17:46:46.213 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=HH-w345cfelHANGvkWATGRn0GitcVVS 2024-03-04T17:46:46.213 DEBUG:boto:Method: DELETE 2024-03-04T17:46:46.213 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=HH-w345cfelHANGvkWATGRn0GitcVVS 2024-03-04T17:46:46.213 DEBUG:boto:Data: 2024-03-04T17:46:46.213 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.213 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:46.213 DEBUG:boto:Port: 8001 2024-03-04T17:46:46.213 DEBUG:boto:Params: {} 2024-03-04T17:46:46.214 DEBUG:boto:Token: None 2024-03-04T17:46:46.214 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-19/obj-test-zone3?versionId=HH-w345cfelHANGvkWATGRn0GitcVVS 2024-03-04T17:46:46.214 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wk1Qow2XSd1IHLdQ4qwv4dAzPiM= 2024-03-04T17:46:46.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wk1Qow2XSd1IHLdQ4qwv4dAzPiM=', 'Content-Length': '0'} 2024-03-04T17:46:46.237 DEBUG:boto:Response headers: [('x-amz-version-id', 'HH-w345cfelHANGvkWATGRn0GitcVVS'), ('x-amz-request-id', 'tx00000a600dbb6ecb0df5d-0065e60906-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.237 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.237 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.237 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.238 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.238 DEBUG:boto:Data: 2024-03-04T17:46:46.238 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.238 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:46.238 DEBUG:boto:Port: 8001 2024-03-04T17:46:46.238 DEBUG:boto:Params: {} 2024-03-04T17:46:46.238 DEBUG:boto:Token: None 2024-03-04T17:46:46.238 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.238 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:w049YAgbWX/4m27FI1MjagbbUu0= 2024-03-04T17:46:46.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:w049YAgbWX/4m27FI1MjagbbUu0='} 2024-03-04T17:46:46.261 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 't4cb5Rml0ov8p4v3nzjKGjRRX-QT4sE'), ('x-amz-request-id', 'tx00000ea37b26a34699305-0065e60906-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.261 DEBUG:rgw_multi.tests:version3 id=t4cb5Rml0ov8p4v3nzjKGjRRX-QT4sE 2024-03-04T17:46:46.261 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.261 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:46:46.261 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=t4cb5Rml0ov8p4v3nzjKGjRRX-QT4sE 2024-03-04T17:46:46.261 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=t4cb5Rml0ov8p4v3nzjKGjRRX-QT4sE 2024-03-04T17:46:46.261 DEBUG:boto:Method: DELETE 2024-03-04T17:46:46.262 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=t4cb5Rml0ov8p4v3nzjKGjRRX-QT4sE 2024-03-04T17:46:46.262 DEBUG:boto:Data: 2024-03-04T17:46:46.262 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.262 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:46.262 DEBUG:boto:Port: 8001 2024-03-04T17:46:46.262 DEBUG:boto:Params: {} 2024-03-04T17:46:46.262 DEBUG:boto:Token: None 2024-03-04T17:46:46.262 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-19/obj-test-zone3?versionId=t4cb5Rml0ov8p4v3nzjKGjRRX-QT4sE 2024-03-04T17:46:46.262 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:su7e7N7849Wqw3EXjmelN8qsus0= 2024-03-04T17:46:46.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:su7e7N7849Wqw3EXjmelN8qsus0=', 'Content-Length': '0'} 2024-03-04T17:46:46.285 DEBUG:boto:Response headers: [('x-amz-version-id', 't4cb5Rml0ov8p4v3nzjKGjRRX-QT4sE'), ('x-amz-request-id', 'tx00000b6a94f087064fa82-0065e60906-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.285 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:46:46.285 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:46:46.285 DEBUG:boto:Method: HEAD 2024-03-04T17:46:46.286 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:46:46.286 DEBUG:boto:Data: 2024-03-04T17:46:46.286 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.286 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:46.286 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.286 DEBUG:boto:Params: {} 2024-03-04T17:46:46.286 DEBUG:boto:Token: None 2024-03-04T17:46:46.286 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/ 2024-03-04T17:46:46.286 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XqMWAawdirZs6HdD8JJTBVt9MmA= 2024-03-04T17:46:46.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XqMWAawdirZs6HdD8JJTBVt9MmA=', 'Content-Length': '0'} 2024-03-04T17:46:46.290 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9bde81a9eae58a4-0065e60906-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.291 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.291 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.291 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.291 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.291 DEBUG:boto:Data: 2024-03-04T17:46:46.291 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.291 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:46.291 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.291 DEBUG:boto:Params: {} 2024-03-04T17:46:46.291 DEBUG:boto:Token: None 2024-03-04T17:46:46.292 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.292 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:n27M4+yZH4vVPTAUyT2KnrBtqRw= 2024-03-04T17:46:46.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:n27M4+yZH4vVPTAUyT2KnrBtqRw='} 2024-03-04T17:46:46.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-amz-request-id', 'tx000007e9b45640d6646a9-0065e60906-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.316 DEBUG:rgw_multi.tests:version1 id=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:46:46.316 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.316 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.316 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.317 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.317 DEBUG:boto:Data: 2024-03-04T17:46:46.317 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.317 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:46.317 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.317 DEBUG:boto:Params: {} 2024-03-04T17:46:46.317 DEBUG:boto:Token: None 2024-03-04T17:46:46.317 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.317 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:frCVmaeO4DAjPqQgTdwvpxSfjhI= 2024-03-04T17:46:46.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:frCVmaeO4DAjPqQgTdwvpxSfjhI='} 2024-03-04T17:46:46.360 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'E6oGY-iKtmt.m0OaQeIbtP25CVUxnQS'), ('x-amz-request-id', 'tx00000c9148ca723ed7c04-0065e60906-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.360 DEBUG:rgw_multi.tests:version2 id=E6oGY-iKtmt.m0OaQeIbtP25CVUxnQS 2024-03-04T17:46:46.360 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.361 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.361 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=E6oGY-iKtmt.m0OaQeIbtP25CVUxnQS 2024-03-04T17:46:46.361 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=E6oGY-iKtmt.m0OaQeIbtP25CVUxnQS 2024-03-04T17:46:46.361 DEBUG:boto:Method: DELETE 2024-03-04T17:46:46.361 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=E6oGY-iKtmt.m0OaQeIbtP25CVUxnQS 2024-03-04T17:46:46.361 DEBUG:boto:Data: 2024-03-04T17:46:46.361 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.361 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:46.361 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.361 DEBUG:boto:Params: {} 2024-03-04T17:46:46.362 DEBUG:boto:Token: None 2024-03-04T17:46:46.362 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone1?versionId=E6oGY-iKtmt.m0OaQeIbtP25CVUxnQS 2024-03-04T17:46:46.362 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/oMago1jc5CIxaAzSnmoFwCxak8= 2024-03-04T17:46:46.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/oMago1jc5CIxaAzSnmoFwCxak8=', 'Content-Length': '0'} 2024-03-04T17:46:46.385 DEBUG:boto:Response headers: [('x-amz-version-id', 'E6oGY-iKtmt.m0OaQeIbtP25CVUxnQS'), ('x-amz-request-id', 'tx000009aa1885fdb7cf5e7-0065e60906-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.386 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.386 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.386 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.386 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.386 DEBUG:boto:Data: 2024-03-04T17:46:46.386 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.387 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:46.387 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.387 DEBUG:boto:Params: {} 2024-03-04T17:46:46.387 DEBUG:boto:Token: None 2024-03-04T17:46:46.387 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.387 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Urc0pI6Bik61GfbQwVlKnUrQcg8= 2024-03-04T17:46:46.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Urc0pI6Bik61GfbQwVlKnUrQcg8='} 2024-03-04T17:46:46.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-AyIQVR7psaBjpSeB6QY--K.PFWxT.t'), ('x-amz-request-id', 'tx00000396254c1e7cc843c-0065e60906-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.409 DEBUG:rgw_multi.tests:version3 id=-AyIQVR7psaBjpSeB6QY--K.PFWxT.t 2024-03-04T17:46:46.409 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.409 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:46:46.409 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=-AyIQVR7psaBjpSeB6QY--K.PFWxT.t 2024-03-04T17:46:46.409 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=-AyIQVR7psaBjpSeB6QY--K.PFWxT.t 2024-03-04T17:46:46.410 DEBUG:boto:Method: DELETE 2024-03-04T17:46:46.410 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=-AyIQVR7psaBjpSeB6QY--K.PFWxT.t 2024-03-04T17:46:46.410 DEBUG:boto:Data: 2024-03-04T17:46:46.410 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.410 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:46.410 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.410 DEBUG:boto:Params: {} 2024-03-04T17:46:46.410 DEBUG:boto:Token: None 2024-03-04T17:46:46.410 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone1?versionId=-AyIQVR7psaBjpSeB6QY--K.PFWxT.t 2024-03-04T17:46:46.410 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4ZN2bec9lv/XXl9BkEDgPfnrD8E= 2024-03-04T17:46:46.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4ZN2bec9lv/XXl9BkEDgPfnrD8E=', 'Content-Length': '0'} 2024-03-04T17:46:46.434 DEBUG:boto:Response headers: [('x-amz-version-id', '-AyIQVR7psaBjpSeB6QY--K.PFWxT.t'), ('x-amz-request-id', 'tx00000f696a45f5dac312d-0065e60906-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.434 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:46:46.434 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:46:46.434 DEBUG:boto:Method: HEAD 2024-03-04T17:46:46.434 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:46:46.434 DEBUG:boto:Data: 2024-03-04T17:46:46.435 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.435 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.435 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.435 DEBUG:boto:Params: {} 2024-03-04T17:46:46.435 DEBUG:boto:Token: None 2024-03-04T17:46:46.435 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/ 2024-03-04T17:46:46.435 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XqMWAawdirZs6HdD8JJTBVt9MmA= 2024-03-04T17:46:46.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XqMWAawdirZs6HdD8JJTBVt9MmA=', 'Content-Length': '0'} 2024-03-04T17:46:46.440 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000285ca7289452b7e7-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.440 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.440 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.440 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.440 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.441 DEBUG:boto:Data: 2024-03-04T17:46:46.441 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.441 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.441 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.441 DEBUG:boto:Params: {} 2024-03-04T17:46:46.441 DEBUG:boto:Token: None 2024-03-04T17:46:46.441 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.441 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IZN+j3tPmCMf7y7H3o1oEdRmypA= 2024-03-04T17:46:46.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IZN+j3tPmCMf7y7H3o1oEdRmypA='} 2024-03-04T17:46:46.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-amz-request-id', 'tx0000083f59413cfc51b08-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.466 DEBUG:rgw_multi.tests:version1 id=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:46:46.467 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.467 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.467 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.467 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.467 DEBUG:boto:Data: 2024-03-04T17:46:46.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.467 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.467 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.467 DEBUG:boto:Params: {} 2024-03-04T17:46:46.468 DEBUG:boto:Token: None 2024-03-04T17:46:46.468 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.468 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TkAztZytuPD2U+1VgNmDwku3KIQ= 2024-03-04T17:46:46.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TkAztZytuPD2U+1VgNmDwku3KIQ='} 2024-03-04T17:46:46.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'he1cwGieKFBSt4RJp9K4QKbdMSJEzcy'), ('x-amz-request-id', 'tx000008cbfbd464d4a25cb-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.490 DEBUG:rgw_multi.tests:version2 id=he1cwGieKFBSt4RJp9K4QKbdMSJEzcy 2024-03-04T17:46:46.491 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.491 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.491 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=he1cwGieKFBSt4RJp9K4QKbdMSJEzcy 2024-03-04T17:46:46.491 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=he1cwGieKFBSt4RJp9K4QKbdMSJEzcy 2024-03-04T17:46:46.491 DEBUG:boto:Method: DELETE 2024-03-04T17:46:46.491 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=he1cwGieKFBSt4RJp9K4QKbdMSJEzcy 2024-03-04T17:46:46.491 DEBUG:boto:Data: 2024-03-04T17:46:46.491 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.491 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.491 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.492 DEBUG:boto:Params: {} 2024-03-04T17:46:46.492 DEBUG:boto:Token: None 2024-03-04T17:46:46.492 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone2?versionId=he1cwGieKFBSt4RJp9K4QKbdMSJEzcy 2024-03-04T17:46:46.492 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wl7I713nxo3jR7mnkE8Knp5ayC8= 2024-03-04T17:46:46.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wl7I713nxo3jR7mnkE8Knp5ayC8=', 'Content-Length': '0'} 2024-03-04T17:46:46.514 DEBUG:boto:Response headers: [('x-amz-version-id', 'he1cwGieKFBSt4RJp9K4QKbdMSJEzcy'), ('x-amz-request-id', 'tx00000bd9051c42591ffef-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.514 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.514 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.515 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.515 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.515 DEBUG:boto:Data: 2024-03-04T17:46:46.515 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.515 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.515 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.515 DEBUG:boto:Params: {} 2024-03-04T17:46:46.515 DEBUG:boto:Token: None 2024-03-04T17:46:46.515 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.516 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7fcXie+TtgD4Ov3nQrwd2mrMYZw= 2024-03-04T17:46:46.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7fcXie+TtgD4Ov3nQrwd2mrMYZw='} 2024-03-04T17:46:46.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-foQHOd2Bp0qA9t6EjpVQB-QDItdJlR'), ('x-amz-request-id', 'tx00000c5e45d0a0fa25e5d-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.560 DEBUG:rgw_multi.tests:version3 id=-foQHOd2Bp0qA9t6EjpVQB-QDItdJlR 2024-03-04T17:46:46.561 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.561 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:46:46.561 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-foQHOd2Bp0qA9t6EjpVQB-QDItdJlR 2024-03-04T17:46:46.561 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-foQHOd2Bp0qA9t6EjpVQB-QDItdJlR 2024-03-04T17:46:46.561 DEBUG:boto:Method: DELETE 2024-03-04T17:46:46.561 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-foQHOd2Bp0qA9t6EjpVQB-QDItdJlR 2024-03-04T17:46:46.561 DEBUG:boto:Data: 2024-03-04T17:46:46.561 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.561 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.561 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.561 DEBUG:boto:Params: {} 2024-03-04T17:46:46.562 DEBUG:boto:Token: None 2024-03-04T17:46:46.562 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone2?versionId=-foQHOd2Bp0qA9t6EjpVQB-QDItdJlR 2024-03-04T17:46:46.562 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yIu8msi2Mn+pNK4seOQfgsRXvIc= 2024-03-04T17:46:46.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yIu8msi2Mn+pNK4seOQfgsRXvIc=', 'Content-Length': '0'} 2024-03-04T17:46:46.585 DEBUG:boto:Response headers: [('x-amz-version-id', '-foQHOd2Bp0qA9t6EjpVQB-QDItdJlR'), ('x-amz-request-id', 'tx0000008bed5332e43045c-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.585 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:46:46.585 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:46:46.585 DEBUG:boto:Method: HEAD 2024-03-04T17:46:46.586 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:46:46.586 DEBUG:boto:Data: 2024-03-04T17:46:46.586 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.586 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:46.586 DEBUG:boto:Port: 8001 2024-03-04T17:46:46.586 DEBUG:boto:Params: {} 2024-03-04T17:46:46.586 DEBUG:boto:Token: None 2024-03-04T17:46:46.586 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/ 2024-03-04T17:46:46.586 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XqMWAawdirZs6HdD8JJTBVt9MmA= 2024-03-04T17:46:46.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XqMWAawdirZs6HdD8JJTBVt9MmA=', 'Content-Length': '0'} 2024-03-04T17:46:46.591 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f1f0adca188feed-0065e60906-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.591 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.591 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.591 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.591 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.591 DEBUG:boto:Data: 2024-03-04T17:46:46.591 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.591 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:46.591 DEBUG:boto:Port: 8001 2024-03-04T17:46:46.592 DEBUG:boto:Params: {} 2024-03-04T17:46:46.592 DEBUG:boto:Token: None 2024-03-04T17:46:46.592 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.592 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iDf84KDEZDeqKXTRVjMzCmdFx0c= 2024-03-04T17:46:46.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iDf84KDEZDeqKXTRVjMzCmdFx0c='} 2024-03-04T17:46:46.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-amz-request-id', 'tx00000ebca936651ffd320-0065e60906-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.614 DEBUG:rgw_multi.tests:version1 id=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:46:46.614 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.614 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.614 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.614 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.615 DEBUG:boto:Data: 2024-03-04T17:46:46.615 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.615 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:46.615 DEBUG:boto:Port: 8001 2024-03-04T17:46:46.615 DEBUG:boto:Params: {} 2024-03-04T17:46:46.615 DEBUG:boto:Token: None 2024-03-04T17:46:46.615 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.615 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Q0PeG+ZtvIgV1auI13mVDRL7Ubw= 2024-03-04T17:46:46.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Q0PeG+ZtvIgV1auI13mVDRL7Ubw='} 2024-03-04T17:46:46.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'bYQ99vdvHB.rkpaMzwMXGThvVPKOBPK'), ('x-amz-request-id', 'tx00000ef488521c71aa968-0065e60906-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.638 DEBUG:rgw_multi.tests:version2 id=bYQ99vdvHB.rkpaMzwMXGThvVPKOBPK 2024-03-04T17:46:46.638 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.638 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.638 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=bYQ99vdvHB.rkpaMzwMXGThvVPKOBPK 2024-03-04T17:46:46.639 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=bYQ99vdvHB.rkpaMzwMXGThvVPKOBPK 2024-03-04T17:46:46.639 DEBUG:boto:Method: DELETE 2024-03-04T17:46:46.639 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=bYQ99vdvHB.rkpaMzwMXGThvVPKOBPK 2024-03-04T17:46:46.639 DEBUG:boto:Data: 2024-03-04T17:46:46.639 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.639 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:46.639 DEBUG:boto:Port: 8001 2024-03-04T17:46:46.639 DEBUG:boto:Params: {} 2024-03-04T17:46:46.639 DEBUG:boto:Token: None 2024-03-04T17:46:46.639 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone3?versionId=bYQ99vdvHB.rkpaMzwMXGThvVPKOBPK 2024-03-04T17:46:46.640 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UFNo8GdBGMu7IkTcPPdr4/LVCIg= 2024-03-04T17:46:46.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UFNo8GdBGMu7IkTcPPdr4/LVCIg=', 'Content-Length': '0'} 2024-03-04T17:46:46.662 DEBUG:boto:Response headers: [('x-amz-version-id', 'bYQ99vdvHB.rkpaMzwMXGThvVPKOBPK'), ('x-amz-request-id', 'tx00000e42d887241ff8323-0065e60906-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.662 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.663 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.663 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.663 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.663 DEBUG:boto:Data: 2024-03-04T17:46:46.663 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.663 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:46.663 DEBUG:boto:Port: 8001 2024-03-04T17:46:46.663 DEBUG:boto:Params: {} 2024-03-04T17:46:46.663 DEBUG:boto:Token: None 2024-03-04T17:46:46.664 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.664 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f56jKlhp5LjRLLzQPo5o8W1CnM8= 2024-03-04T17:46:46.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f56jKlhp5LjRLLzQPo5o8W1CnM8='} 2024-03-04T17:46:46.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'yHcpf.syoalu8CYIpg2uPe9lU-5mzeb'), ('x-amz-request-id', 'tx00000e84cd320f642aaf4-0065e60906-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.708 DEBUG:rgw_multi.tests:version3 id=yHcpf.syoalu8CYIpg2uPe9lU-5mzeb 2024-03-04T17:46:46.709 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.709 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:46:46.709 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=yHcpf.syoalu8CYIpg2uPe9lU-5mzeb 2024-03-04T17:46:46.709 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=yHcpf.syoalu8CYIpg2uPe9lU-5mzeb 2024-03-04T17:46:46.709 DEBUG:boto:Method: DELETE 2024-03-04T17:46:46.709 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=yHcpf.syoalu8CYIpg2uPe9lU-5mzeb 2024-03-04T17:46:46.709 DEBUG:boto:Data: 2024-03-04T17:46:46.709 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.709 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:46.709 DEBUG:boto:Port: 8001 2024-03-04T17:46:46.710 DEBUG:boto:Params: {} 2024-03-04T17:46:46.710 DEBUG:boto:Token: None 2024-03-04T17:46:46.710 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-20/obj-test-zone3?versionId=yHcpf.syoalu8CYIpg2uPe9lU-5mzeb 2024-03-04T17:46:46.710 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MiRczXDO5S948ceh8pzcQdGv25I= 2024-03-04T17:46:46.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MiRczXDO5S948ceh8pzcQdGv25I=', 'Content-Length': '0'} 2024-03-04T17:46:46.732 DEBUG:boto:Response headers: [('x-amz-version-id', 'yHcpf.syoalu8CYIpg2uPe9lU-5mzeb'), ('x-amz-request-id', 'tx000008ed9e340ca4851dc-0065e60906-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.733 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:46:46.733 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:46:46.733 DEBUG:boto:Method: HEAD 2024-03-04T17:46:46.733 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:46:46.733 DEBUG:boto:Data: 2024-03-04T17:46:46.733 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.733 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:46.733 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.733 DEBUG:boto:Params: {} 2024-03-04T17:46:46.734 DEBUG:boto:Token: None 2024-03-04T17:46:46.734 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-21/ 2024-03-04T17:46:46.734 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5HQ/lsfA8V5v+4IXw+gm8iDuPNg= 2024-03-04T17:46:46.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5HQ/lsfA8V5v+4IXw+gm8iDuPNg=', 'Content-Length': '0'} 2024-03-04T17:46:46.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002473d755d0ada7e6-0065e60906-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.738 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.738 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.739 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.739 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.739 DEBUG:boto:Data: 2024-03-04T17:46:46.739 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.739 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:46.739 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.739 DEBUG:boto:Params: {} 2024-03-04T17:46:46.739 DEBUG:boto:Token: None 2024-03-04T17:46:46.739 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.739 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:q+l7+AsWZ77TB/kr1+HNPa5AWsg= 2024-03-04T17:46:46.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:q+l7+AsWZ77TB/kr1+HNPa5AWsg='} 2024-03-04T17:46:46.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-amz-request-id', 'tx000006bf1cf1d70e3adfe-0065e60906-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.764 DEBUG:rgw_multi.tests:version1 id=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:46:46.764 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.764 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.765 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.765 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.765 DEBUG:boto:Data: 2024-03-04T17:46:46.765 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.765 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:46.765 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.765 DEBUG:boto:Params: {} 2024-03-04T17:46:46.765 DEBUG:boto:Token: None 2024-03-04T17:46:46.765 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.766 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:n2+pS+x1oruSlCvg/lXnnLDYgvM= 2024-03-04T17:46:46.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:n2+pS+x1oruSlCvg/lXnnLDYgvM='} 2024-03-04T17:46:46.787 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-z59fuyvKSyjacV5Q4nMQHkw5BJ7LkT'), ('x-amz-request-id', 'tx00000f0fe0040a390dd79-0065e60906-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.787 DEBUG:rgw_multi.tests:version2 id=-z59fuyvKSyjacV5Q4nMQHkw5BJ7LkT 2024-03-04T17:46:46.787 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.788 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.788 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=-z59fuyvKSyjacV5Q4nMQHkw5BJ7LkT 2024-03-04T17:46:46.788 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=-z59fuyvKSyjacV5Q4nMQHkw5BJ7LkT 2024-03-04T17:46:46.788 DEBUG:boto:Method: DELETE 2024-03-04T17:46:46.788 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=-z59fuyvKSyjacV5Q4nMQHkw5BJ7LkT 2024-03-04T17:46:46.788 DEBUG:boto:Data: 2024-03-04T17:46:46.788 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.788 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:46.788 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.788 DEBUG:boto:Params: {} 2024-03-04T17:46:46.789 DEBUG:boto:Token: None 2024-03-04T17:46:46.789 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-21/obj-test-zone1?versionId=-z59fuyvKSyjacV5Q4nMQHkw5BJ7LkT 2024-03-04T17:46:46.789 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:r4gDgFdO7yJYjbSQUXaWpbmLeqU= 2024-03-04T17:46:46.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:r4gDgFdO7yJYjbSQUXaWpbmLeqU=', 'Content-Length': '0'} 2024-03-04T17:46:46.811 DEBUG:boto:Response headers: [('x-amz-version-id', '-z59fuyvKSyjacV5Q4nMQHkw5BJ7LkT'), ('x-amz-request-id', 'tx000007d4919749f2a0393-0065e60906-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.812 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.812 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.812 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.812 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.812 DEBUG:boto:Data: 2024-03-04T17:46:46.812 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.812 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:46.812 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.812 DEBUG:boto:Params: {} 2024-03-04T17:46:46.813 DEBUG:boto:Token: None 2024-03-04T17:46:46.813 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.813 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Fk0qj6mZPlzG0wCQ0JwScANsnZE= 2024-03-04T17:46:46.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Fk0qj6mZPlzG0wCQ0JwScANsnZE='} 2024-03-04T17:46:46.856 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BFZJPv2iuxlGdbvtOsQvbC0zk4gSwac'), ('x-amz-request-id', 'tx000004826a54b21c27948-0065e60906-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.856 DEBUG:rgw_multi.tests:version3 id=BFZJPv2iuxlGdbvtOsQvbC0zk4gSwac 2024-03-04T17:46:46.857 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.857 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:46:46.857 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=BFZJPv2iuxlGdbvtOsQvbC0zk4gSwac 2024-03-04T17:46:46.857 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=BFZJPv2iuxlGdbvtOsQvbC0zk4gSwac 2024-03-04T17:46:46.857 DEBUG:boto:Method: DELETE 2024-03-04T17:46:46.857 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=BFZJPv2iuxlGdbvtOsQvbC0zk4gSwac 2024-03-04T17:46:46.857 DEBUG:boto:Data: 2024-03-04T17:46:46.857 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.857 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:46:46.857 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.857 DEBUG:boto:Params: {} 2024-03-04T17:46:46.858 DEBUG:boto:Token: None 2024-03-04T17:46:46.858 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-21/obj-test-zone1?versionId=BFZJPv2iuxlGdbvtOsQvbC0zk4gSwac 2024-03-04T17:46:46.858 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVwyY4fij/LtuvzQUClr4GiNiNQ= 2024-03-04T17:46:46.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVwyY4fij/LtuvzQUClr4GiNiNQ=', 'Content-Length': '0'} 2024-03-04T17:46:46.880 DEBUG:boto:Response headers: [('x-amz-version-id', 'BFZJPv2iuxlGdbvtOsQvbC0zk4gSwac'), ('x-amz-request-id', 'tx0000096002ea373ed8b2a-0065e60906-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.880 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:46:46.880 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:46:46.881 DEBUG:boto:Method: HEAD 2024-03-04T17:46:46.881 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:46:46.881 DEBUG:boto:Data: 2024-03-04T17:46:46.881 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.881 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.881 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.881 DEBUG:boto:Params: {} 2024-03-04T17:46:46.881 DEBUG:boto:Token: None 2024-03-04T17:46:46.881 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-21/ 2024-03-04T17:46:46.882 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5HQ/lsfA8V5v+4IXw+gm8iDuPNg= 2024-03-04T17:46:46.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5HQ/lsfA8V5v+4IXw+gm8iDuPNg=', 'Content-Length': '0'} 2024-03-04T17:46:46.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099c4b20a99f98173-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.886 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.887 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.887 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.887 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.887 DEBUG:boto:Data: 2024-03-04T17:46:46.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.887 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.887 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.887 DEBUG:boto:Params: {} 2024-03-04T17:46:46.887 DEBUG:boto:Token: None 2024-03-04T17:46:46.888 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.888 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G82n0hDNLbz6Agq2JcgzdFJE9dM= 2024-03-04T17:46:46.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G82n0hDNLbz6Agq2JcgzdFJE9dM='} 2024-03-04T17:46:46.917 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-amz-request-id', 'tx000003fbfbb228a2fc5ce-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.917 DEBUG:rgw_multi.tests:version1 id=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:46:46.917 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.917 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.917 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.918 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.918 DEBUG:boto:Data: 2024-03-04T17:46:46.918 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.918 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.918 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.918 DEBUG:boto:Params: {} 2024-03-04T17:46:46.918 DEBUG:boto:Token: None 2024-03-04T17:46:46.918 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.918 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RQvYN3x+6pPl09nkGd2Z54w+/lA= 2024-03-04T17:46:46.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RQvYN3x+6pPl09nkGd2Z54w+/lA='} 2024-03-04T17:46:46.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MOF5rcWgM9wJXi1tFJmBqIdsg-VWw2F'), ('x-amz-request-id', 'tx0000052225993d1c5fba0-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.960 DEBUG:rgw_multi.tests:version2 id=MOF5rcWgM9wJXi1tFJmBqIdsg-VWw2F 2024-03-04T17:46:46.961 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.961 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.961 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=MOF5rcWgM9wJXi1tFJmBqIdsg-VWw2F 2024-03-04T17:46:46.961 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=MOF5rcWgM9wJXi1tFJmBqIdsg-VWw2F 2024-03-04T17:46:46.961 DEBUG:boto:Method: DELETE 2024-03-04T17:46:46.961 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=MOF5rcWgM9wJXi1tFJmBqIdsg-VWw2F 2024-03-04T17:46:46.961 DEBUG:boto:Data: 2024-03-04T17:46:46.961 DEBUG:boto:Headers: {} 2024-03-04T17:46:46.961 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.962 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.962 DEBUG:boto:Params: {} 2024-03-04T17:46:46.962 DEBUG:boto:Token: None 2024-03-04T17:46:46.962 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-21/obj-test-zone2?versionId=MOF5rcWgM9wJXi1tFJmBqIdsg-VWw2F 2024-03-04T17:46:46.962 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jxmYpgX1Cz7wXgjOZp4/PuqJfDk= 2024-03-04T17:46:46.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jxmYpgX1Cz7wXgjOZp4/PuqJfDk=', 'Content-Length': '0'} 2024-03-04T17:46:46.987 DEBUG:boto:Response headers: [('x-amz-version-id', 'MOF5rcWgM9wJXi1tFJmBqIdsg-VWw2F'), ('x-amz-request-id', 'tx00000973d3f4c0f1a96a4-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:46.988 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.988 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.988 DEBUG:boto:Method: PUT 2024-03-04T17:46:46.988 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.988 DEBUG:boto:Data: 2024-03-04T17:46:46.988 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:46.988 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:46.988 DEBUG:boto:Port: 8000 2024-03-04T17:46:46.988 DEBUG:boto:Params: {} 2024-03-04T17:46:46.989 DEBUG:boto:Token: None 2024-03-04T17:46:46.989 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 17:46:46 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:46:46.989 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9WFiGRQ8zTOwdfbmjU4wiLpoPvE= 2024-03-04T17:46:46.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9WFiGRQ8zTOwdfbmjU4wiLpoPvE='} 2024-03-04T17:46:47.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2FSp.blhokCXUro3mBfwf0366JRR57G'), ('x-amz-request-id', 'tx00000f6625e5c8ebc6b19-0065e60906-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:47.012 DEBUG:rgw_multi.tests:version3 id=2FSp.blhokCXUro3mBfwf0366JRR57G 2024-03-04T17:46:47.012 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:46:47.012 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:46:47.012 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=2FSp.blhokCXUro3mBfwf0366JRR57G 2024-03-04T17:46:47.012 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=2FSp.blhokCXUro3mBfwf0366JRR57G 2024-03-04T17:46:47.012 DEBUG:boto:Method: DELETE 2024-03-04T17:46:47.012 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=2FSp.blhokCXUro3mBfwf0366JRR57G 2024-03-04T17:46:47.013 DEBUG:boto:Data: 2024-03-04T17:46:47.013 DEBUG:boto:Headers: {} 2024-03-04T17:46:47.013 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:46:47.013 DEBUG:boto:Port: 8000 2024-03-04T17:46:47.013 DEBUG:boto:Params: {} 2024-03-04T17:46:47.013 DEBUG:boto:Token: None 2024-03-04T17:46:47.013 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:47 GMT /ppfahn-21/obj-test-zone2?versionId=2FSp.blhokCXUro3mBfwf0366JRR57G 2024-03-04T17:46:47.013 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kKvUGWPsrgrTDkk+8hp134qLMZw= 2024-03-04T17:46:47.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kKvUGWPsrgrTDkk+8hp134qLMZw=', 'Content-Length': '0'} 2024-03-04T17:46:47.036 DEBUG:boto:Response headers: [('x-amz-version-id', '2FSp.blhokCXUro3mBfwf0366JRR57G'), ('x-amz-request-id', 'tx0000018d1907de9849987-0065e60907-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:47.036 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:46:47.036 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:46:47.037 DEBUG:boto:Method: HEAD 2024-03-04T17:46:47.037 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:46:47.037 DEBUG:boto:Data: 2024-03-04T17:46:47.037 DEBUG:boto:Headers: {} 2024-03-04T17:46:47.037 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:47.037 DEBUG:boto:Port: 8001 2024-03-04T17:46:47.037 DEBUG:boto:Params: {} 2024-03-04T17:46:47.037 DEBUG:boto:Token: None 2024-03-04T17:46:47.037 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:46:47 GMT /ppfahn-21/ 2024-03-04T17:46:47.037 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4CpRDnUYr/KpUO1uIbM4fn8RMls= 2024-03-04T17:46:47.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4CpRDnUYr/KpUO1uIbM4fn8RMls=', 'Content-Length': '0'} 2024-03-04T17:46:47.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc9e46e9040c8903-0065e60907-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:47.042 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.042 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.042 DEBUG:boto:Method: PUT 2024-03-04T17:46:47.042 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.043 DEBUG:boto:Data: 2024-03-04T17:46:47.043 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:47.043 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:47.043 DEBUG:boto:Port: 8001 2024-03-04T17:46:47.043 DEBUG:boto:Params: {} 2024-03-04T17:46:47.043 DEBUG:boto:Token: None 2024-03-04T17:46:47.043 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:46:47 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.043 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:941KR5TNBdazQrWDOQiKxn8hDEM= 2024-03-04T17:46:47.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:941KR5TNBdazQrWDOQiKxn8hDEM='} 2024-03-04T17:46:47.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-amz-request-id', 'tx000007bb89654b84abab8-0065e60907-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:47.081 DEBUG:rgw_multi.tests:version1 id=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:46:47.082 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.082 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.082 DEBUG:boto:Method: PUT 2024-03-04T17:46:47.082 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.082 DEBUG:boto:Data: 2024-03-04T17:46:47.082 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:47.082 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:47.082 DEBUG:boto:Port: 8001 2024-03-04T17:46:47.082 DEBUG:boto:Params: {} 2024-03-04T17:46:47.083 DEBUG:boto:Token: None 2024-03-04T17:46:47.083 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 17:46:47 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.083 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0i6IPZY3VqFORxpFh/5OYQaUwjo= 2024-03-04T17:46:47.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0i6IPZY3VqFORxpFh/5OYQaUwjo='} 2024-03-04T17:46:47.132 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '740AgvX9qjRmMTheqaQOFuaJISBAVrA'), ('x-amz-request-id', 'tx000007c8d83d16b5e170f-0065e60907-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:47.133 DEBUG:rgw_multi.tests:version2 id=740AgvX9qjRmMTheqaQOFuaJISBAVrA 2024-03-04T17:46:47.133 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.133 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.133 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=740AgvX9qjRmMTheqaQOFuaJISBAVrA 2024-03-04T17:46:47.133 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=740AgvX9qjRmMTheqaQOFuaJISBAVrA 2024-03-04T17:46:47.133 DEBUG:boto:Method: DELETE 2024-03-04T17:46:47.133 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=740AgvX9qjRmMTheqaQOFuaJISBAVrA 2024-03-04T17:46:47.133 DEBUG:boto:Data: 2024-03-04T17:46:47.133 DEBUG:boto:Headers: {} 2024-03-04T17:46:47.134 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:47.134 DEBUG:boto:Port: 8001 2024-03-04T17:46:47.134 DEBUG:boto:Params: {} 2024-03-04T17:46:47.134 DEBUG:boto:Token: None 2024-03-04T17:46:47.134 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:47 GMT /ppfahn-21/obj-test-zone3?versionId=740AgvX9qjRmMTheqaQOFuaJISBAVrA 2024-03-04T17:46:47.134 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:h0HYj1RsAjs1MK6fZrG8NpAwaPU= 2024-03-04T17:46:47.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:h0HYj1RsAjs1MK6fZrG8NpAwaPU=', 'Content-Length': '0'} 2024-03-04T17:46:47.150 DEBUG:boto:Response headers: [('x-amz-version-id', '740AgvX9qjRmMTheqaQOFuaJISBAVrA'), ('x-amz-request-id', 'tx0000024904bd6688516a8-0065e60907-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:47.150 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.150 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.151 DEBUG:boto:Method: PUT 2024-03-04T17:46:47.151 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.151 DEBUG:boto:Data: 2024-03-04T17:46:47.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:46:47.151 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:47.151 DEBUG:boto:Port: 8001 2024-03-04T17:46:47.151 DEBUG:boto:Params: {} 2024-03-04T17:46:47.151 DEBUG:boto:Token: None 2024-03-04T17:46:47.152 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 17:46:47 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.152 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JZW2o7DhfaiDqBwpCVEkuHmQhEQ= 2024-03-04T17:46:47.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:46:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JZW2o7DhfaiDqBwpCVEkuHmQhEQ='} 2024-03-04T17:46:47.177 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NSX9CzMWkufvTpYZ-za509v1Dmb1ktR'), ('x-amz-request-id', 'tx000005285ef47642a1c69-0065e60907-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:47.177 DEBUG:rgw_multi.tests:version3 id=NSX9CzMWkufvTpYZ-za509v1Dmb1ktR 2024-03-04T17:46:47.177 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.177 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:46:47.177 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=NSX9CzMWkufvTpYZ-za509v1Dmb1ktR 2024-03-04T17:46:47.177 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=NSX9CzMWkufvTpYZ-za509v1Dmb1ktR 2024-03-04T17:46:47.178 DEBUG:boto:Method: DELETE 2024-03-04T17:46:47.178 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=NSX9CzMWkufvTpYZ-za509v1Dmb1ktR 2024-03-04T17:46:47.178 DEBUG:boto:Data: 2024-03-04T17:46:47.178 DEBUG:boto:Headers: {} 2024-03-04T17:46:47.178 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:46:47.178 DEBUG:boto:Port: 8001 2024-03-04T17:46:47.178 DEBUG:boto:Params: {} 2024-03-04T17:46:47.178 DEBUG:boto:Token: None 2024-03-04T17:46:47.179 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:46:47 GMT /ppfahn-21/obj-test-zone3?versionId=NSX9CzMWkufvTpYZ-za509v1Dmb1ktR 2024-03-04T17:46:47.179 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:C8BTgJi5JsCvQr1VnrkqM2MIgps= 2024-03-04T17:46:47.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:46:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:C8BTgJi5JsCvQr1VnrkqM2MIgps=', 'Content-Length': '0'} 2024-03-04T17:46:47.188 DEBUG:boto:Response headers: [('x-amz-version-id', 'NSX9CzMWkufvTpYZ-za509v1Dmb1ktR'), ('x-amz-request-id', 'tx00000e5ce6b718ea9e12a-0065e60907-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:46:47.188 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-19 2024-03-04T17:46:47.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:47.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:47.189 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:46:47.242 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:47.243 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:47.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:46:47.330+0000 7f19cb7eb980 1 bucket sync caught up with source: 2024-03-04T17:46:47.332 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000010.30.6, , , , 00000000004.19.10, , , , , ] 2024-03-04T17:46:47.332 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000010.30.6, , , , 00000000004.19.10, , , , , ] 2024-03-04T17:46:47.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:46:47.330+0000 7f19cb7eb980 0 bucket checkpoint complete 2024-03-04T17:46:47.352 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-19 2024-03-04T17:46:47.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:47.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:47.353 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:46:47.395 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:47.396 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:47.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:47.459+0000 7f609147c980 1 waiting for incremental sync to catch up: 2024-03-04T17:46:47.454 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.19.10, , , , , ] 2024-03-04T17:46:47.455 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000010.30.6, , , , 00000000004.19.10, , , , , ] 2024-03-04T17:46:52.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:52.459+0000 7f609147c980 1 bucket sync caught up with source: 2024-03-04T17:46:52.457 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000010.30.6, , , , 00000000004.19.10, , , , , ] 2024-03-04T17:46:52.457 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000010.30.6, , , , 00000000004.19.10, , , , , ] 2024-03-04T17:46:52.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:52.459+0000 7f609147c980 0 bucket checkpoint complete 2024-03-04T17:46:52.497 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-19 2024-03-04T17:46:52.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:52.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:46:52.498 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:46:52.590 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:52.590 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:52.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:52.659+0000 7f5704b64980 1 waiting for incremental sync to catch up: 2024-03-04T17:46:52.653 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.45.10, , , , , ] 2024-03-04T17:46:52.653 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.15.11, , 00000000010.54.6, , 00000000004.45.10, , , , , ] 2024-03-04T17:46:57.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:46:57.659+0000 7f5704b64980 1 waiting for incremental sync to catch up: 2024-03-04T17:46:57.656 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.45.10, , , , , ] 2024-03-04T17:46:57.656 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.15.11, , 00000000010.54.6, , 00000000004.45.10, , , , , ] 2024-03-04T17:47:02.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:02.663+0000 7f5704b64980 1 waiting for incremental sync to catch up: 2024-03-04T17:47:02.658 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , 00000000010.54.6, , 00000000004.45.10, , , , , ] 2024-03-04T17:47:02.658 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.15.11, , 00000000010.54.6, , 00000000004.45.10, , , , , ] 2024-03-04T17:47:07.660 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:07.663+0000 7f5704b64980 1 bucket sync caught up with source: 2024-03-04T17:47:07.660 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:47:07.660 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000002.15.11, , 00000000010.54.6, , 00000000004.45.10, , , , , ] 2024-03-04T17:47:07.660 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:07.663+0000 7f5704b64980 0 bucket checkpoint complete 2024-03-04T17:47:07.700 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-19 2024-03-04T17:47:07.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:47:07.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:47:07.700 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:47:07.749 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:07.750 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:07.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:07.815+0000 7f2110768980 1 waiting for incremental sync to catch up: 2024-03-04T17:47:07.811 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000002.15.11, , 00000000004.42.12, , 00000000004.45.10, , , , , ] 2024-03-04T17:47:07.811 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:47:12.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:12.819+0000 7f2110768980 1 waiting for incremental sync to catch up: 2024-03-04T17:47:12.813 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000008.50.12, , 00000000007.51.10, , , , , ] 2024-03-04T17:47:12.814 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:47:17.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:17.819+0000 7f2110768980 1 waiting for incremental sync to catch up: 2024-03-04T17:47:17.816 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000008.50.12, , 00000000007.51.10, , , , , ] 2024-03-04T17:47:17.816 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:47:22.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:22.823+0000 7f2110768980 1 waiting for incremental sync to catch up: 2024-03-04T17:47:22.818 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000008.50.12, , 00000000007.51.10, , , , , ] 2024-03-04T17:47:22.818 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:47:25.677 INFO:tasks.rgw.c2.client.0.smithi153.stdout:2024-03-04T17:47:25.677+0000 7fa53962a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:47:27.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:27.826+0000 7f2110768980 1 waiting for incremental sync to catch up: 2024-03-04T17:47:27.821 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000008.50.12, , 00000000007.51.10, , , , , ] 2024-03-04T17:47:27.821 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:47:32.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:32.826+0000 7f2110768980 1 waiting for incremental sync to catch up: 2024-03-04T17:47:32.823 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000008.50.12, , 00000000007.51.10, , , , , ] 2024-03-04T17:47:32.823 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:47:37.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:37.830+0000 7f2110768980 1 waiting for incremental sync to catch up: 2024-03-04T17:47:37.826 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000008.50.12, , 00000000007.51.10, , , , , ] 2024-03-04T17:47:37.826 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:47:42.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:42.830+0000 7f2110768980 1 waiting for incremental sync to catch up: 2024-03-04T17:47:42.828 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000008.50.12, , 00000000007.51.10, , , , , ] 2024-03-04T17:47:42.828 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:47:47.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:47.834+0000 7f2110768980 1 waiting for incremental sync to catch up: 2024-03-04T17:47:47.830 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000008.50.12, , 00000000007.51.10, , , , , ] 2024-03-04T17:47:47.831 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:47:52.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:52.834+0000 7f2110768980 1 waiting for incremental sync to catch up: 2024-03-04T17:47:52.835 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000008.50.12, , 00000000007.51.10, , , , , ] 2024-03-04T17:47:52.835 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:47:57.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:47:57.837+0000 7f2110768980 1 waiting for incremental sync to catch up: 2024-03-04T17:47:57.835 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000008.50.12, , 00000000007.51.10, , , , , ] 2024-03-04T17:47:57.835 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:48:02.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:02.841+0000 7f2110768980 1 waiting for incremental sync to catch up: 2024-03-04T17:48:02.837 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000008.50.12, , 00000000007.51.10, , , , , ] 2024-03-04T17:48:02.837 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:48:07.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:07.841+0000 7f2110768980 1 waiting for incremental sync to catch up: 2024-03-04T17:48:07.840 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000008.50.12, , 00000000007.51.10, , , , , ] 2024-03-04T17:48:07.840 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:48:12.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:12.845+0000 7f2110768980 1 bucket sync caught up with source: 2024-03-04T17:48:12.842 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:48:12.842 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000010.54.6, , 00000000006.49.11, , , , , ] 2024-03-04T17:48:12.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:12.845+0000 7f2110768980 0 bucket checkpoint complete 2024-03-04T17:48:12.878 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-19 2024-03-04T17:48:12.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:12.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:12.879 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:12.926 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:12.926 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:12.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:12.985+0000 7faedb2aa980 1 bucket sync caught up with source: 2024-03-04T17:48:12.984 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.45.10, , 00000000003.96.10, , 00000000014.180.6, , , , , ] 2024-03-04T17:48:12.984 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.45.10, , 00000000003.96.10, , 00000000014.180.6, , , , , ] 2024-03-04T17:48:12.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:12.985+0000 7faedb2aa980 0 bucket checkpoint complete 2024-03-04T17:48:13.007 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-19 2024-03-04T17:48:13.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:13.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:13.007 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:13.058 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:13.058 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:13.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:13.125+0000 7f4159d1c980 1 bucket sync caught up with source: 2024-03-04T17:48:13.123 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000003.45.10, , 00000000003.96.10, , 00000000014.180.6, , , , , ] 2024-03-04T17:48:13.123 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000003.45.10, , 00000000003.96.10, , 00000000014.180.6, , , , , ] 2024-03-04T17:48:13.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:13.125+0000 7f4159d1c980 0 bucket checkpoint complete 2024-03-04T17:48:13.145 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-19 zones={test-zone2, test-zone1} 2024-03-04T17:48:13.145 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.145 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.145 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.145 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:48:13.146 DEBUG:boto:Data: 2024-03-04T17:48:13.146 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.146 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.146 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.146 DEBUG:boto:Params: {} 2024-03-04T17:48:13.146 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:48:13.146 DEBUG:boto:Token: None 2024-03-04T17:48:13.147 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.147 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:289Y4Fvhq9fSBVd90tScMF8iK2A= 2024-03-04T17:48:13.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:289Y4Fvhq9fSBVd90tScMF8iK2A=', 'Content-Length': '0'} 2024-03-04T17:48:13.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fc832e5c1eb1ccb-0065e6095d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.151 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.152 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.152 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.152 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:48:13.152 DEBUG:boto:Data: 2024-03-04T17:48:13.152 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.152 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.152 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.152 DEBUG:boto:Params: {} 2024-03-04T17:48:13.152 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:48:13.153 DEBUG:boto:Token: None 2024-03-04T17:48:13.153 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.153 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:289Y4Fvhq9fSBVd90tScMF8iK2A= 2024-03-04T17:48:13.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:289Y4Fvhq9fSBVd90tScMF8iK2A=', 'Content-Length': '0'} 2024-03-04T17:48:13.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000efee858c6985b3-0065e6095d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.159 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:48:13.160 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.160 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.160 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:13.160 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:13.160 DEBUG:boto:Method: GET 2024-03-04T17:48:13.160 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:13.160 DEBUG:boto:Data: 2024-03-04T17:48:13.160 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.160 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.161 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.161 DEBUG:boto:Params: {} 2024-03-04T17:48:13.161 DEBUG:boto:Token: None 2024-03-04T17:48:13.161 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/?versions 2024-03-04T17:48:13.161 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Jg++o+zJndsVD1cAhE+su2jVwjI= 2024-03-04T17:48:13.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Jg++o+zJndsVD1cAhE+su2jVwjI=', 'Content-Length': '0'} 2024-03-04T17:48:13.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef5c0079c4da58ec-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.163 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.164 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:13.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:13.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:13.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:13.165 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:48:13.165 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.165 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.165 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:13.165 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:13.165 DEBUG:boto:Method: GET 2024-03-04T17:48:13.165 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:13.166 DEBUG:boto:Data: 2024-03-04T17:48:13.166 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.166 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.166 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.166 DEBUG:boto:Params: {} 2024-03-04T17:48:13.166 DEBUG:boto:Token: None 2024-03-04T17:48:13.166 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/?versions 2024-03-04T17:48:13.166 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Jg++o+zJndsVD1cAhE+su2jVwjI= 2024-03-04T17:48:13.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Jg++o+zJndsVD1cAhE+su2jVwjI=', 'Content-Length': '0'} 2024-03-04T17:48:13.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000724562ec89af5b51-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.171 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.172 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:13.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:13.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:13.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:13.172 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.172 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.172 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:13.172 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:13.173 DEBUG:boto:Method: GET 2024-03-04T17:48:13.173 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:13.173 DEBUG:boto:Data: 2024-03-04T17:48:13.173 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.173 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.173 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.173 DEBUG:boto:Params: {} 2024-03-04T17:48:13.173 DEBUG:boto:Token: None 2024-03-04T17:48:13.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/?versions 2024-03-04T17:48:13.173 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Jg++o+zJndsVD1cAhE+su2jVwjI= 2024-03-04T17:48:13.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Jg++o+zJndsVD1cAhE+su2jVwjI=', 'Content-Length': '0'} 2024-03-04T17:48:13.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de2838054c4e7674-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.176 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.177 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.177 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.177 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:13.177 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:13.177 DEBUG:boto:Method: GET 2024-03-04T17:48:13.177 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:13.177 DEBUG:boto:Data: 2024-03-04T17:48:13.177 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.177 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.177 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.178 DEBUG:boto:Params: {} 2024-03-04T17:48:13.178 DEBUG:boto:Token: None 2024-03-04T17:48:13.178 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/?versions 2024-03-04T17:48:13.178 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Jg++o+zJndsVD1cAhE+su2jVwjI= 2024-03-04T17:48:13.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Jg++o+zJndsVD1cAhE+su2jVwjI=', 'Content-Length': '0'} 2024-03-04T17:48:13.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a8cff26b353c7b1-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.183 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:13.184 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:13.184 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:13.184 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.184 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.184 DEBUG:boto:Method: GET 2024-03-04T17:48:13.184 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.184 DEBUG:boto:Data: 2024-03-04T17:48:13.185 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.185 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.185 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.185 DEBUG:boto:Params: {} 2024-03-04T17:48:13.185 DEBUG:boto:Token: None 2024-03-04T17:48:13.185 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.185 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+ijiAZuSQEOcmbvybJb5iO7FNW4= 2024-03-04T17:48:13.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+ijiAZuSQEOcmbvybJb5iO7FNW4=', 'Content-Length': '0'} 2024-03-04T17:48:13.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f89454b8423eb7f2-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.188 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:13.188 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:13.188 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.188 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.188 DEBUG:boto:Method: GET 2024-03-04T17:48:13.188 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.189 DEBUG:boto:Data: 2024-03-04T17:48:13.189 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.189 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.189 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.189 DEBUG:boto:Params: {} 2024-03-04T17:48:13.189 DEBUG:boto:Token: None 2024-03-04T17:48:13.189 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.189 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+ijiAZuSQEOcmbvybJb5iO7FNW4= 2024-03-04T17:48:13.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+ijiAZuSQEOcmbvybJb5iO7FNW4=', 'Content-Length': '0'} 2024-03-04T17:48:13.192 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de770a5638b6b52f-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.193 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:13.193 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:13.193 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.193 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.193 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.193 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.194 DEBUG:boto:Data: 2024-03-04T17:48:13.194 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.194 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.194 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.194 DEBUG:boto:Params: {} 2024-03-04T17:48:13.194 DEBUG:boto:Token: None 2024-03-04T17:48:13.194 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.194 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U9j/XuInjRpG+2hfynj3tIS88H8= 2024-03-04T17:48:13.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U9j/XuInjRpG+2hfynj3tIS88H8=', 'Content-Length': '0'} 2024-03-04T17:48:13.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000169a9077a09b41e0-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.196 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:13.197 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:13.197 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.197 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.197 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.197 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.197 DEBUG:boto:Data: 2024-03-04T17:48:13.197 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.197 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.197 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.197 DEBUG:boto:Params: {} 2024-03-04T17:48:13.198 DEBUG:boto:Token: None 2024-03-04T17:48:13.198 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.198 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U9j/XuInjRpG+2hfynj3tIS88H8= 2024-03-04T17:48:13.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U9j/XuInjRpG+2hfynj3tIS88H8=', 'Content-Length': '0'} 2024-03-04T17:48:13.201 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e30fa90aa60af793-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:13.201 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:13.201 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:13.201 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.202 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.202 DEBUG:boto:Method: GET 2024-03-04T17:48:13.202 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.202 DEBUG:boto:Data: 2024-03-04T17:48:13.202 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.202 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.202 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.202 DEBUG:boto:Params: {} 2024-03-04T17:48:13.202 DEBUG:boto:Token: None 2024-03-04T17:48:13.202 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.203 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+ijiAZuSQEOcmbvybJb5iO7FNW4= 2024-03-04T17:48:13.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+ijiAZuSQEOcmbvybJb5iO7FNW4=', 'Content-Length': '0'} 2024-03-04T17:48:13.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eefe5e703fde63a7-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.205 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:13.205 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:13.207 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.207 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.207 DEBUG:boto:Method: GET 2024-03-04T17:48:13.208 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.208 DEBUG:boto:Data: 2024-03-04T17:48:13.208 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.208 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.208 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.208 DEBUG:boto:Params: {} 2024-03-04T17:48:13.208 DEBUG:boto:Token: None 2024-03-04T17:48:13.208 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.208 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+ijiAZuSQEOcmbvybJb5iO7FNW4= 2024-03-04T17:48:13.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+ijiAZuSQEOcmbvybJb5iO7FNW4=', 'Content-Length': '0'} 2024-03-04T17:48:13.211 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dce290cc4999f3ac-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.211 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.212 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.212 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:13.212 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:13.212 DEBUG:boto:Method: GET 2024-03-04T17:48:13.212 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:48:13.212 DEBUG:boto:Data: 2024-03-04T17:48:13.212 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.212 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.212 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.212 DEBUG:boto:Params: {} 2024-03-04T17:48:13.213 DEBUG:boto:Token: None 2024-03-04T17:48:13.213 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.213 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA= 2024-03-04T17:48:13.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA=', 'Content-Length': '0'} 2024-03-04T17:48:13.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8b64ed86f0f02dc-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.264 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.264 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.265 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.265 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:13.265 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:13.265 DEBUG:boto:Method: GET 2024-03-04T17:48:13.265 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:48:13.265 DEBUG:boto:Data: 2024-03-04T17:48:13.265 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.265 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.265 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.265 DEBUG:boto:Params: {} 2024-03-04T17:48:13.266 DEBUG:boto:Token: None 2024-03-04T17:48:13.266 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.266 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA= 2024-03-04T17:48:13.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA=', 'Content-Length': '0'} 2024-03-04T17:48:13.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6a1dac3fdbe40c4-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.271 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.271 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:13.271 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:13.271 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.271 DEBUG:boto:Path: /ppfahn-19/dummy 2024-03-04T17:48:13.272 DEBUG:boto:Data: 2024-03-04T17:48:13.272 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.272 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.272 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.272 DEBUG:boto:Params: {} 2024-03-04T17:48:13.272 DEBUG:boto:Token: None 2024-03-04T17:48:13.272 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/dummy 2024-03-04T17:48:13.272 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VMi4Itk5zGg9t9xvaudumpepptE= 2024-03-04T17:48:13.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VMi4Itk5zGg9t9xvaudumpepptE=', 'Content-Length': '0'} 2024-03-04T17:48:13.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007f77a3429122455c-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.276 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:13.277 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:13.277 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.277 DEBUG:boto:Path: /ppfahn-19/dummy 2024-03-04T17:48:13.277 DEBUG:boto:Data: 2024-03-04T17:48:13.277 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.277 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.277 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.277 DEBUG:boto:Params: {} 2024-03-04T17:48:13.277 DEBUG:boto:Token: None 2024-03-04T17:48:13.277 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/dummy 2024-03-04T17:48:13.278 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VMi4Itk5zGg9t9xvaudumpepptE= 2024-03-04T17:48:13.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VMi4Itk5zGg9t9xvaudumpepptE=', 'Content-Length': '0'} 2024-03-04T17:48:13.281 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b0beffe8d44511c-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:13.282 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:13.282 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:13.282 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.282 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.282 DEBUG:boto:Method: GET 2024-03-04T17:48:13.282 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.282 DEBUG:boto:Data: 2024-03-04T17:48:13.282 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.283 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.283 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.283 DEBUG:boto:Params: {} 2024-03-04T17:48:13.283 DEBUG:boto:Token: None 2024-03-04T17:48:13.283 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.283 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+ijiAZuSQEOcmbvybJb5iO7FNW4= 2024-03-04T17:48:13.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+ijiAZuSQEOcmbvybJb5iO7FNW4=', 'Content-Length': '0'} 2024-03-04T17:48:13.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a860b5d62903cbc2-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.286 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:13.286 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:13.287 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.287 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.287 DEBUG:boto:Method: GET 2024-03-04T17:48:13.287 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.287 DEBUG:boto:Data: 2024-03-04T17:48:13.287 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.287 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.287 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.287 DEBUG:boto:Params: {} 2024-03-04T17:48:13.287 DEBUG:boto:Token: None 2024-03-04T17:48:13.288 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:13.288 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+ijiAZuSQEOcmbvybJb5iO7FNW4= 2024-03-04T17:48:13.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+ijiAZuSQEOcmbvybJb5iO7FNW4=', 'Content-Length': '0'} 2024-03-04T17:48:13.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000001c4406fefb3f23-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.291 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.291 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.291 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:13.291 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:13.291 DEBUG:boto:Method: GET 2024-03-04T17:48:13.292 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:48:13.292 DEBUG:boto:Data: 2024-03-04T17:48:13.292 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.292 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.292 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.292 DEBUG:boto:Params: {} 2024-03-04T17:48:13.292 DEBUG:boto:Token: None 2024-03-04T17:48:13.292 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.292 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA= 2024-03-04T17:48:13.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA=', 'Content-Length': '0'} 2024-03-04T17:48:13.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0bf0f0d5bf20b7b-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.340 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.340 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.340 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.341 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:13.341 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:13.341 DEBUG:boto:Method: GET 2024-03-04T17:48:13.341 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:48:13.341 DEBUG:boto:Data: 2024-03-04T17:48:13.341 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.341 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.341 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.341 DEBUG:boto:Params: {} 2024-03-04T17:48:13.341 DEBUG:boto:Token: None 2024-03-04T17:48:13.342 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.342 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA= 2024-03-04T17:48:13.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA=', 'Content-Length': '0'} 2024-03-04T17:48:13.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098541f6e1468c796-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.347 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:13.347 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.347 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.348 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.348 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.348 DEBUG:boto:Method: GET 2024-03-04T17:48:13.348 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.348 DEBUG:boto:Data: 2024-03-04T17:48:13.348 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.348 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.348 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.348 DEBUG:boto:Params: {} 2024-03-04T17:48:13.348 DEBUG:boto:Token: None 2024-03-04T17:48:13.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.349 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HopLElH8fRn0eRF++vh+zmvR0eg= 2024-03-04T17:48:13.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HopLElH8fRn0eRF++vh+zmvR0eg=', 'Content-Length': '0'} 2024-03-04T17:48:13.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005db8bcefa016d41b-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.392 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.392 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.392 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.392 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.392 DEBUG:boto:Method: GET 2024-03-04T17:48:13.392 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.393 DEBUG:boto:Data: 2024-03-04T17:48:13.393 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.393 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.393 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.393 DEBUG:boto:Params: {} 2024-03-04T17:48:13.393 DEBUG:boto:Token: None 2024-03-04T17:48:13.393 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.393 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HopLElH8fRn0eRF++vh+zmvR0eg= 2024-03-04T17:48:13.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HopLElH8fRn0eRF++vh+zmvR0eg=', 'Content-Length': '0'} 2024-03-04T17:48:13.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfeb1343c4d22fb4-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.397 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.397 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.397 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.397 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.397 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.397 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.397 DEBUG:boto:Data: 2024-03-04T17:48:13.397 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.397 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.397 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.397 DEBUG:boto:Params: {} 2024-03-04T17:48:13.398 DEBUG:boto:Token: None 2024-03-04T17:48:13.398 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.398 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ztmBgKyPa3BKDSHXE0WXpr11174= 2024-03-04T17:48:13.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ztmBgKyPa3BKDSHXE0WXpr11174=', 'Content-Length': '0'} 2024-03-04T17:48:13.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c871dc5934ae9ad1-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.401 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.401 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.401 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.401 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.401 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.402 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.402 DEBUG:boto:Data: 2024-03-04T17:48:13.402 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.402 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.402 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.402 DEBUG:boto:Params: {} 2024-03-04T17:48:13.402 DEBUG:boto:Token: None 2024-03-04T17:48:13.402 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.402 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ztmBgKyPa3BKDSHXE0WXpr11174= 2024-03-04T17:48:13.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ztmBgKyPa3BKDSHXE0WXpr11174=', 'Content-Length': '0'} 2024-03-04T17:48:13.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009745145b93651ba7-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:13.406 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.406 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.406 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.406 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.406 DEBUG:boto:Method: GET 2024-03-04T17:48:13.406 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.407 DEBUG:boto:Data: 2024-03-04T17:48:13.407 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.407 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.407 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.407 DEBUG:boto:Params: {} 2024-03-04T17:48:13.407 DEBUG:boto:Token: None 2024-03-04T17:48:13.407 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.407 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HopLElH8fRn0eRF++vh+zmvR0eg= 2024-03-04T17:48:13.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HopLElH8fRn0eRF++vh+zmvR0eg=', 'Content-Length': '0'} 2024-03-04T17:48:13.410 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c42f817284fc5c52-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.456 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.456 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.456 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.457 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.457 DEBUG:boto:Method: GET 2024-03-04T17:48:13.457 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.457 DEBUG:boto:Data: 2024-03-04T17:48:13.457 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.457 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.457 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.457 DEBUG:boto:Params: {} 2024-03-04T17:48:13.457 DEBUG:boto:Token: None 2024-03-04T17:48:13.458 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.458 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HopLElH8fRn0eRF++vh+zmvR0eg= 2024-03-04T17:48:13.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HopLElH8fRn0eRF++vh+zmvR0eg=', 'Content-Length': '0'} 2024-03-04T17:48:13.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7f6897cd5a17f05-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.461 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.462 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.462 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:13.462 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:13.462 DEBUG:boto:Method: GET 2024-03-04T17:48:13.462 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:13.462 DEBUG:boto:Data: 2024-03-04T17:48:13.462 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.462 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.462 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.462 DEBUG:boto:Params: {} 2024-03-04T17:48:13.462 DEBUG:boto:Token: None 2024-03-04T17:48:13.463 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.463 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA= 2024-03-04T17:48:13.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA=', 'Content-Length': '0'} 2024-03-04T17:48:13.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021d980501588ad24-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.467 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.468 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.468 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.468 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:13.468 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:13.468 DEBUG:boto:Method: GET 2024-03-04T17:48:13.468 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:13.468 DEBUG:boto:Data: 2024-03-04T17:48:13.468 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.468 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.468 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.469 DEBUG:boto:Params: {} 2024-03-04T17:48:13.469 DEBUG:boto:Token: None 2024-03-04T17:48:13.469 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.469 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA= 2024-03-04T17:48:13.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA=', 'Content-Length': '0'} 2024-03-04T17:48:13.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006213180b9ed482aa-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.474 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.474 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.474 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.474 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.474 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.474 DEBUG:boto:Data: 2024-03-04T17:48:13.474 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.475 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.475 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.475 DEBUG:boto:Params: {} 2024-03-04T17:48:13.475 DEBUG:boto:Token: None 2024-03-04T17:48:13.475 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.475 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ote1scc+hDVAANnLPdkscSOXhRw= 2024-03-04T17:48:13.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ote1scc+hDVAANnLPdkscSOXhRw=', 'Content-Length': '0'} 2024-03-04T17:48:13.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003185c505447d7a2f-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.479 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.479 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.479 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.479 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.480 DEBUG:boto:Data: 2024-03-04T17:48:13.480 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.480 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.480 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.480 DEBUG:boto:Params: {} 2024-03-04T17:48:13.480 DEBUG:boto:Token: None 2024-03-04T17:48:13.480 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.480 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ote1scc+hDVAANnLPdkscSOXhRw= 2024-03-04T17:48:13.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ote1scc+hDVAANnLPdkscSOXhRw=', 'Content-Length': '0'} 2024-03-04T17:48:13.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099e9ce50a22b7fd5-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:13.485 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.485 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.485 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.485 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.485 DEBUG:boto:Method: GET 2024-03-04T17:48:13.485 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.485 DEBUG:boto:Data: 2024-03-04T17:48:13.485 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.485 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.485 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.486 DEBUG:boto:Params: {} 2024-03-04T17:48:13.486 DEBUG:boto:Token: None 2024-03-04T17:48:13.486 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.486 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HopLElH8fRn0eRF++vh+zmvR0eg= 2024-03-04T17:48:13.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HopLElH8fRn0eRF++vh+zmvR0eg=', 'Content-Length': '0'} 2024-03-04T17:48:13.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000704d87e022f5745-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.536 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.536 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:13.536 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.536 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.537 DEBUG:boto:Method: GET 2024-03-04T17:48:13.537 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.537 DEBUG:boto:Data: 2024-03-04T17:48:13.537 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.537 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.537 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.537 DEBUG:boto:Params: {} 2024-03-04T17:48:13.537 DEBUG:boto:Token: None 2024-03-04T17:48:13.537 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:13.538 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HopLElH8fRn0eRF++vh+zmvR0eg= 2024-03-04T17:48:13.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HopLElH8fRn0eRF++vh+zmvR0eg=', 'Content-Length': '0'} 2024-03-04T17:48:13.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecfe1518864167cf-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.541 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.541 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.541 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:13.541 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:13.541 DEBUG:boto:Method: GET 2024-03-04T17:48:13.542 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:13.542 DEBUG:boto:Data: 2024-03-04T17:48:13.542 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.542 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.542 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.542 DEBUG:boto:Params: {} 2024-03-04T17:48:13.542 DEBUG:boto:Token: None 2024-03-04T17:48:13.542 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.542 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA= 2024-03-04T17:48:13.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA=', 'Content-Length': '0'} 2024-03-04T17:48:13.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be5b526d4196ac99-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.547 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.547 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.548 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.548 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:13.548 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:13.548 DEBUG:boto:Method: GET 2024-03-04T17:48:13.548 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:13.548 DEBUG:boto:Data: 2024-03-04T17:48:13.548 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.548 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.548 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.548 DEBUG:boto:Params: {} 2024-03-04T17:48:13.549 DEBUG:boto:Token: None 2024-03-04T17:48:13.549 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.549 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA= 2024-03-04T17:48:13.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA=', 'Content-Length': '0'} 2024-03-04T17:48:13.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008d19c5d4f9b1493-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.596 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:13.597 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.597 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.597 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.597 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.597 DEBUG:boto:Method: GET 2024-03-04T17:48:13.597 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.597 DEBUG:boto:Data: 2024-03-04T17:48:13.597 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.598 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.598 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.598 DEBUG:boto:Params: {} 2024-03-04T17:48:13.598 DEBUG:boto:Token: None 2024-03-04T17:48:13.598 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.598 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FtTqLeCCy8MWMmou5owMS4SmVXE= 2024-03-04T17:48:13.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FtTqLeCCy8MWMmou5owMS4SmVXE=', 'Content-Length': '0'} 2024-03-04T17:48:13.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001589c65bba568de-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.602 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.602 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.602 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.602 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.602 DEBUG:boto:Method: GET 2024-03-04T17:48:13.602 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.602 DEBUG:boto:Data: 2024-03-04T17:48:13.602 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.602 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.603 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.603 DEBUG:boto:Params: {} 2024-03-04T17:48:13.603 DEBUG:boto:Token: None 2024-03-04T17:48:13.603 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.603 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FtTqLeCCy8MWMmou5owMS4SmVXE= 2024-03-04T17:48:13.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FtTqLeCCy8MWMmou5owMS4SmVXE=', 'Content-Length': '0'} 2024-03-04T17:48:13.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a309fa7b3c1dd8c-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.606 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.606 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.606 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.607 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.607 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.607 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.607 DEBUG:boto:Data: 2024-03-04T17:48:13.607 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.607 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.607 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.607 DEBUG:boto:Params: {} 2024-03-04T17:48:13.607 DEBUG:boto:Token: None 2024-03-04T17:48:13.607 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.608 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IXyPS1iHeWS9RwbIwfBEVsipQ9w= 2024-03-04T17:48:13.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IXyPS1iHeWS9RwbIwfBEVsipQ9w=', 'Content-Length': '0'} 2024-03-04T17:48:13.611 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056dccd5d18eb5393-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.611 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.611 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.611 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.611 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.611 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.611 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.611 DEBUG:boto:Data: 2024-03-04T17:48:13.612 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.612 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.612 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.612 DEBUG:boto:Params: {} 2024-03-04T17:48:13.612 DEBUG:boto:Token: None 2024-03-04T17:48:13.612 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.612 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IXyPS1iHeWS9RwbIwfBEVsipQ9w= 2024-03-04T17:48:13.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IXyPS1iHeWS9RwbIwfBEVsipQ9w=', 'Content-Length': '0'} 2024-03-04T17:48:13.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000004d91bac7fed15e-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:13.615 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.615 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.615 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.616 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.616 DEBUG:boto:Method: GET 2024-03-04T17:48:13.616 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.616 DEBUG:boto:Data: 2024-03-04T17:48:13.616 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.616 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.616 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.616 DEBUG:boto:Params: {} 2024-03-04T17:48:13.616 DEBUG:boto:Token: None 2024-03-04T17:48:13.616 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.617 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FtTqLeCCy8MWMmou5owMS4SmVXE= 2024-03-04T17:48:13.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FtTqLeCCy8MWMmou5owMS4SmVXE=', 'Content-Length': '0'} 2024-03-04T17:48:13.620 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000905e56c781af8bcb-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.620 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.620 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.620 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.620 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.620 DEBUG:boto:Method: GET 2024-03-04T17:48:13.620 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.620 DEBUG:boto:Data: 2024-03-04T17:48:13.621 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.621 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.621 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.621 DEBUG:boto:Params: {} 2024-03-04T17:48:13.621 DEBUG:boto:Token: None 2024-03-04T17:48:13.621 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.621 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FtTqLeCCy8MWMmou5owMS4SmVXE= 2024-03-04T17:48:13.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FtTqLeCCy8MWMmou5owMS4SmVXE=', 'Content-Length': '0'} 2024-03-04T17:48:13.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077f51d539addf7ec-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.668 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.668 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.668 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:13.668 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:13.669 DEBUG:boto:Method: GET 2024-03-04T17:48:13.669 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:13.669 DEBUG:boto:Data: 2024-03-04T17:48:13.669 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.669 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.669 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.669 DEBUG:boto:Params: {} 2024-03-04T17:48:13.669 DEBUG:boto:Token: None 2024-03-04T17:48:13.669 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.670 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA= 2024-03-04T17:48:13.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA=', 'Content-Length': '0'} 2024-03-04T17:48:13.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b25db4c6829a9906-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.675 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.675 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.675 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.676 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:13.676 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:13.676 DEBUG:boto:Method: GET 2024-03-04T17:48:13.676 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:13.676 DEBUG:boto:Data: 2024-03-04T17:48:13.676 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.676 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.676 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.676 DEBUG:boto:Params: {} 2024-03-04T17:48:13.676 DEBUG:boto:Token: None 2024-03-04T17:48:13.677 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.677 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA= 2024-03-04T17:48:13.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA=', 'Content-Length': '0'} 2024-03-04T17:48:13.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0706a2906d5739d-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.728 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.729 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.729 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.729 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.729 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.729 DEBUG:boto:Data: 2024-03-04T17:48:13.729 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.729 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.729 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.729 DEBUG:boto:Params: {} 2024-03-04T17:48:13.730 DEBUG:boto:Token: None 2024-03-04T17:48:13.730 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.730 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNF0iknAHep7qcUsQ4tq+ttt4MA= 2024-03-04T17:48:13.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNF0iknAHep7qcUsQ4tq+ttt4MA=', 'Content-Length': '0'} 2024-03-04T17:48:13.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec6d9a83edc3face-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.734 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.734 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.734 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.734 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.734 DEBUG:boto:Data: 2024-03-04T17:48:13.734 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.735 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.735 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.735 DEBUG:boto:Params: {} 2024-03-04T17:48:13.735 DEBUG:boto:Token: None 2024-03-04T17:48:13.735 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.735 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MNF0iknAHep7qcUsQ4tq+ttt4MA= 2024-03-04T17:48:13.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MNF0iknAHep7qcUsQ4tq+ttt4MA=', 'Content-Length': '0'} 2024-03-04T17:48:13.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a44e1ec77860284-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:13.740 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.740 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.740 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.740 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.740 DEBUG:boto:Method: GET 2024-03-04T17:48:13.740 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.740 DEBUG:boto:Data: 2024-03-04T17:48:13.740 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.740 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.741 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.741 DEBUG:boto:Params: {} 2024-03-04T17:48:13.741 DEBUG:boto:Token: None 2024-03-04T17:48:13.741 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.741 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FtTqLeCCy8MWMmou5owMS4SmVXE= 2024-03-04T17:48:13.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FtTqLeCCy8MWMmou5owMS4SmVXE=', 'Content-Length': '0'} 2024-03-04T17:48:13.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0c2dfb570d19f87-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.788 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.788 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:13.788 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.788 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.789 DEBUG:boto:Method: GET 2024-03-04T17:48:13.789 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.789 DEBUG:boto:Data: 2024-03-04T17:48:13.789 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.789 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.789 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.789 DEBUG:boto:Params: {} 2024-03-04T17:48:13.789 DEBUG:boto:Token: None 2024-03-04T17:48:13.789 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:13.790 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FtTqLeCCy8MWMmou5owMS4SmVXE= 2024-03-04T17:48:13.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FtTqLeCCy8MWMmou5owMS4SmVXE=', 'Content-Length': '0'} 2024-03-04T17:48:13.792 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a751e63cc6803bcd-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.836 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.836 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.836 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:13.836 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:13.837 DEBUG:boto:Method: GET 2024-03-04T17:48:13.837 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:13.837 DEBUG:boto:Data: 2024-03-04T17:48:13.837 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.837 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.837 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.837 DEBUG:boto:Params: {} 2024-03-04T17:48:13.837 DEBUG:boto:Token: None 2024-03-04T17:48:13.838 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.838 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA= 2024-03-04T17:48:13.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA=', 'Content-Length': '0'} 2024-03-04T17:48:13.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000606117c00b39c89a-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.843 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.844 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:13.844 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:13.844 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:13.844 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:13.844 DEBUG:boto:Method: GET 2024-03-04T17:48:13.844 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:13.845 DEBUG:boto:Data: 2024-03-04T17:48:13.845 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.845 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.845 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.845 DEBUG:boto:Params: {} 2024-03-04T17:48:13.846 DEBUG:boto:Token: None 2024-03-04T17:48:13.846 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/ 2024-03-04T17:48:13.846 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA= 2024-03-04T17:48:13.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2K/Lwrp1AybMPyk8rGg8ZMDXmnA=', 'Content-Length': '0'} 2024-03-04T17:48:13.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef2cb35102b20fe1-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.896 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:13.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:13.897 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:13.897 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:13.897 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.897 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.897 DEBUG:boto:Method: GET 2024-03-04T17:48:13.897 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.897 DEBUG:boto:Data: 2024-03-04T17:48:13.897 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.898 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.898 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.898 DEBUG:boto:Params: {} 2024-03-04T17:48:13.898 DEBUG:boto:Token: None 2024-03-04T17:48:13.898 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.898 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nbmDOkFZxkyjHv7NQ8S26h7/QtE= 2024-03-04T17:48:13.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nbmDOkFZxkyjHv7NQ8S26h7/QtE=', 'Content-Length': '0'} 2024-03-04T17:48:13.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6f1bf9616edec33-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.944 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:13.945 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:13.945 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.945 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.945 DEBUG:boto:Method: GET 2024-03-04T17:48:13.945 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.945 DEBUG:boto:Data: 2024-03-04T17:48:13.945 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.945 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.945 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.946 DEBUG:boto:Params: {} 2024-03-04T17:48:13.946 DEBUG:boto:Token: None 2024-03-04T17:48:13.946 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.946 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nbmDOkFZxkyjHv7NQ8S26h7/QtE= 2024-03-04T17:48:13.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nbmDOkFZxkyjHv7NQ8S26h7/QtE=', 'Content-Length': '0'} 2024-03-04T17:48:13.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e33a13eae6fe7c7e-0065e6095d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.992 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:13.992 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:13.993 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.993 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.993 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.993 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.993 DEBUG:boto:Data: 2024-03-04T17:48:13.993 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.993 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:13.993 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.993 DEBUG:boto:Params: {} 2024-03-04T17:48:13.993 DEBUG:boto:Token: None 2024-03-04T17:48:13.994 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.994 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:a3Dzqh04ewFoqGWrKpo8jMiGhIo= 2024-03-04T17:48:13.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:a3Dzqh04ewFoqGWrKpo8jMiGhIo=', 'Content-Length': '0'} 2024-03-04T17:48:13.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e022dbe4a0b1b97-0065e6095d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:13.997 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:13.997 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:13.997 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.997 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.997 DEBUG:boto:Method: HEAD 2024-03-04T17:48:13.998 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.998 DEBUG:boto:Data: 2024-03-04T17:48:13.998 DEBUG:boto:Headers: {} 2024-03-04T17:48:13.998 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:13.998 DEBUG:boto:Port: 8000 2024-03-04T17:48:13.998 DEBUG:boto:Params: {} 2024-03-04T17:48:13.998 DEBUG:boto:Token: None 2024-03-04T17:48:13.998 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:13 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:13.998 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:a3Dzqh04ewFoqGWrKpo8jMiGhIo= 2024-03-04T17:48:13.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:a3Dzqh04ewFoqGWrKpo8jMiGhIo=', 'Content-Length': '0'} 2024-03-04T17:48:14.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032d495e47a27f8fc-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:14.002 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.002 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.002 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.002 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.002 DEBUG:boto:Method: GET 2024-03-04T17:48:14.003 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.003 DEBUG:boto:Data: 2024-03-04T17:48:14.003 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.003 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:14.003 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.003 DEBUG:boto:Params: {} 2024-03-04T17:48:14.003 DEBUG:boto:Token: None 2024-03-04T17:48:14.004 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.004 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PG9BG2b6lsIIk5DpEDkO2kRERm8= 2024-03-04T17:48:14.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PG9BG2b6lsIIk5DpEDkO2kRERm8=', 'Content-Length': '0'} 2024-03-04T17:48:14.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af6dc8b4637322cf-0065e6095e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.048 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.048 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.048 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.048 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.048 DEBUG:boto:Method: GET 2024-03-04T17:48:14.049 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.049 DEBUG:boto:Data: 2024-03-04T17:48:14.049 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.049 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.049 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.049 DEBUG:boto:Params: {} 2024-03-04T17:48:14.049 DEBUG:boto:Token: None 2024-03-04T17:48:14.049 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.049 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PG9BG2b6lsIIk5DpEDkO2kRERm8= 2024-03-04T17:48:14.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PG9BG2b6lsIIk5DpEDkO2kRERm8=', 'Content-Length': '0'} 2024-03-04T17:48:14.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010b4504219314bdc-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.100 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.100 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.100 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:14.101 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:14.101 DEBUG:boto:Method: GET 2024-03-04T17:48:14.101 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:14.101 DEBUG:boto:Data: 2024-03-04T17:48:14.101 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.101 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:14.101 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.101 DEBUG:boto:Params: {} 2024-03-04T17:48:14.101 DEBUG:boto:Token: None 2024-03-04T17:48:14.101 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.102 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a7114f8426bf893-0065e6095e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.148 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.149 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.149 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.149 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:14.149 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:14.149 DEBUG:boto:Method: GET 2024-03-04T17:48:14.149 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:14.149 DEBUG:boto:Data: 2024-03-04T17:48:14.149 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.149 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.149 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.150 DEBUG:boto:Params: {} 2024-03-04T17:48:14.150 DEBUG:boto:Token: None 2024-03-04T17:48:14.150 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.150 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4ffcc5e378835aa-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.196 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.196 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.197 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.197 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.197 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.197 DEBUG:boto:Data: 2024-03-04T17:48:14.197 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.197 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:14.197 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.197 DEBUG:boto:Params: {} 2024-03-04T17:48:14.197 DEBUG:boto:Token: None 2024-03-04T17:48:14.197 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.198 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TvwZNselmz+I1UxxfIlXcqSAbs= 2024-03-04T17:48:14.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TvwZNselmz+I1UxxfIlXcqSAbs=', 'Content-Length': '0'} 2024-03-04T17:48:14.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9340a840a70577f-0065e6095e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.202 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.202 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.202 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.202 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.202 DEBUG:boto:Data: 2024-03-04T17:48:14.202 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.202 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.202 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.202 DEBUG:boto:Params: {} 2024-03-04T17:48:14.203 DEBUG:boto:Token: None 2024-03-04T17:48:14.203 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.203 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TvwZNselmz+I1UxxfIlXcqSAbs= 2024-03-04T17:48:14.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TvwZNselmz+I1UxxfIlXcqSAbs=', 'Content-Length': '0'} 2024-03-04T17:48:14.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a1a045c5bc301d9-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:14.207 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.207 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.207 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.207 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.207 DEBUG:boto:Method: GET 2024-03-04T17:48:14.207 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.207 DEBUG:boto:Data: 2024-03-04T17:48:14.207 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.207 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:14.208 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.208 DEBUG:boto:Params: {} 2024-03-04T17:48:14.208 DEBUG:boto:Token: None 2024-03-04T17:48:14.208 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.208 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PG9BG2b6lsIIk5DpEDkO2kRERm8= 2024-03-04T17:48:14.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PG9BG2b6lsIIk5DpEDkO2kRERm8=', 'Content-Length': '0'} 2024-03-04T17:48:14.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a51c6381412a2e4-0065e6095e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.256 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.256 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:14.256 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.257 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.257 DEBUG:boto:Method: GET 2024-03-04T17:48:14.257 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.257 DEBUG:boto:Data: 2024-03-04T17:48:14.257 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.257 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.257 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.257 DEBUG:boto:Params: {} 2024-03-04T17:48:14.257 DEBUG:boto:Token: None 2024-03-04T17:48:14.258 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:14.258 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PG9BG2b6lsIIk5DpEDkO2kRERm8= 2024-03-04T17:48:14.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PG9BG2b6lsIIk5DpEDkO2kRERm8=', 'Content-Length': '0'} 2024-03-04T17:48:14.261 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d34b10e57d5bbc0f-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.304 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.304 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.304 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:14.304 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:14.305 DEBUG:boto:Method: GET 2024-03-04T17:48:14.305 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:14.305 DEBUG:boto:Data: 2024-03-04T17:48:14.305 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.305 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:14.305 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.305 DEBUG:boto:Params: {} 2024-03-04T17:48:14.305 DEBUG:boto:Token: None 2024-03-04T17:48:14.305 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.305 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f00fbe8df313876c-0065e6095e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.311 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.312 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.312 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.312 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:14.312 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:14.312 DEBUG:boto:Method: GET 2024-03-04T17:48:14.312 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:14.312 DEBUG:boto:Data: 2024-03-04T17:48:14.312 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.313 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.313 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.313 DEBUG:boto:Params: {} 2024-03-04T17:48:14.313 DEBUG:boto:Token: None 2024-03-04T17:48:14.313 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.313 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a026d722ab5388ef-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.360 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.360 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-19 zones={test-zone2, test-zone1} 2024-03-04T17:48:14.361 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-19 zones={test-zone3, test-zone1} 2024-03-04T17:48:14.361 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.361 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.361 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.361 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:48:14.361 DEBUG:boto:Data: 2024-03-04T17:48:14.361 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.361 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.361 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.361 DEBUG:boto:Params: {} 2024-03-04T17:48:14.362 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T17:48:14.362 DEBUG:boto:Token: None 2024-03-04T17:48:14.362 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.362 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WzwUpeIN1sY3T69Q18nOa7GWg/Q= 2024-03-04T17:48:14.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WzwUpeIN1sY3T69Q18nOa7GWg/Q=', 'Content-Length': '0'} 2024-03-04T17:48:14.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a7f7706c2c20e63-0065e6095e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.368 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.368 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.369 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.369 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:48:14.369 DEBUG:boto:Data: 2024-03-04T17:48:14.369 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.369 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.369 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.369 DEBUG:boto:Params: {} 2024-03-04T17:48:14.369 DEBUG:boto:Token: None 2024-03-04T17:48:14.369 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.370 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WzwUpeIN1sY3T69Q18nOa7GWg/Q= 2024-03-04T17:48:14.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WzwUpeIN1sY3T69Q18nOa7GWg/Q=', 'Content-Length': '0'} 2024-03-04T17:48:14.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dc4b39342c2d082-0065e6095e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.376 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:48:14.376 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.376 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.376 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:14.376 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:14.376 DEBUG:boto:Method: GET 2024-03-04T17:48:14.376 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:14.376 DEBUG:boto:Data: 2024-03-04T17:48:14.377 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.377 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.377 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.377 DEBUG:boto:Params: {} 2024-03-04T17:48:14.377 DEBUG:boto:Token: None 2024-03-04T17:48:14.377 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/?versions 2024-03-04T17:48:14.377 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Z6Yh+rEFPuHlWU8wnTe1/Y7LUJg= 2024-03-04T17:48:14.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Z6Yh+rEFPuHlWU8wnTe1/Y7LUJg=', 'Content-Length': '0'} 2024-03-04T17:48:14.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088d852d6c0ee0d69-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.382 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.383 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:14.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:14.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:14.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:14.383 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:48:14.383 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.383 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.383 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:14.383 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:14.384 DEBUG:boto:Method: GET 2024-03-04T17:48:14.384 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:14.384 DEBUG:boto:Data: 2024-03-04T17:48:14.384 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.384 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.384 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.384 DEBUG:boto:Params: {} 2024-03-04T17:48:14.384 DEBUG:boto:Token: None 2024-03-04T17:48:14.384 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/?versions 2024-03-04T17:48:14.385 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Z6Yh+rEFPuHlWU8wnTe1/Y7LUJg= 2024-03-04T17:48:14.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Z6Yh+rEFPuHlWU8wnTe1/Y7LUJg=', 'Content-Length': '0'} 2024-03-04T17:48:14.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012661a2610e55581-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.390 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.390 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:14.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:14.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:14.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:14.391 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.391 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.391 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:14.391 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:14.391 DEBUG:boto:Method: GET 2024-03-04T17:48:14.391 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:14.391 DEBUG:boto:Data: 2024-03-04T17:48:14.391 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.392 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.392 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.392 DEBUG:boto:Params: {} 2024-03-04T17:48:14.392 DEBUG:boto:Token: None 2024-03-04T17:48:14.392 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/?versions 2024-03-04T17:48:14.392 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Z6Yh+rEFPuHlWU8wnTe1/Y7LUJg= 2024-03-04T17:48:14.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Z6Yh+rEFPuHlWU8wnTe1/Y7LUJg=', 'Content-Length': '0'} 2024-03-04T17:48:14.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095df8305b2aa5702-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.397 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.398 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.398 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.398 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:14.398 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:14.398 DEBUG:boto:Method: GET 2024-03-04T17:48:14.398 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:14.398 DEBUG:boto:Data: 2024-03-04T17:48:14.398 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.399 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.399 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.399 DEBUG:boto:Params: {} 2024-03-04T17:48:14.399 DEBUG:boto:Token: None 2024-03-04T17:48:14.399 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/?versions 2024-03-04T17:48:14.399 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Z6Yh+rEFPuHlWU8wnTe1/Y7LUJg= 2024-03-04T17:48:14.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Z6Yh+rEFPuHlWU8wnTe1/Y7LUJg=', 'Content-Length': '0'} 2024-03-04T17:48:14.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fbafc5d7b328660-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.404 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:14.405 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:14.405 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:14.405 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.405 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.406 DEBUG:boto:Method: GET 2024-03-04T17:48:14.406 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.406 DEBUG:boto:Data: 2024-03-04T17:48:14.406 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.406 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.406 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.406 DEBUG:boto:Params: {} 2024-03-04T17:48:14.406 DEBUG:boto:Token: None 2024-03-04T17:48:14.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.407 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+U5uWpUg+S3ScAuARNhRNbfYmz0= 2024-03-04T17:48:14.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+U5uWpUg+S3ScAuARNhRNbfYmz0=', 'Content-Length': '0'} 2024-03-04T17:48:14.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000004860472e30426e2-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.410 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:14.410 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:14.410 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.410 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.410 DEBUG:boto:Method: GET 2024-03-04T17:48:14.410 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.410 DEBUG:boto:Data: 2024-03-04T17:48:14.410 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.410 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.410 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.410 DEBUG:boto:Params: {} 2024-03-04T17:48:14.411 DEBUG:boto:Token: None 2024-03-04T17:48:14.411 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.411 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+U5uWpUg+S3ScAuARNhRNbfYmz0= 2024-03-04T17:48:14.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+U5uWpUg+S3ScAuARNhRNbfYmz0=', 'Content-Length': '0'} 2024-03-04T17:48:14.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a10992cdf2e64a45-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.414 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:14.414 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:14.414 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.414 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.414 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.414 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.414 DEBUG:boto:Data: 2024-03-04T17:48:14.414 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.414 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.415 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.415 DEBUG:boto:Params: {} 2024-03-04T17:48:14.415 DEBUG:boto:Token: None 2024-03-04T17:48:14.415 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.415 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+360hehXVfHWKEeuieMrbu/iiUs= 2024-03-04T17:48:14.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+360hehXVfHWKEeuieMrbu/iiUs=', 'Content-Length': '0'} 2024-03-04T17:48:14.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009608413e17c7fb5b-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.418 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:14.418 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:14.418 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.418 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.418 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.419 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.419 DEBUG:boto:Data: 2024-03-04T17:48:14.419 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.419 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.419 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.419 DEBUG:boto:Params: {} 2024-03-04T17:48:14.419 DEBUG:boto:Token: None 2024-03-04T17:48:14.419 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.419 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+360hehXVfHWKEeuieMrbu/iiUs= 2024-03-04T17:48:14.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+360hehXVfHWKEeuieMrbu/iiUs=', 'Content-Length': '0'} 2024-03-04T17:48:14.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ac01bc029bbf4a5-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:14.422 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:14.422 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:14.423 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.423 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.423 DEBUG:boto:Method: GET 2024-03-04T17:48:14.423 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.423 DEBUG:boto:Data: 2024-03-04T17:48:14.423 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.423 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.423 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.423 DEBUG:boto:Params: {} 2024-03-04T17:48:14.423 DEBUG:boto:Token: None 2024-03-04T17:48:14.424 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.424 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+U5uWpUg+S3ScAuARNhRNbfYmz0= 2024-03-04T17:48:14.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+U5uWpUg+S3ScAuARNhRNbfYmz0=', 'Content-Length': '0'} 2024-03-04T17:48:14.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dde8950529805833-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.427 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:14.427 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:14.427 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.427 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.427 DEBUG:boto:Method: GET 2024-03-04T17:48:14.427 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.428 DEBUG:boto:Data: 2024-03-04T17:48:14.428 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.428 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.428 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.428 DEBUG:boto:Params: {} 2024-03-04T17:48:14.428 DEBUG:boto:Token: None 2024-03-04T17:48:14.428 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.428 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+U5uWpUg+S3ScAuARNhRNbfYmz0= 2024-03-04T17:48:14.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+U5uWpUg+S3ScAuARNhRNbfYmz0=', 'Content-Length': '0'} 2024-03-04T17:48:14.431 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000734be070a7ca3086-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.431 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.431 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.431 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:14.432 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:14.432 DEBUG:boto:Method: GET 2024-03-04T17:48:14.432 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:48:14.432 DEBUG:boto:Data: 2024-03-04T17:48:14.432 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.432 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.432 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.432 DEBUG:boto:Params: {} 2024-03-04T17:48:14.432 DEBUG:boto:Token: None 2024-03-04T17:48:14.433 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.433 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046507ae382273efb-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.484 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.484 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.485 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.485 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:14.485 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:14.485 DEBUG:boto:Method: GET 2024-03-04T17:48:14.485 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:48:14.485 DEBUG:boto:Data: 2024-03-04T17:48:14.485 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.485 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.485 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.485 DEBUG:boto:Params: {} 2024-03-04T17:48:14.486 DEBUG:boto:Token: None 2024-03-04T17:48:14.486 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.486 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f84e5debe0babb0d-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.491 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.492 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:14.492 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:14.492 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.492 DEBUG:boto:Path: /ppfahn-19/dummy 2024-03-04T17:48:14.492 DEBUG:boto:Data: 2024-03-04T17:48:14.492 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.492 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.492 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.492 DEBUG:boto:Params: {} 2024-03-04T17:48:14.492 DEBUG:boto:Token: None 2024-03-04T17:48:14.493 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/dummy 2024-03-04T17:48:14.493 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xxZroVwiRU0Wi8o7J0kO4GV+3Eo= 2024-03-04T17:48:14.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xxZroVwiRU0Wi8o7J0kO4GV+3Eo=', 'Content-Length': '0'} 2024-03-04T17:48:14.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000072687f508b5cbc5d-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.497 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:14.497 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:14.497 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.497 DEBUG:boto:Path: /ppfahn-19/dummy 2024-03-04T17:48:14.497 DEBUG:boto:Data: 2024-03-04T17:48:14.497 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.497 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.497 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.497 DEBUG:boto:Params: {} 2024-03-04T17:48:14.498 DEBUG:boto:Token: None 2024-03-04T17:48:14.498 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/dummy 2024-03-04T17:48:14.498 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xxZroVwiRU0Wi8o7J0kO4GV+3Eo= 2024-03-04T17:48:14.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xxZroVwiRU0Wi8o7J0kO4GV+3Eo=', 'Content-Length': '0'} 2024-03-04T17:48:14.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e251edab5850eeb5-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:14.502 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:14.502 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:14.502 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.502 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.502 DEBUG:boto:Method: GET 2024-03-04T17:48:14.503 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.503 DEBUG:boto:Data: 2024-03-04T17:48:14.503 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.503 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.503 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.503 DEBUG:boto:Params: {} 2024-03-04T17:48:14.503 DEBUG:boto:Token: None 2024-03-04T17:48:14.503 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.503 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+U5uWpUg+S3ScAuARNhRNbfYmz0= 2024-03-04T17:48:14.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+U5uWpUg+S3ScAuARNhRNbfYmz0=', 'Content-Length': '0'} 2024-03-04T17:48:14.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000313fe1926ce5e1a9-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.506 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:14.506 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:14.507 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.507 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.507 DEBUG:boto:Method: GET 2024-03-04T17:48:14.507 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.507 DEBUG:boto:Data: 2024-03-04T17:48:14.507 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.507 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.507 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.507 DEBUG:boto:Params: {} 2024-03-04T17:48:14.507 DEBUG:boto:Token: None 2024-03-04T17:48:14.508 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:14.508 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+U5uWpUg+S3ScAuARNhRNbfYmz0= 2024-03-04T17:48:14.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+U5uWpUg+S3ScAuARNhRNbfYmz0=', 'Content-Length': '0'} 2024-03-04T17:48:14.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cee47c84b9c73081-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.511 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.511 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.511 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:14.511 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:14.511 DEBUG:boto:Method: GET 2024-03-04T17:48:14.511 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:48:14.511 DEBUG:boto:Data: 2024-03-04T17:48:14.512 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.512 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.512 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.512 DEBUG:boto:Params: {} 2024-03-04T17:48:14.512 DEBUG:boto:Token: None 2024-03-04T17:48:14.512 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.512 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062619e05e4162b3e-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.517 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.517 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.517 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.518 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:14.518 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:14.518 DEBUG:boto:Method: GET 2024-03-04T17:48:14.518 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:48:14.518 DEBUG:boto:Data: 2024-03-04T17:48:14.518 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.518 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.518 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.518 DEBUG:boto:Params: {} 2024-03-04T17:48:14.518 DEBUG:boto:Token: None 2024-03-04T17:48:14.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.519 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa6d55f53699f3d2-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.523 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:14.524 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.524 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.524 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.524 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.524 DEBUG:boto:Method: GET 2024-03-04T17:48:14.524 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.525 DEBUG:boto:Data: 2024-03-04T17:48:14.525 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.525 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.525 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.525 DEBUG:boto:Params: {} 2024-03-04T17:48:14.525 DEBUG:boto:Token: None 2024-03-04T17:48:14.525 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.525 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s5bruyBQNZr3Fik/f3mGu3SEqTc= 2024-03-04T17:48:14.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s5bruyBQNZr3Fik/f3mGu3SEqTc=', 'Content-Length': '0'} 2024-03-04T17:48:14.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003420cd4dda8afb50-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.576 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.576 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.576 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.576 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.577 DEBUG:boto:Method: GET 2024-03-04T17:48:14.577 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.577 DEBUG:boto:Data: 2024-03-04T17:48:14.577 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.577 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.577 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.577 DEBUG:boto:Params: {} 2024-03-04T17:48:14.577 DEBUG:boto:Token: None 2024-03-04T17:48:14.578 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.578 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s5bruyBQNZr3Fik/f3mGu3SEqTc= 2024-03-04T17:48:14.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s5bruyBQNZr3Fik/f3mGu3SEqTc=', 'Content-Length': '0'} 2024-03-04T17:48:14.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fc58a46e4de086a-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.581 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.581 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.581 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.581 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.581 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.582 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.582 DEBUG:boto:Data: 2024-03-04T17:48:14.582 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.582 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.582 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.582 DEBUG:boto:Params: {} 2024-03-04T17:48:14.582 DEBUG:boto:Token: None 2024-03-04T17:48:14.582 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.582 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/uv9uMJTdRRsyKRr6gqJPiTfZh4= 2024-03-04T17:48:14.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/uv9uMJTdRRsyKRr6gqJPiTfZh4=', 'Content-Length': '0'} 2024-03-04T17:48:14.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006a01274d3b73b9e-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.585 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.586 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.586 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.586 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.586 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.586 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.586 DEBUG:boto:Data: 2024-03-04T17:48:14.586 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.586 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.586 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.586 DEBUG:boto:Params: {} 2024-03-04T17:48:14.587 DEBUG:boto:Token: None 2024-03-04T17:48:14.587 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.587 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/uv9uMJTdRRsyKRr6gqJPiTfZh4= 2024-03-04T17:48:14.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/uv9uMJTdRRsyKRr6gqJPiTfZh4=', 'Content-Length': '0'} 2024-03-04T17:48:14.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066e14156818dd846-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:14.590 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.590 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.590 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.590 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.591 DEBUG:boto:Method: GET 2024-03-04T17:48:14.591 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.591 DEBUG:boto:Data: 2024-03-04T17:48:14.591 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.591 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.591 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.591 DEBUG:boto:Params: {} 2024-03-04T17:48:14.591 DEBUG:boto:Token: None 2024-03-04T17:48:14.591 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.591 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s5bruyBQNZr3Fik/f3mGu3SEqTc= 2024-03-04T17:48:14.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s5bruyBQNZr3Fik/f3mGu3SEqTc=', 'Content-Length': '0'} 2024-03-04T17:48:14.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001fcb6810470ce888-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.636 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.636 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.637 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.637 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.637 DEBUG:boto:Method: GET 2024-03-04T17:48:14.637 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.637 DEBUG:boto:Data: 2024-03-04T17:48:14.637 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.637 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.637 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.637 DEBUG:boto:Params: {} 2024-03-04T17:48:14.637 DEBUG:boto:Token: None 2024-03-04T17:48:14.638 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.638 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s5bruyBQNZr3Fik/f3mGu3SEqTc= 2024-03-04T17:48:14.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s5bruyBQNZr3Fik/f3mGu3SEqTc=', 'Content-Length': '0'} 2024-03-04T17:48:14.641 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000152c606e542cc2cf-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.641 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.641 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.642 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:14.642 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:14.642 DEBUG:boto:Method: GET 2024-03-04T17:48:14.642 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:14.642 DEBUG:boto:Data: 2024-03-04T17:48:14.642 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.642 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.642 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.642 DEBUG:boto:Params: {} 2024-03-04T17:48:14.642 DEBUG:boto:Token: None 2024-03-04T17:48:14.643 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.643 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a81dec0de8741fae-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.688 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.688 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.689 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.689 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:14.689 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:14.689 DEBUG:boto:Method: GET 2024-03-04T17:48:14.689 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:14.689 DEBUG:boto:Data: 2024-03-04T17:48:14.689 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.689 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.689 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.689 DEBUG:boto:Params: {} 2024-03-04T17:48:14.690 DEBUG:boto:Token: None 2024-03-04T17:48:14.690 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.690 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eed1e14a6ef8062-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.695 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.695 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.695 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.696 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.696 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.696 DEBUG:boto:Data: 2024-03-04T17:48:14.696 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.696 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.696 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.696 DEBUG:boto:Params: {} 2024-03-04T17:48:14.696 DEBUG:boto:Token: None 2024-03-04T17:48:14.696 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.697 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QGOM0aekabi1TWGsuOaqLM1OhOA= 2024-03-04T17:48:14.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QGOM0aekabi1TWGsuOaqLM1OhOA=', 'Content-Length': '0'} 2024-03-04T17:48:14.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9a0e1ba9c070eb7-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.701 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.701 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.701 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.701 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.701 DEBUG:boto:Data: 2024-03-04T17:48:14.701 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.701 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.701 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.701 DEBUG:boto:Params: {} 2024-03-04T17:48:14.702 DEBUG:boto:Token: None 2024-03-04T17:48:14.702 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.702 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QGOM0aekabi1TWGsuOaqLM1OhOA= 2024-03-04T17:48:14.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QGOM0aekabi1TWGsuOaqLM1OhOA=', 'Content-Length': '0'} 2024-03-04T17:48:14.706 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8244e96ceb8c6b8-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:14.706 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.706 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.707 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.707 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.707 DEBUG:boto:Method: GET 2024-03-04T17:48:14.707 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.707 DEBUG:boto:Data: 2024-03-04T17:48:14.707 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.707 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.707 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.707 DEBUG:boto:Params: {} 2024-03-04T17:48:14.707 DEBUG:boto:Token: None 2024-03-04T17:48:14.708 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.708 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s5bruyBQNZr3Fik/f3mGu3SEqTc= 2024-03-04T17:48:14.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s5bruyBQNZr3Fik/f3mGu3SEqTc=', 'Content-Length': '0'} 2024-03-04T17:48:14.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e6d521d1c9fea6c-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.756 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.756 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:14.756 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.757 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.757 DEBUG:boto:Method: GET 2024-03-04T17:48:14.757 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.757 DEBUG:boto:Data: 2024-03-04T17:48:14.757 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.757 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.757 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.757 DEBUG:boto:Params: {} 2024-03-04T17:48:14.757 DEBUG:boto:Token: None 2024-03-04T17:48:14.758 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:14.758 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s5bruyBQNZr3Fik/f3mGu3SEqTc= 2024-03-04T17:48:14.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s5bruyBQNZr3Fik/f3mGu3SEqTc=', 'Content-Length': '0'} 2024-03-04T17:48:14.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0f85d469f88bc2e-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.761 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.761 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.761 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:14.761 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:14.761 DEBUG:boto:Method: GET 2024-03-04T17:48:14.762 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:14.762 DEBUG:boto:Data: 2024-03-04T17:48:14.762 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.762 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.762 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.762 DEBUG:boto:Params: {} 2024-03-04T17:48:14.762 DEBUG:boto:Token: None 2024-03-04T17:48:14.762 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.762 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e50a949e9f32451-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.808 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.808 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.809 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.809 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:14.809 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:14.809 DEBUG:boto:Method: GET 2024-03-04T17:48:14.809 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:14.809 DEBUG:boto:Data: 2024-03-04T17:48:14.809 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.809 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.809 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.809 DEBUG:boto:Params: {} 2024-03-04T17:48:14.810 DEBUG:boto:Token: None 2024-03-04T17:48:14.810 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.810 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2735531f9f1ea46-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.815 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:14.815 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.815 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.816 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.816 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.816 DEBUG:boto:Method: GET 2024-03-04T17:48:14.816 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.816 DEBUG:boto:Data: 2024-03-04T17:48:14.816 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.816 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.816 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.816 DEBUG:boto:Params: {} 2024-03-04T17:48:14.816 DEBUG:boto:Token: None 2024-03-04T17:48:14.817 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.817 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wXbp5T30DX3+kYKfzoloLLA85nI= 2024-03-04T17:48:14.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wXbp5T30DX3+kYKfzoloLLA85nI=', 'Content-Length': '0'} 2024-03-04T17:48:14.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000106b0d4586bcb914-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.820 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.820 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.820 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.820 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.820 DEBUG:boto:Method: GET 2024-03-04T17:48:14.821 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.821 DEBUG:boto:Data: 2024-03-04T17:48:14.821 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.821 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.821 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.821 DEBUG:boto:Params: {} 2024-03-04T17:48:14.821 DEBUG:boto:Token: None 2024-03-04T17:48:14.821 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.821 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wXbp5T30DX3+kYKfzoloLLA85nI= 2024-03-04T17:48:14.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wXbp5T30DX3+kYKfzoloLLA85nI=', 'Content-Length': '0'} 2024-03-04T17:48:14.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a76e80cdf1f0dbc-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.825 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.825 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.825 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.825 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.825 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.825 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.825 DEBUG:boto:Data: 2024-03-04T17:48:14.825 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.825 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.826 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.826 DEBUG:boto:Params: {} 2024-03-04T17:48:14.826 DEBUG:boto:Token: None 2024-03-04T17:48:14.826 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.826 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:19sBgE7OpKGByOtNQTlvZKGTfw4= 2024-03-04T17:48:14.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:19sBgE7OpKGByOtNQTlvZKGTfw4=', 'Content-Length': '0'} 2024-03-04T17:48:14.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000907d2d9bd28862d8-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.829 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.829 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.829 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.830 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.830 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.830 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.830 DEBUG:boto:Data: 2024-03-04T17:48:14.830 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.830 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.830 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.830 DEBUG:boto:Params: {} 2024-03-04T17:48:14.830 DEBUG:boto:Token: None 2024-03-04T17:48:14.830 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.831 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:19sBgE7OpKGByOtNQTlvZKGTfw4= 2024-03-04T17:48:14.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:19sBgE7OpKGByOtNQTlvZKGTfw4=', 'Content-Length': '0'} 2024-03-04T17:48:14.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ed42d45dc813fb0-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:14.834 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.834 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.834 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.834 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.834 DEBUG:boto:Method: GET 2024-03-04T17:48:14.834 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.834 DEBUG:boto:Data: 2024-03-04T17:48:14.834 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.834 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.835 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.835 DEBUG:boto:Params: {} 2024-03-04T17:48:14.835 DEBUG:boto:Token: None 2024-03-04T17:48:14.835 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.835 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wXbp5T30DX3+kYKfzoloLLA85nI= 2024-03-04T17:48:14.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wXbp5T30DX3+kYKfzoloLLA85nI=', 'Content-Length': '0'} 2024-03-04T17:48:14.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b774a40f37ae2459-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.880 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.880 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.880 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.880 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.880 DEBUG:boto:Method: GET 2024-03-04T17:48:14.881 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.881 DEBUG:boto:Data: 2024-03-04T17:48:14.881 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.881 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.881 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.881 DEBUG:boto:Params: {} 2024-03-04T17:48:14.881 DEBUG:boto:Token: None 2024-03-04T17:48:14.881 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.882 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wXbp5T30DX3+kYKfzoloLLA85nI= 2024-03-04T17:48:14.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wXbp5T30DX3+kYKfzoloLLA85nI=', 'Content-Length': '0'} 2024-03-04T17:48:14.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e0ffd2b3f2d8618-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.928 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.928 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.928 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:14.928 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:14.929 DEBUG:boto:Method: GET 2024-03-04T17:48:14.929 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:14.929 DEBUG:boto:Data: 2024-03-04T17:48:14.929 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.929 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.929 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.929 DEBUG:boto:Params: {} 2024-03-04T17:48:14.929 DEBUG:boto:Token: None 2024-03-04T17:48:14.929 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.930 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3e1b1035cd56fbf-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.980 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.981 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:14.981 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:14.981 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:14.981 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:14.981 DEBUG:boto:Method: GET 2024-03-04T17:48:14.981 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:14.981 DEBUG:boto:Data: 2024-03-04T17:48:14.981 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.981 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.982 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.982 DEBUG:boto:Params: {} 2024-03-04T17:48:14.982 DEBUG:boto:Token: None 2024-03-04T17:48:14.982 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/ 2024-03-04T17:48:14.982 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug= 2024-03-04T17:48:14.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:afrSS0qhsK7Vp1H2d21z8oVIjug=', 'Content-Length': '0'} 2024-03-04T17:48:14.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d87daa4ce27c0fcf-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.987 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:14.987 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.987 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.987 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.988 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.988 DEBUG:boto:Data: 2024-03-04T17:48:14.988 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.988 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.988 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.988 DEBUG:boto:Params: {} 2024-03-04T17:48:14.988 DEBUG:boto:Token: None 2024-03-04T17:48:14.988 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.988 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:O1fiz7FUgXiuNZqiVCy18Na4usg= 2024-03-04T17:48:14.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:O1fiz7FUgXiuNZqiVCy18Na4usg=', 'Content-Length': '0'} 2024-03-04T17:48:14.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f01740b74cda63f3-0065e6095e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.992 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.992 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.993 DEBUG:boto:Method: HEAD 2024-03-04T17:48:14.993 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.993 DEBUG:boto:Data: 2024-03-04T17:48:14.993 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.993 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:14.993 DEBUG:boto:Port: 8000 2024-03-04T17:48:14.993 DEBUG:boto:Params: {} 2024-03-04T17:48:14.993 DEBUG:boto:Token: None 2024-03-04T17:48:14.993 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.993 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:O1fiz7FUgXiuNZqiVCy18Na4usg= 2024-03-04T17:48:14.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:O1fiz7FUgXiuNZqiVCy18Na4usg=', 'Content-Length': '0'} 2024-03-04T17:48:14.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056a045e40e193f5b-0065e6095e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:14.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:14.998 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.998 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:14.998 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.998 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.998 DEBUG:boto:Method: GET 2024-03-04T17:48:14.998 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.998 DEBUG:boto:Data: 2024-03-04T17:48:14.998 DEBUG:boto:Headers: {} 2024-03-04T17:48:14.998 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:14.999 DEBUG:boto:Port: 8001 2024-03-04T17:48:14.999 DEBUG:boto:Params: {} 2024-03-04T17:48:14.999 DEBUG:boto:Token: None 2024-03-04T17:48:14.999 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:14 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:14.999 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wXbp5T30DX3+kYKfzoloLLA85nI= 2024-03-04T17:48:14.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wXbp5T30DX3+kYKfzoloLLA85nI=', 'Content-Length': '0'} 2024-03-04T17:48:15.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c033c3cc0c4059c-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.048 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:15.048 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:15.048 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:15.048 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:15.048 DEBUG:boto:Method: GET 2024-03-04T17:48:15.049 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:15.049 DEBUG:boto:Data: 2024-03-04T17:48:15.049 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.049 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:15.049 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.049 DEBUG:boto:Params: {} 2024-03-04T17:48:15.049 DEBUG:boto:Token: None 2024-03-04T17:48:15.049 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:15.049 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Inzwy2PvIKQ/7aw/8aNzjX5GeGg= 2024-03-04T17:48:15.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Inzwy2PvIKQ/7aw/8aNzjX5GeGg=', 'Content-Length': '0'} 2024-03-04T17:48:15.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebf1058c89864443-0065e6095f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.096 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.096 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.096 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:15.096 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:15.096 DEBUG:boto:Method: GET 2024-03-04T17:48:15.097 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:15.097 DEBUG:boto:Data: 2024-03-04T17:48:15.097 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.097 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.097 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.097 DEBUG:boto:Params: {} 2024-03-04T17:48:15.097 DEBUG:boto:Token: None 2024-03-04T17:48:15.097 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.097 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea6e9f665f8f49bb-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.102 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.103 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.103 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.103 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:15.103 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:15.103 DEBUG:boto:Method: GET 2024-03-04T17:48:15.104 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:15.104 DEBUG:boto:Data: 2024-03-04T17:48:15.104 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.104 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:15.104 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.104 DEBUG:boto:Params: {} 2024-03-04T17:48:15.104 DEBUG:boto:Token: None 2024-03-04T17:48:15.104 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.104 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a22fdb7d5f496c6-0065e6095f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.110 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:15.110 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.111 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.111 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.111 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.111 DEBUG:boto:Method: GET 2024-03-04T17:48:15.111 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.111 DEBUG:boto:Data: 2024-03-04T17:48:15.111 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.111 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.111 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.111 DEBUG:boto:Params: {} 2024-03-04T17:48:15.111 DEBUG:boto:Token: None 2024-03-04T17:48:15.112 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.112 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DHONWha7Aqhpf0Xtsrpz0MZ+tdw= 2024-03-04T17:48:15.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DHONWha7Aqhpf0Xtsrpz0MZ+tdw=', 'Content-Length': '0'} 2024-03-04T17:48:15.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038a977f699608e2b-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.160 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.160 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.161 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.161 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.161 DEBUG:boto:Method: GET 2024-03-04T17:48:15.161 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.161 DEBUG:boto:Data: 2024-03-04T17:48:15.161 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.161 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:15.161 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.161 DEBUG:boto:Params: {} 2024-03-04T17:48:15.161 DEBUG:boto:Token: None 2024-03-04T17:48:15.162 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.162 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DHONWha7Aqhpf0Xtsrpz0MZ+tdw= 2024-03-04T17:48:15.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DHONWha7Aqhpf0Xtsrpz0MZ+tdw=', 'Content-Length': '0'} 2024-03-04T17:48:15.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010684fe59f09db51-0065e6095f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.208 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.208 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.208 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.208 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.208 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.209 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.209 DEBUG:boto:Data: 2024-03-04T17:48:15.209 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.209 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.209 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.209 DEBUG:boto:Params: {} 2024-03-04T17:48:15.209 DEBUG:boto:Token: None 2024-03-04T17:48:15.209 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.209 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iLOv3aKh39EDfDLLboOabrAm0E0= 2024-03-04T17:48:15.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iLOv3aKh39EDfDLLboOabrAm0E0=', 'Content-Length': '0'} 2024-03-04T17:48:15.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1eedf3c0b4bf53a-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.213 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.213 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.213 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.213 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.213 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.213 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.213 DEBUG:boto:Data: 2024-03-04T17:48:15.213 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.214 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:15.214 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.214 DEBUG:boto:Params: {} 2024-03-04T17:48:15.214 DEBUG:boto:Token: None 2024-03-04T17:48:15.214 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.214 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iLOv3aKh39EDfDLLboOabrAm0E0= 2024-03-04T17:48:15.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iLOv3aKh39EDfDLLboOabrAm0E0=', 'Content-Length': '0'} 2024-03-04T17:48:15.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000292f057c7766f907-0065e6095f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:15.217 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.217 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.217 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.218 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.218 DEBUG:boto:Method: GET 2024-03-04T17:48:15.218 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.218 DEBUG:boto:Data: 2024-03-04T17:48:15.218 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.218 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.218 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.218 DEBUG:boto:Params: {} 2024-03-04T17:48:15.218 DEBUG:boto:Token: None 2024-03-04T17:48:15.218 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.219 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DHONWha7Aqhpf0Xtsrpz0MZ+tdw= 2024-03-04T17:48:15.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DHONWha7Aqhpf0Xtsrpz0MZ+tdw=', 'Content-Length': '0'} 2024-03-04T17:48:15.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000368ad1a94d6b12bd-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.222 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.222 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.222 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.222 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.222 DEBUG:boto:Method: GET 2024-03-04T17:48:15.222 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.222 DEBUG:boto:Data: 2024-03-04T17:48:15.223 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.223 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:15.223 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.223 DEBUG:boto:Params: {} 2024-03-04T17:48:15.223 DEBUG:boto:Token: None 2024-03-04T17:48:15.223 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.223 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DHONWha7Aqhpf0Xtsrpz0MZ+tdw= 2024-03-04T17:48:15.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DHONWha7Aqhpf0Xtsrpz0MZ+tdw=', 'Content-Length': '0'} 2024-03-04T17:48:15.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a4bc91ee1d4632d-0065e6095f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.268 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.268 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.268 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:15.268 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:15.269 DEBUG:boto:Method: GET 2024-03-04T17:48:15.269 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:15.269 DEBUG:boto:Data: 2024-03-04T17:48:15.269 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.269 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.269 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.269 DEBUG:boto:Params: {} 2024-03-04T17:48:15.269 DEBUG:boto:Token: None 2024-03-04T17:48:15.269 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.270 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c61aca3af9538c3-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.275 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.275 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.275 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.276 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:15.276 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:15.276 DEBUG:boto:Method: GET 2024-03-04T17:48:15.276 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:15.276 DEBUG:boto:Data: 2024-03-04T17:48:15.276 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.276 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:15.276 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.276 DEBUG:boto:Params: {} 2024-03-04T17:48:15.277 DEBUG:boto:Token: None 2024-03-04T17:48:15.277 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.277 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006514b18c4ed7a664-0065e6095f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.282 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.282 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.282 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.283 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.283 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.283 DEBUG:boto:Data: 2024-03-04T17:48:15.283 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.283 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.283 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.283 DEBUG:boto:Params: {} 2024-03-04T17:48:15.283 DEBUG:boto:Token: None 2024-03-04T17:48:15.283 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.284 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9ExkttHxV4iExy0c2TeSFGkqB+M= 2024-03-04T17:48:15.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9ExkttHxV4iExy0c2TeSFGkqB+M=', 'Content-Length': '0'} 2024-03-04T17:48:15.287 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001bc06f737eaabd3-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.287 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.287 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.288 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.288 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.288 DEBUG:boto:Data: 2024-03-04T17:48:15.288 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.288 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:15.288 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.288 DEBUG:boto:Params: {} 2024-03-04T17:48:15.288 DEBUG:boto:Token: None 2024-03-04T17:48:15.288 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.289 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9ExkttHxV4iExy0c2TeSFGkqB+M= 2024-03-04T17:48:15.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9ExkttHxV4iExy0c2TeSFGkqB+M=', 'Content-Length': '0'} 2024-03-04T17:48:15.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006e9a78a90469646-0065e6095f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:15.292 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.293 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.293 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.293 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.293 DEBUG:boto:Method: GET 2024-03-04T17:48:15.293 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.293 DEBUG:boto:Data: 2024-03-04T17:48:15.293 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.293 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.293 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.293 DEBUG:boto:Params: {} 2024-03-04T17:48:15.294 DEBUG:boto:Token: None 2024-03-04T17:48:15.294 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.294 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DHONWha7Aqhpf0Xtsrpz0MZ+tdw= 2024-03-04T17:48:15.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DHONWha7Aqhpf0Xtsrpz0MZ+tdw=', 'Content-Length': '0'} 2024-03-04T17:48:15.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082fee4d24cc7ec9c-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.297 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.297 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:15.297 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.297 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.297 DEBUG:boto:Method: GET 2024-03-04T17:48:15.297 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.297 DEBUG:boto:Data: 2024-03-04T17:48:15.297 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.298 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:15.298 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.298 DEBUG:boto:Params: {} 2024-03-04T17:48:15.298 DEBUG:boto:Token: None 2024-03-04T17:48:15.298 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:15.298 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DHONWha7Aqhpf0Xtsrpz0MZ+tdw= 2024-03-04T17:48:15.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DHONWha7Aqhpf0Xtsrpz0MZ+tdw=', 'Content-Length': '0'} 2024-03-04T17:48:15.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f49d6d8d44a8e10-0065e6095f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.348 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.348 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.348 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:15.348 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:15.349 DEBUG:boto:Method: GET 2024-03-04T17:48:15.349 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:15.349 DEBUG:boto:Data: 2024-03-04T17:48:15.349 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.349 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.349 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.349 DEBUG:boto:Params: {} 2024-03-04T17:48:15.349 DEBUG:boto:Token: None 2024-03-04T17:48:15.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.350 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036f493528a168903-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.354 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.355 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.355 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.355 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:15.355 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:15.355 DEBUG:boto:Method: GET 2024-03-04T17:48:15.356 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:15.356 DEBUG:boto:Data: 2024-03-04T17:48:15.356 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.356 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:15.356 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.356 DEBUG:boto:Params: {} 2024-03-04T17:48:15.356 DEBUG:boto:Token: None 2024-03-04T17:48:15.356 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.356 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac91c818fb8b606d-0065e6095f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.363 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.363 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-19 zones={test-zone3, test-zone1} 2024-03-04T17:48:15.363 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-19 zones={test-zone3, test-zone2} 2024-03-04T17:48:15.363 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.363 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.363 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.364 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:48:15.364 DEBUG:boto:Data: 2024-03-04T17:48:15.364 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.364 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.364 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.364 DEBUG:boto:Params: {} 2024-03-04T17:48:15.364 DEBUG:boto:Token: None 2024-03-04T17:48:15.364 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.365 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uX3Lq3H/9ndXm18hPJaW5Qp4rcA= 2024-03-04T17:48:15.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uX3Lq3H/9ndXm18hPJaW5Qp4rcA=', 'Content-Length': '0'} 2024-03-04T17:48:15.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b6b84cd320a4c26-0065e6095f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.370 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.370 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.370 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.370 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:48:15.370 DEBUG:boto:Data: 2024-03-04T17:48:15.370 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.370 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.370 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.370 DEBUG:boto:Params: {} 2024-03-04T17:48:15.370 DEBUG:boto:Token: None 2024-03-04T17:48:15.371 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.371 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uX3Lq3H/9ndXm18hPJaW5Qp4rcA= 2024-03-04T17:48:15.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uX3Lq3H/9ndXm18hPJaW5Qp4rcA=', 'Content-Length': '0'} 2024-03-04T17:48:15.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022e337c2db374ccd-0065e6095f-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.375 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:48:15.375 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.375 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.375 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:15.375 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:15.376 DEBUG:boto:Method: GET 2024-03-04T17:48:15.376 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:15.376 DEBUG:boto:Data: 2024-03-04T17:48:15.376 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.376 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.376 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.376 DEBUG:boto:Params: {} 2024-03-04T17:48:15.376 DEBUG:boto:Token: None 2024-03-04T17:48:15.376 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/?versions 2024-03-04T17:48:15.377 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kbw/dmvvH4AgCQlekRvMDLITf2w= 2024-03-04T17:48:15.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kbw/dmvvH4AgCQlekRvMDLITf2w=', 'Content-Length': '0'} 2024-03-04T17:48:15.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8962cdeb461dffd-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.382 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.382 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:15.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:15.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:15.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:15.383 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:48:15.383 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.383 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.383 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:15.383 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:15.383 DEBUG:boto:Method: GET 2024-03-04T17:48:15.383 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:15.384 DEBUG:boto:Data: 2024-03-04T17:48:15.384 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.384 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.384 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.384 DEBUG:boto:Params: {} 2024-03-04T17:48:15.384 DEBUG:boto:Token: None 2024-03-04T17:48:15.384 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/?versions 2024-03-04T17:48:15.384 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kbw/dmvvH4AgCQlekRvMDLITf2w= 2024-03-04T17:48:15.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kbw/dmvvH4AgCQlekRvMDLITf2w=', 'Content-Length': '0'} 2024-03-04T17:48:15.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eee726f357ce694-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.390 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.390 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:15.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:15.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:15.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:15.391 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.391 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.391 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:15.391 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:15.391 DEBUG:boto:Method: GET 2024-03-04T17:48:15.391 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:15.391 DEBUG:boto:Data: 2024-03-04T17:48:15.391 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.391 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.392 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.392 DEBUG:boto:Params: {} 2024-03-04T17:48:15.392 DEBUG:boto:Token: None 2024-03-04T17:48:15.392 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/?versions 2024-03-04T17:48:15.392 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kbw/dmvvH4AgCQlekRvMDLITf2w= 2024-03-04T17:48:15.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kbw/dmvvH4AgCQlekRvMDLITf2w=', 'Content-Length': '0'} 2024-03-04T17:48:15.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b094a8991d41c98a-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.397 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.398 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.398 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.398 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:15.398 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:15.398 DEBUG:boto:Method: GET 2024-03-04T17:48:15.398 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:48:15.398 DEBUG:boto:Data: 2024-03-04T17:48:15.398 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.399 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.399 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.399 DEBUG:boto:Params: {} 2024-03-04T17:48:15.399 DEBUG:boto:Token: None 2024-03-04T17:48:15.399 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/?versions 2024-03-04T17:48:15.399 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kbw/dmvvH4AgCQlekRvMDLITf2w= 2024-03-04T17:48:15.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kbw/dmvvH4AgCQlekRvMDLITf2w=', 'Content-Length': '0'} 2024-03-04T17:48:15.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068227f9441434fd6-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.404 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:15.405 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:15.405 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:15.405 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.405 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.405 DEBUG:boto:Method: GET 2024-03-04T17:48:15.405 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.405 DEBUG:boto:Data: 2024-03-04T17:48:15.405 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.405 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.405 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.405 DEBUG:boto:Params: {} 2024-03-04T17:48:15.406 DEBUG:boto:Token: None 2024-03-04T17:48:15.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.406 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BV8MiOdHmX8bvVIWUH0Uy2t+jHg= 2024-03-04T17:48:15.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BV8MiOdHmX8bvVIWUH0Uy2t+jHg=', 'Content-Length': '0'} 2024-03-04T17:48:15.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073a9de4fdbdef202-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.409 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:15.409 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:15.409 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.409 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.409 DEBUG:boto:Method: GET 2024-03-04T17:48:15.410 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.410 DEBUG:boto:Data: 2024-03-04T17:48:15.410 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.410 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.410 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.410 DEBUG:boto:Params: {} 2024-03-04T17:48:15.410 DEBUG:boto:Token: None 2024-03-04T17:48:15.410 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.410 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BV8MiOdHmX8bvVIWUH0Uy2t+jHg= 2024-03-04T17:48:15.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BV8MiOdHmX8bvVIWUH0Uy2t+jHg=', 'Content-Length': '0'} 2024-03-04T17:48:15.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000526e517dc7986392-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.413 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:15.414 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:15.414 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.414 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.414 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.414 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.414 DEBUG:boto:Data: 2024-03-04T17:48:15.414 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.414 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.414 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.414 DEBUG:boto:Params: {} 2024-03-04T17:48:15.414 DEBUG:boto:Token: None 2024-03-04T17:48:15.415 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.415 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PL812aHTkPyFqDIIoU6u2RZWUPc= 2024-03-04T17:48:15.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PL812aHTkPyFqDIIoU6u2RZWUPc=', 'Content-Length': '0'} 2024-03-04T17:48:15.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b58870c24b26e71e-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.418 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:15.418 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:15.418 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.418 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.418 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.418 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.419 DEBUG:boto:Data: 2024-03-04T17:48:15.419 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.419 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.419 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.419 DEBUG:boto:Params: {} 2024-03-04T17:48:15.419 DEBUG:boto:Token: None 2024-03-04T17:48:15.419 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.419 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PL812aHTkPyFqDIIoU6u2RZWUPc= 2024-03-04T17:48:15.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PL812aHTkPyFqDIIoU6u2RZWUPc=', 'Content-Length': '0'} 2024-03-04T17:48:15.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007772fe445f5c5011-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:15.422 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:15.423 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:15.423 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.423 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.423 DEBUG:boto:Method: GET 2024-03-04T17:48:15.423 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.423 DEBUG:boto:Data: 2024-03-04T17:48:15.423 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.423 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.423 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.423 DEBUG:boto:Params: {} 2024-03-04T17:48:15.423 DEBUG:boto:Token: None 2024-03-04T17:48:15.424 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.424 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BV8MiOdHmX8bvVIWUH0Uy2t+jHg= 2024-03-04T17:48:15.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BV8MiOdHmX8bvVIWUH0Uy2t+jHg=', 'Content-Length': '0'} 2024-03-04T17:48:15.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c02646b6a14329d6-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.427 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:15.427 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:15.427 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.427 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.427 DEBUG:boto:Method: GET 2024-03-04T17:48:15.427 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.427 DEBUG:boto:Data: 2024-03-04T17:48:15.428 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.428 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.428 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.428 DEBUG:boto:Params: {} 2024-03-04T17:48:15.428 DEBUG:boto:Token: None 2024-03-04T17:48:15.428 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.428 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BV8MiOdHmX8bvVIWUH0Uy2t+jHg= 2024-03-04T17:48:15.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BV8MiOdHmX8bvVIWUH0Uy2t+jHg=', 'Content-Length': '0'} 2024-03-04T17:48:15.431 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007621b62b65f8ac7d-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.431 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.432 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.432 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:15.432 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:15.432 DEBUG:boto:Method: GET 2024-03-04T17:48:15.432 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:48:15.432 DEBUG:boto:Data: 2024-03-04T17:48:15.432 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.432 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.432 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.432 DEBUG:boto:Params: {} 2024-03-04T17:48:15.432 DEBUG:boto:Token: None 2024-03-04T17:48:15.433 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.433 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ac064852f0ec2c6-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.484 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.485 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.485 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.485 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:15.485 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:15.485 DEBUG:boto:Method: GET 2024-03-04T17:48:15.485 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:48:15.485 DEBUG:boto:Data: 2024-03-04T17:48:15.485 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.485 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.486 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.486 DEBUG:boto:Params: {} 2024-03-04T17:48:15.486 DEBUG:boto:Token: None 2024-03-04T17:48:15.486 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.486 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b4468c943677f7e-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.491 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.491 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:15.492 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:15.492 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.492 DEBUG:boto:Path: /ppfahn-19/dummy 2024-03-04T17:48:15.492 DEBUG:boto:Data: 2024-03-04T17:48:15.492 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.492 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.492 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.492 DEBUG:boto:Params: {} 2024-03-04T17:48:15.492 DEBUG:boto:Token: None 2024-03-04T17:48:15.492 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/dummy 2024-03-04T17:48:15.493 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7eg5HK1Xv02NG6RM9WnZiIf+Pd4= 2024-03-04T17:48:15.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7eg5HK1Xv02NG6RM9WnZiIf+Pd4=', 'Content-Length': '0'} 2024-03-04T17:48:15.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000002ac967796e2e96f-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.497 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:15.497 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:15.497 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.497 DEBUG:boto:Path: /ppfahn-19/dummy 2024-03-04T17:48:15.497 DEBUG:boto:Data: 2024-03-04T17:48:15.497 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.497 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.497 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.497 DEBUG:boto:Params: {} 2024-03-04T17:48:15.498 DEBUG:boto:Token: None 2024-03-04T17:48:15.498 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/dummy 2024-03-04T17:48:15.498 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7eg5HK1Xv02NG6RM9WnZiIf+Pd4= 2024-03-04T17:48:15.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7eg5HK1Xv02NG6RM9WnZiIf+Pd4=', 'Content-Length': '0'} 2024-03-04T17:48:15.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036901017fde1aa33-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:15.502 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:15.502 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:15.502 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.502 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.502 DEBUG:boto:Method: GET 2024-03-04T17:48:15.502 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.503 DEBUG:boto:Data: 2024-03-04T17:48:15.503 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.503 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.503 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.503 DEBUG:boto:Params: {} 2024-03-04T17:48:15.503 DEBUG:boto:Token: None 2024-03-04T17:48:15.503 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.503 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BV8MiOdHmX8bvVIWUH0Uy2t+jHg= 2024-03-04T17:48:15.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BV8MiOdHmX8bvVIWUH0Uy2t+jHg=', 'Content-Length': '0'} 2024-03-04T17:48:15.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a087a335cfe16ed-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.506 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:48:15.507 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:48:15.507 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.507 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.507 DEBUG:boto:Method: GET 2024-03-04T17:48:15.507 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.507 DEBUG:boto:Data: 2024-03-04T17:48:15.507 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.507 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.507 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.507 DEBUG:boto:Params: {} 2024-03-04T17:48:15.508 DEBUG:boto:Token: None 2024-03-04T17:48:15.508 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:48:15.508 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BV8MiOdHmX8bvVIWUH0Uy2t+jHg= 2024-03-04T17:48:15.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BV8MiOdHmX8bvVIWUH0Uy2t+jHg=', 'Content-Length': '0'} 2024-03-04T17:48:15.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b29efd80fdb10eb2-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.511 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.511 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.511 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:15.511 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:15.511 DEBUG:boto:Method: GET 2024-03-04T17:48:15.512 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:48:15.512 DEBUG:boto:Data: 2024-03-04T17:48:15.512 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.512 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.512 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.512 DEBUG:boto:Params: {} 2024-03-04T17:48:15.512 DEBUG:boto:Token: None 2024-03-04T17:48:15.512 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.512 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f801c6b999382ea5-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.564 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.565 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.565 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.565 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:15.565 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:48:15.565 DEBUG:boto:Method: GET 2024-03-04T17:48:15.565 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:48:15.565 DEBUG:boto:Data: 2024-03-04T17:48:15.565 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.566 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.566 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.566 DEBUG:boto:Params: {} 2024-03-04T17:48:15.566 DEBUG:boto:Token: None 2024-03-04T17:48:15.566 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.566 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c5f3e1bf050a6d8-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.571 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:15.572 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.572 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.572 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.572 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.572 DEBUG:boto:Method: GET 2024-03-04T17:48:15.572 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.572 DEBUG:boto:Data: 2024-03-04T17:48:15.572 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.572 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.573 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.573 DEBUG:boto:Params: {} 2024-03-04T17:48:15.573 DEBUG:boto:Token: None 2024-03-04T17:48:15.573 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.573 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tq4vIBHHH+U6S1E7O/4yg/idlJM= 2024-03-04T17:48:15.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tq4vIBHHH+U6S1E7O/4yg/idlJM=', 'Content-Length': '0'} 2024-03-04T17:48:15.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1e9a46cdcafc021-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.620 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.620 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.620 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.621 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.621 DEBUG:boto:Method: GET 2024-03-04T17:48:15.621 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.621 DEBUG:boto:Data: 2024-03-04T17:48:15.621 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.621 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.621 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.621 DEBUG:boto:Params: {} 2024-03-04T17:48:15.621 DEBUG:boto:Token: None 2024-03-04T17:48:15.621 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.622 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tq4vIBHHH+U6S1E7O/4yg/idlJM= 2024-03-04T17:48:15.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tq4vIBHHH+U6S1E7O/4yg/idlJM=', 'Content-Length': '0'} 2024-03-04T17:48:15.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000633e08006e07fe09-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.668 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.668 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.668 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.668 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.668 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.669 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.669 DEBUG:boto:Data: 2024-03-04T17:48:15.669 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.669 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.669 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.669 DEBUG:boto:Params: {} 2024-03-04T17:48:15.669 DEBUG:boto:Token: None 2024-03-04T17:48:15.669 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.669 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1EJl6l2kwirwYliqmgsyoMI0MIM= 2024-03-04T17:48:15.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1EJl6l2kwirwYliqmgsyoMI0MIM=', 'Content-Length': '0'} 2024-03-04T17:48:15.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e4407ab6f974a5b-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.673 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.673 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.673 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.673 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.673 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.673 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.674 DEBUG:boto:Data: 2024-03-04T17:48:15.674 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.674 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.674 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.674 DEBUG:boto:Params: {} 2024-03-04T17:48:15.674 DEBUG:boto:Token: None 2024-03-04T17:48:15.674 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.674 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1EJl6l2kwirwYliqmgsyoMI0MIM= 2024-03-04T17:48:15.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1EJl6l2kwirwYliqmgsyoMI0MIM=', 'Content-Length': '0'} 2024-03-04T17:48:15.677 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037cc84d9c91ab18c-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:15.678 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.678 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.678 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.678 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.678 DEBUG:boto:Method: GET 2024-03-04T17:48:15.678 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.678 DEBUG:boto:Data: 2024-03-04T17:48:15.678 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.678 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.679 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.679 DEBUG:boto:Params: {} 2024-03-04T17:48:15.679 DEBUG:boto:Token: None 2024-03-04T17:48:15.679 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.679 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tq4vIBHHH+U6S1E7O/4yg/idlJM= 2024-03-04T17:48:15.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tq4vIBHHH+U6S1E7O/4yg/idlJM=', 'Content-Length': '0'} 2024-03-04T17:48:15.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8c31234dab77f17-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.724 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.724 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.724 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.724 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.725 DEBUG:boto:Method: GET 2024-03-04T17:48:15.725 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.725 DEBUG:boto:Data: 2024-03-04T17:48:15.725 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.725 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.725 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.725 DEBUG:boto:Params: {} 2024-03-04T17:48:15.725 DEBUG:boto:Token: None 2024-03-04T17:48:15.725 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.726 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tq4vIBHHH+U6S1E7O/4yg/idlJM= 2024-03-04T17:48:15.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tq4vIBHHH+U6S1E7O/4yg/idlJM=', 'Content-Length': '0'} 2024-03-04T17:48:15.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e8900746ed62025-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.772 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.772 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.772 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:15.773 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:15.773 DEBUG:boto:Method: GET 2024-03-04T17:48:15.773 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:15.773 DEBUG:boto:Data: 2024-03-04T17:48:15.773 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.773 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.773 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.773 DEBUG:boto:Params: {} 2024-03-04T17:48:15.773 DEBUG:boto:Token: None 2024-03-04T17:48:15.774 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.774 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f167fc407e27b5e6-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.779 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.780 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.780 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.780 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:15.780 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:15.780 DEBUG:boto:Method: GET 2024-03-04T17:48:15.780 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:15.780 DEBUG:boto:Data: 2024-03-04T17:48:15.780 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.780 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.781 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.781 DEBUG:boto:Params: {} 2024-03-04T17:48:15.781 DEBUG:boto:Token: None 2024-03-04T17:48:15.781 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.781 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa2210b61c68b24f-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.828 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.829 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.829 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.829 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.829 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.829 DEBUG:boto:Data: 2024-03-04T17:48:15.829 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.830 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.830 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.830 DEBUG:boto:Params: {} 2024-03-04T17:48:15.830 DEBUG:boto:Token: None 2024-03-04T17:48:15.830 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.830 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lI3YcTWSgCJ3D3D+ROPR9cKkFu0= 2024-03-04T17:48:15.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lI3YcTWSgCJ3D3D+ROPR9cKkFu0=', 'Content-Length': '0'} 2024-03-04T17:48:15.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ed7ee91a981768e-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.834 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.834 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.835 DEBUG:boto:Method: HEAD 2024-03-04T17:48:15.835 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.835 DEBUG:boto:Data: 2024-03-04T17:48:15.835 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.835 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.835 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.835 DEBUG:boto:Params: {} 2024-03-04T17:48:15.835 DEBUG:boto:Token: None 2024-03-04T17:48:15.836 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.836 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lI3YcTWSgCJ3D3D+ROPR9cKkFu0= 2024-03-04T17:48:15.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lI3YcTWSgCJ3D3D+ROPR9cKkFu0=', 'Content-Length': '0'} 2024-03-04T17:48:15.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e32162183cdc5e0c-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:15.840 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.840 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.840 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.840 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.840 DEBUG:boto:Method: GET 2024-03-04T17:48:15.840 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.840 DEBUG:boto:Data: 2024-03-04T17:48:15.840 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.840 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.841 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.841 DEBUG:boto:Params: {} 2024-03-04T17:48:15.841 DEBUG:boto:Token: None 2024-03-04T17:48:15.841 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.841 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tq4vIBHHH+U6S1E7O/4yg/idlJM= 2024-03-04T17:48:15.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tq4vIBHHH+U6S1E7O/4yg/idlJM=', 'Content-Length': '0'} 2024-03-04T17:48:15.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd33e117fd31cc52-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.888 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.888 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:15.888 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.888 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.889 DEBUG:boto:Method: GET 2024-03-04T17:48:15.889 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.889 DEBUG:boto:Data: 2024-03-04T17:48:15.889 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.889 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.889 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.889 DEBUG:boto:Params: {} 2024-03-04T17:48:15.889 DEBUG:boto:Token: None 2024-03-04T17:48:15.889 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:48:15.890 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tq4vIBHHH+U6S1E7O/4yg/idlJM= 2024-03-04T17:48:15.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tq4vIBHHH+U6S1E7O/4yg/idlJM=', 'Content-Length': '0'} 2024-03-04T17:48:15.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8b434a59b0ab2fc-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.936 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.936 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.936 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:15.936 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:15.937 DEBUG:boto:Method: GET 2024-03-04T17:48:15.937 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:15.937 DEBUG:boto:Data: 2024-03-04T17:48:15.937 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.937 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.937 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.937 DEBUG:boto:Params: {} 2024-03-04T17:48:15.937 DEBUG:boto:Token: None 2024-03-04T17:48:15.937 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.938 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038454efdbb758c70-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.943 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.943 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:15.944 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:15.944 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:15.944 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:15.944 DEBUG:boto:Method: GET 2024-03-04T17:48:15.944 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:48:15.944 DEBUG:boto:Data: 2024-03-04T17:48:15.944 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.944 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:15.944 DEBUG:boto:Port: 8000 2024-03-04T17:48:15.944 DEBUG:boto:Params: {} 2024-03-04T17:48:15.945 DEBUG:boto:Token: None 2024-03-04T17:48:15.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/ 2024-03-04T17:48:15.945 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE= 2024-03-04T17:48:15.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e2WOHHbVI1SJFjXDS8hVH3t4yDE=', 'Content-Length': '0'} 2024-03-04T17:48:15.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bff32d35ec7085b0-0065e6095f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:15.950 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:15.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:15.951 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:15.951 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:15.951 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:15.951 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:15.951 DEBUG:boto:Method: GET 2024-03-04T17:48:15.951 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:15.951 DEBUG:boto:Data: 2024-03-04T17:48:15.952 DEBUG:boto:Headers: {} 2024-03-04T17:48:15.952 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:15.952 DEBUG:boto:Port: 8001 2024-03-04T17:48:15.952 DEBUG:boto:Params: {} 2024-03-04T17:48:15.952 DEBUG:boto:Token: None 2024-03-04T17:48:15.952 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:15 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:15.952 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Inzwy2PvIKQ/7aw/8aNzjX5GeGg= 2024-03-04T17:48:15.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Inzwy2PvIKQ/7aw/8aNzjX5GeGg=', 'Content-Length': '0'} 2024-03-04T17:48:15.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d5d1456be50d654-0065e6095f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.000 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.000 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.000 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.001 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.001 DEBUG:boto:Method: GET 2024-03-04T17:48:16.001 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.001 DEBUG:boto:Data: 2024-03-04T17:48:16.001 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.001 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.001 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.001 DEBUG:boto:Params: {} 2024-03-04T17:48:16.001 DEBUG:boto:Token: None 2024-03-04T17:48:16.002 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.002 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YWnrBQWqkrluMiR39J0AKws6Am4= 2024-03-04T17:48:16.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YWnrBQWqkrluMiR39J0AKws6Am4=', 'Content-Length': '0'} 2024-03-04T17:48:16.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000145cb03fd0fc0909-0065e60960-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.052 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.052 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.052 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.052 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.052 DEBUG:boto:Method: HEAD 2024-03-04T17:48:16.053 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.053 DEBUG:boto:Data: 2024-03-04T17:48:16.053 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.053 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:16.053 DEBUG:boto:Port: 8001 2024-03-04T17:48:16.053 DEBUG:boto:Params: {} 2024-03-04T17:48:16.053 DEBUG:boto:Token: None 2024-03-04T17:48:16.053 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.053 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pQkcZ7iYqf7Y8qM2Vym+csv3YBA= 2024-03-04T17:48:16.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pQkcZ7iYqf7Y8qM2Vym+csv3YBA=', 'Content-Length': '0'} 2024-03-04T17:48:16.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d68f3266cc32b358-0065e60960-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.057 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.057 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.057 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.057 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.057 DEBUG:boto:Method: HEAD 2024-03-04T17:48:16.057 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.057 DEBUG:boto:Data: 2024-03-04T17:48:16.057 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.057 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.058 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.058 DEBUG:boto:Params: {} 2024-03-04T17:48:16.058 DEBUG:boto:Token: None 2024-03-04T17:48:16.058 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.058 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pQkcZ7iYqf7Y8qM2Vym+csv3YBA= 2024-03-04T17:48:16.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pQkcZ7iYqf7Y8qM2Vym+csv3YBA=', 'Content-Length': '0'} 2024-03-04T17:48:16.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c189c52771d2ee1-0065e60960-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:16.061 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.061 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.061 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.062 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.062 DEBUG:boto:Method: GET 2024-03-04T17:48:16.062 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.062 DEBUG:boto:Data: 2024-03-04T17:48:16.062 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.062 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:16.062 DEBUG:boto:Port: 8001 2024-03-04T17:48:16.062 DEBUG:boto:Params: {} 2024-03-04T17:48:16.062 DEBUG:boto:Token: None 2024-03-04T17:48:16.062 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.063 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YWnrBQWqkrluMiR39J0AKws6Am4= 2024-03-04T17:48:16.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YWnrBQWqkrluMiR39J0AKws6Am4=', 'Content-Length': '0'} 2024-03-04T17:48:16.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d3ddb2d1b0a034c-0065e60960-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.108 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.108 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.108 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.108 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.109 DEBUG:boto:Method: GET 2024-03-04T17:48:16.109 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.109 DEBUG:boto:Data: 2024-03-04T17:48:16.109 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.109 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.109 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.109 DEBUG:boto:Params: {} 2024-03-04T17:48:16.109 DEBUG:boto:Token: None 2024-03-04T17:48:16.110 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.110 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YWnrBQWqkrluMiR39J0AKws6Am4= 2024-03-04T17:48:16.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YWnrBQWqkrluMiR39J0AKws6Am4=', 'Content-Length': '0'} 2024-03-04T17:48:16.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043eb6e9c9ba95b00-0065e60960-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.113 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:16.113 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:16.113 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:16.114 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:16.114 DEBUG:boto:Method: GET 2024-03-04T17:48:16.114 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:16.114 DEBUG:boto:Data: 2024-03-04T17:48:16.114 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.114 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:16.114 DEBUG:boto:Port: 8001 2024-03-04T17:48:16.114 DEBUG:boto:Params: {} 2024-03-04T17:48:16.114 DEBUG:boto:Token: None 2024-03-04T17:48:16.115 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/ 2024-03-04T17:48:16.115 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc= 2024-03-04T17:48:16.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc=', 'Content-Length': '0'} 2024-03-04T17:48:16.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000569daff23a9f530a-0065e60960-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.120 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:16.120 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:16.120 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:16.121 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:16.121 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:16.121 DEBUG:boto:Method: GET 2024-03-04T17:48:16.121 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:16.121 DEBUG:boto:Data: 2024-03-04T17:48:16.121 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.121 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.121 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.121 DEBUG:boto:Params: {} 2024-03-04T17:48:16.121 DEBUG:boto:Token: None 2024-03-04T17:48:16.121 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/ 2024-03-04T17:48:16.122 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc= 2024-03-04T17:48:16.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc=', 'Content-Length': '0'} 2024-03-04T17:48:16.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bdcac1719da33ef-0065e60960-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.127 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:16.127 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.127 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.127 DEBUG:boto:Method: HEAD 2024-03-04T17:48:16.127 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.127 DEBUG:boto:Data: 2024-03-04T17:48:16.127 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.127 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:16.128 DEBUG:boto:Port: 8001 2024-03-04T17:48:16.128 DEBUG:boto:Params: {} 2024-03-04T17:48:16.128 DEBUG:boto:Token: None 2024-03-04T17:48:16.128 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.128 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uFI7ZZLAVBut9CMQ464LVMTwf5U= 2024-03-04T17:48:16.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uFI7ZZLAVBut9CMQ464LVMTwf5U=', 'Content-Length': '0'} 2024-03-04T17:48:16.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ae73cbb3aecf4f3-0065e60960-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.132 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.132 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.132 DEBUG:boto:Method: HEAD 2024-03-04T17:48:16.132 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.132 DEBUG:boto:Data: 2024-03-04T17:48:16.132 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.133 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.133 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.133 DEBUG:boto:Params: {} 2024-03-04T17:48:16.133 DEBUG:boto:Token: None 2024-03-04T17:48:16.133 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.133 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uFI7ZZLAVBut9CMQ464LVMTwf5U= 2024-03-04T17:48:16.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uFI7ZZLAVBut9CMQ464LVMTwf5U=', 'Content-Length': '0'} 2024-03-04T17:48:16.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a2b3e21114d31ce-0065e60960-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:16.137 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.137 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.138 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.138 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.138 DEBUG:boto:Method: GET 2024-03-04T17:48:16.138 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.138 DEBUG:boto:Data: 2024-03-04T17:48:16.138 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.138 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:16.138 DEBUG:boto:Port: 8001 2024-03-04T17:48:16.138 DEBUG:boto:Params: {} 2024-03-04T17:48:16.138 DEBUG:boto:Token: None 2024-03-04T17:48:16.139 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.139 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YWnrBQWqkrluMiR39J0AKws6Am4= 2024-03-04T17:48:16.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YWnrBQWqkrluMiR39J0AKws6Am4=', 'Content-Length': '0'} 2024-03-04T17:48:16.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e799819bffb60aca-0065e60960-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.188 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.188 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:16.188 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.188 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.188 DEBUG:boto:Method: GET 2024-03-04T17:48:16.189 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.189 DEBUG:boto:Data: 2024-03-04T17:48:16.189 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.189 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.189 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.189 DEBUG:boto:Params: {} 2024-03-04T17:48:16.189 DEBUG:boto:Token: None 2024-03-04T17:48:16.189 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:48:16.190 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YWnrBQWqkrluMiR39J0AKws6Am4= 2024-03-04T17:48:16.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YWnrBQWqkrluMiR39J0AKws6Am4=', 'Content-Length': '0'} 2024-03-04T17:48:16.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089ad6cd58ad50efa-0065e60960-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.193 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:16.193 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:16.193 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:16.193 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:16.193 DEBUG:boto:Method: GET 2024-03-04T17:48:16.193 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:16.193 DEBUG:boto:Data: 2024-03-04T17:48:16.193 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.194 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:16.194 DEBUG:boto:Port: 8001 2024-03-04T17:48:16.194 DEBUG:boto:Params: {} 2024-03-04T17:48:16.194 DEBUG:boto:Token: None 2024-03-04T17:48:16.194 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/ 2024-03-04T17:48:16.194 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc= 2024-03-04T17:48:16.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc=', 'Content-Length': '0'} 2024-03-04T17:48:16.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047c9f40f18458991-0065e60960-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.199 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:16.200 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:16.200 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:16.200 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:16.200 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:16.200 DEBUG:boto:Method: GET 2024-03-04T17:48:16.200 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:48:16.201 DEBUG:boto:Data: 2024-03-04T17:48:16.201 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.201 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.201 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.201 DEBUG:boto:Params: {} 2024-03-04T17:48:16.201 DEBUG:boto:Token: None 2024-03-04T17:48:16.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/ 2024-03-04T17:48:16.201 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc= 2024-03-04T17:48:16.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc=', 'Content-Length': '0'} 2024-03-04T17:48:16.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000615fc031a5088eee-0065e60960-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.206 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:16.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:16.207 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.207 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.207 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.207 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.207 DEBUG:boto:Method: GET 2024-03-04T17:48:16.207 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.207 DEBUG:boto:Data: 2024-03-04T17:48:16.207 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.207 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:16.208 DEBUG:boto:Port: 8001 2024-03-04T17:48:16.208 DEBUG:boto:Params: {} 2024-03-04T17:48:16.208 DEBUG:boto:Token: None 2024-03-04T17:48:16.208 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.208 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uh31rqvc6H87On/UVrChP5LBOHA= 2024-03-04T17:48:16.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uh31rqvc6H87On/UVrChP5LBOHA=', 'Content-Length': '0'} 2024-03-04T17:48:16.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bfdc058b0c7e58f-0065e60960-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.211 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.212 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.212 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.212 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.212 DEBUG:boto:Method: GET 2024-03-04T17:48:16.212 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.212 DEBUG:boto:Data: 2024-03-04T17:48:16.212 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.212 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.212 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.212 DEBUG:boto:Params: {} 2024-03-04T17:48:16.213 DEBUG:boto:Token: None 2024-03-04T17:48:16.213 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.213 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uh31rqvc6H87On/UVrChP5LBOHA= 2024-03-04T17:48:16.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uh31rqvc6H87On/UVrChP5LBOHA=', 'Content-Length': '0'} 2024-03-04T17:48:16.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d926ada2100b860e-0065e60960-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.260 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.260 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.260 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.260 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.261 DEBUG:boto:Method: HEAD 2024-03-04T17:48:16.261 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.261 DEBUG:boto:Data: 2024-03-04T17:48:16.261 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.261 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:16.261 DEBUG:boto:Port: 8001 2024-03-04T17:48:16.261 DEBUG:boto:Params: {} 2024-03-04T17:48:16.261 DEBUG:boto:Token: None 2024-03-04T17:48:16.261 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.262 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Rz8m6zYgM9lmhQIuX8E5iEHGVZI= 2024-03-04T17:48:16.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Rz8m6zYgM9lmhQIuX8E5iEHGVZI=', 'Content-Length': '0'} 2024-03-04T17:48:16.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bdee21a1c224ffe-0065e60960-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.265 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.265 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.265 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.265 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.265 DEBUG:boto:Method: HEAD 2024-03-04T17:48:16.265 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.265 DEBUG:boto:Data: 2024-03-04T17:48:16.265 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.266 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.266 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.266 DEBUG:boto:Params: {} 2024-03-04T17:48:16.266 DEBUG:boto:Token: None 2024-03-04T17:48:16.266 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.266 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Rz8m6zYgM9lmhQIuX8E5iEHGVZI= 2024-03-04T17:48:16.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Rz8m6zYgM9lmhQIuX8E5iEHGVZI=', 'Content-Length': '0'} 2024-03-04T17:48:16.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab6233decaf096cc-0065e60960-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:16.269 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.269 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.269 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.270 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.270 DEBUG:boto:Method: GET 2024-03-04T17:48:16.270 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.270 DEBUG:boto:Data: 2024-03-04T17:48:16.270 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.270 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:16.270 DEBUG:boto:Port: 8001 2024-03-04T17:48:16.270 DEBUG:boto:Params: {} 2024-03-04T17:48:16.270 DEBUG:boto:Token: None 2024-03-04T17:48:16.271 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.271 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uh31rqvc6H87On/UVrChP5LBOHA= 2024-03-04T17:48:16.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uh31rqvc6H87On/UVrChP5LBOHA=', 'Content-Length': '0'} 2024-03-04T17:48:16.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa53cd561e4f18c2-0065e60960-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.316 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.316 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.316 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.316 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.316 DEBUG:boto:Method: GET 2024-03-04T17:48:16.317 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.317 DEBUG:boto:Data: 2024-03-04T17:48:16.317 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.317 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.317 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.317 DEBUG:boto:Params: {} 2024-03-04T17:48:16.317 DEBUG:boto:Token: None 2024-03-04T17:48:16.317 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.318 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uh31rqvc6H87On/UVrChP5LBOHA= 2024-03-04T17:48:16.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uh31rqvc6H87On/UVrChP5LBOHA=', 'Content-Length': '0'} 2024-03-04T17:48:16.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5ba9b6ce57c2a0b-0065e60960-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.364 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:16.364 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:16.364 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:16.365 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:16.365 DEBUG:boto:Method: GET 2024-03-04T17:48:16.365 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:16.365 DEBUG:boto:Data: 2024-03-04T17:48:16.365 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.365 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:16.365 DEBUG:boto:Port: 8001 2024-03-04T17:48:16.365 DEBUG:boto:Params: {} 2024-03-04T17:48:16.365 DEBUG:boto:Token: None 2024-03-04T17:48:16.366 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/ 2024-03-04T17:48:16.366 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc= 2024-03-04T17:48:16.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc=', 'Content-Length': '0'} 2024-03-04T17:48:16.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6e76a72ca60670b-0065e60960-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.372 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:16.372 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:16.372 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:16.372 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:16.372 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:16.373 DEBUG:boto:Method: GET 2024-03-04T17:48:16.373 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:16.373 DEBUG:boto:Data: 2024-03-04T17:48:16.373 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.373 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.373 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.373 DEBUG:boto:Params: {} 2024-03-04T17:48:16.373 DEBUG:boto:Token: None 2024-03-04T17:48:16.373 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/ 2024-03-04T17:48:16.374 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc= 2024-03-04T17:48:16.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc=', 'Content-Length': '0'} 2024-03-04T17:48:16.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afb95a9ea4f60ec0-0065e60960-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.379 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:16.379 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.379 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.379 DEBUG:boto:Method: HEAD 2024-03-04T17:48:16.379 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.380 DEBUG:boto:Data: 2024-03-04T17:48:16.380 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.380 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:16.380 DEBUG:boto:Port: 8001 2024-03-04T17:48:16.380 DEBUG:boto:Params: {} 2024-03-04T17:48:16.380 DEBUG:boto:Token: None 2024-03-04T17:48:16.380 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.380 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gEBsvbzom4mA3Y7IuKSz/TcU0x0= 2024-03-04T17:48:16.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gEBsvbzom4mA3Y7IuKSz/TcU0x0=', 'Content-Length': '0'} 2024-03-04T17:48:16.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcda784f62bb4afa-0065e60960-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.384 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.384 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.384 DEBUG:boto:Method: HEAD 2024-03-04T17:48:16.385 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.385 DEBUG:boto:Data: 2024-03-04T17:48:16.385 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.385 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.385 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.385 DEBUG:boto:Params: {} 2024-03-04T17:48:16.385 DEBUG:boto:Token: None 2024-03-04T17:48:16.385 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.385 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gEBsvbzom4mA3Y7IuKSz/TcU0x0= 2024-03-04T17:48:16.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gEBsvbzom4mA3Y7IuKSz/TcU0x0=', 'Content-Length': '0'} 2024-03-04T17:48:16.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f91bed254c89b08-0065e60960-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:16.389 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.389 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.390 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.390 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.390 DEBUG:boto:Method: GET 2024-03-04T17:48:16.390 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.390 DEBUG:boto:Data: 2024-03-04T17:48:16.390 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.390 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:16.390 DEBUG:boto:Port: 8001 2024-03-04T17:48:16.390 DEBUG:boto:Params: {} 2024-03-04T17:48:16.390 DEBUG:boto:Token: None 2024-03-04T17:48:16.391 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.391 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uh31rqvc6H87On/UVrChP5LBOHA= 2024-03-04T17:48:16.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uh31rqvc6H87On/UVrChP5LBOHA=', 'Content-Length': '0'} 2024-03-04T17:48:16.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a65794e2e68a954-0065e60960-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.394 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.394 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:16.394 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.394 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.395 DEBUG:boto:Method: GET 2024-03-04T17:48:16.395 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.395 DEBUG:boto:Data: 2024-03-04T17:48:16.395 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.395 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.395 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.395 DEBUG:boto:Params: {} 2024-03-04T17:48:16.395 DEBUG:boto:Token: None 2024-03-04T17:48:16.395 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:48:16.396 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uh31rqvc6H87On/UVrChP5LBOHA= 2024-03-04T17:48:16.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uh31rqvc6H87On/UVrChP5LBOHA=', 'Content-Length': '0'} 2024-03-04T17:48:16.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004751472f34ecf055-0065e60960-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.440 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:16.440 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:16.440 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:16.441 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:16.441 DEBUG:boto:Method: GET 2024-03-04T17:48:16.441 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:16.441 DEBUG:boto:Data: 2024-03-04T17:48:16.441 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.441 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:16.441 DEBUG:boto:Port: 8001 2024-03-04T17:48:16.441 DEBUG:boto:Params: {} 2024-03-04T17:48:16.441 DEBUG:boto:Token: None 2024-03-04T17:48:16.442 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/ 2024-03-04T17:48:16.442 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc= 2024-03-04T17:48:16.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc=', 'Content-Length': '0'} 2024-03-04T17:48:16.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0608630c0944d36-0065e60960-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.447 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:16.447 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:16.448 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:16.448 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:16.448 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:16.448 DEBUG:boto:Method: GET 2024-03-04T17:48:16.448 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:48:16.448 DEBUG:boto:Data: 2024-03-04T17:48:16.448 DEBUG:boto:Headers: {} 2024-03-04T17:48:16.448 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:16.448 DEBUG:boto:Port: 8000 2024-03-04T17:48:16.448 DEBUG:boto:Params: {} 2024-03-04T17:48:16.449 DEBUG:boto:Token: None 2024-03-04T17:48:16.449 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:16 GMT /ppfahn-19/ 2024-03-04T17:48:16.449 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc= 2024-03-04T17:48:16.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:16 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kisgHpksona+BT9kE3q3bLBAwtc=', 'Content-Length': '0'} 2024-03-04T17:48:16.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a7ae0fd08a7e389-0065e60960-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:16.454 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:16.455 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-19 zones={test-zone3, test-zone2} 2024-03-04T17:48:16.455 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-20 2024-03-04T17:48:16.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:16.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:16.455 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:16.505 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:16.506 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:16.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:16.572+0000 7f0b9f562980 1 bucket sync caught up with source: 2024-03-04T17:48:16.571 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000010.66.6, , 00000000003.11.10, , 00000000008.76.10, , , , , ] 2024-03-04T17:48:16.571 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000010.66.6, , 00000000003.11.10, , 00000000008.76.10, , , , , ] 2024-03-04T17:48:16.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:16.572+0000 7f0b9f562980 0 bucket checkpoint complete 2024-03-04T17:48:16.594 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-20 2024-03-04T17:48:16.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:16.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:16.595 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:16.645 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:16.645 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:16.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:16.705+0000 7f8629ec2980 1 bucket sync caught up with source: 2024-03-04T17:48:16.704 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000010.66.6, , 00000000003.11.10, , 00000000008.76.10, , , , , ] 2024-03-04T17:48:16.704 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000010.66.6, , 00000000003.11.10, , 00000000008.76.10, , , , , ] 2024-03-04T17:48:16.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:16.705+0000 7f8629ec2980 0 bucket checkpoint complete 2024-03-04T17:48:16.725 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-20 2024-03-04T17:48:16.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:16.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:16.726 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:16.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:16.829+0000 7fc6372aa980 1 bucket sync caught up with source: 2024-03-04T17:48:16.827 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.58.10, , 00000000010.34.6, , 00000000008.64.10, , , , , ] 2024-03-04T17:48:16.827 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.58.10, , 00000000010.34.6, , 00000000008.64.10, , , , , ] 2024-03-04T17:48:16.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:16.829+0000 7fc6372aa980 0 bucket checkpoint complete 2024-03-04T17:48:16.848 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-20 2024-03-04T17:48:16.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:16.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:16.849 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:16.931 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:16.931 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:16.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:16.993+0000 7f2fd28f0980 1 bucket sync caught up with source: 2024-03-04T17:48:16.991 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.58.10, , 00000000010.34.6, , 00000000008.64.10, , , , , ] 2024-03-04T17:48:16.991 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.58.10, , 00000000010.34.6, , 00000000008.64.10, , , , , ] 2024-03-04T17:48:16.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:16.993+0000 7f2fd28f0980 0 bucket checkpoint complete 2024-03-04T17:48:17.013 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-20 2024-03-04T17:48:17.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:17.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:17.014 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:17.100 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:17.100 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:17.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:17.161+0000 7f06d547b980 1 bucket sync caught up with source: 2024-03-04T17:48:17.157 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.124.10, , 00000000003.62.10, , 00000000013.118.6, , , , , ] 2024-03-04T17:48:17.157 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.124.10, , 00000000003.62.10, , 00000000013.118.6, , , , , ] 2024-03-04T17:48:17.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:17.161+0000 7f06d547b980 0 bucket checkpoint complete 2024-03-04T17:48:17.179 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-20 2024-03-04T17:48:17.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:17.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:17.179 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:17.229 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:17.230 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:17.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:17.296+0000 7fa6c7864980 1 bucket sync caught up with source: 2024-03-04T17:48:17.295 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000003.124.10, , 00000000003.62.10, , 00000000013.118.6, , , , , ] 2024-03-04T17:48:17.295 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000003.124.10, , 00000000003.62.10, , 00000000013.118.6, , , , , ] 2024-03-04T17:48:17.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:17.296+0000 7fa6c7864980 0 bucket checkpoint complete 2024-03-04T17:48:17.317 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-20 zones={test-zone2, test-zone1} 2024-03-04T17:48:17.317 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.317 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.317 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.318 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:48:17.318 DEBUG:boto:Data: 2024-03-04T17:48:17.318 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.318 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.318 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.318 DEBUG:boto:Params: {} 2024-03-04T17:48:17.318 DEBUG:boto:Token: None 2024-03-04T17:48:17.318 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/ 2024-03-04T17:48:17.319 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SEU4DiA2vump6zSNVbtXT8R0HUw= 2024-03-04T17:48:17.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SEU4DiA2vump6zSNVbtXT8R0HUw=', 'Content-Length': '0'} 2024-03-04T17:48:17.321 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049b40c46f05533b8-0065e60961-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.321 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.321 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.321 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.322 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:48:17.322 DEBUG:boto:Data: 2024-03-04T17:48:17.322 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.322 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.322 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.322 DEBUG:boto:Params: {} 2024-03-04T17:48:17.322 DEBUG:boto:Token: None 2024-03-04T17:48:17.322 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/ 2024-03-04T17:48:17.322 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SEU4DiA2vump6zSNVbtXT8R0HUw= 2024-03-04T17:48:17.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SEU4DiA2vump6zSNVbtXT8R0HUw=', 'Content-Length': '0'} 2024-03-04T17:48:17.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4f273c17f54d4f0-0065e60961-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.326 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:48:17.326 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.327 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.327 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:17.327 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:17.327 DEBUG:boto:Method: GET 2024-03-04T17:48:17.327 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:17.327 DEBUG:boto:Data: 2024-03-04T17:48:17.327 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.327 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.327 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.327 DEBUG:boto:Params: {} 2024-03-04T17:48:17.328 DEBUG:boto:Token: None 2024-03-04T17:48:17.328 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/?versions 2024-03-04T17:48:17.328 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tDj44mu63ePQVj+wZ6wpGN7r9mg= 2024-03-04T17:48:17.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tDj44mu63ePQVj+wZ6wpGN7r9mg=', 'Content-Length': '0'} 2024-03-04T17:48:17.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000180f2170b27cd7fe-0065e60961-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.330 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.331 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:17.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:17.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:17.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:17.331 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:48:17.331 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.331 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.331 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:17.331 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:17.332 DEBUG:boto:Method: GET 2024-03-04T17:48:17.332 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:17.332 DEBUG:boto:Data: 2024-03-04T17:48:17.332 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.332 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.332 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.332 DEBUG:boto:Params: {} 2024-03-04T17:48:17.332 DEBUG:boto:Token: None 2024-03-04T17:48:17.332 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/?versions 2024-03-04T17:48:17.333 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tDj44mu63ePQVj+wZ6wpGN7r9mg= 2024-03-04T17:48:17.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tDj44mu63ePQVj+wZ6wpGN7r9mg=', 'Content-Length': '0'} 2024-03-04T17:48:17.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006e369d65ce56292-0065e60961-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.337 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.338 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:17.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:17.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:17.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:17.338 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.338 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.338 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:17.338 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:17.339 DEBUG:boto:Method: GET 2024-03-04T17:48:17.339 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:17.339 DEBUG:boto:Data: 2024-03-04T17:48:17.339 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.339 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.339 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.339 DEBUG:boto:Params: {} 2024-03-04T17:48:17.339 DEBUG:boto:Token: None 2024-03-04T17:48:17.339 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/?versions 2024-03-04T17:48:17.339 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tDj44mu63ePQVj+wZ6wpGN7r9mg= 2024-03-04T17:48:17.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tDj44mu63ePQVj+wZ6wpGN7r9mg=', 'Content-Length': '0'} 2024-03-04T17:48:17.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2592c2e8f664b40-0065e60961-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.342 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.342 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.343 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.343 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:17.343 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:17.343 DEBUG:boto:Method: GET 2024-03-04T17:48:17.343 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:17.343 DEBUG:boto:Data: 2024-03-04T17:48:17.343 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.343 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.343 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.343 DEBUG:boto:Params: {} 2024-03-04T17:48:17.343 DEBUG:boto:Token: None 2024-03-04T17:48:17.344 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/?versions 2024-03-04T17:48:17.344 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tDj44mu63ePQVj+wZ6wpGN7r9mg= 2024-03-04T17:48:17.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tDj44mu63ePQVj+wZ6wpGN7r9mg=', 'Content-Length': '0'} 2024-03-04T17:48:17.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c86022819d759bf4-0065e60961-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.349 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:17.350 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:17.350 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:17.350 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.350 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.350 DEBUG:boto:Method: GET 2024-03-04T17:48:17.350 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.350 DEBUG:boto:Data: 2024-03-04T17:48:17.351 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.351 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.351 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.351 DEBUG:boto:Params: {} 2024-03-04T17:48:17.351 DEBUG:boto:Token: None 2024-03-04T17:48:17.351 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.351 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qdpe52QNlunT7bkh01NhmI6bX+E= 2024-03-04T17:48:17.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qdpe52QNlunT7bkh01NhmI6bX+E=', 'Content-Length': '0'} 2024-03-04T17:48:17.353 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080f8b5e50cfb1307-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.353 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:17.353 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:17.353 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.354 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.354 DEBUG:boto:Method: GET 2024-03-04T17:48:17.354 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.354 DEBUG:boto:Data: 2024-03-04T17:48:17.354 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.354 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.354 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.354 DEBUG:boto:Params: {} 2024-03-04T17:48:17.354 DEBUG:boto:Token: None 2024-03-04T17:48:17.354 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.355 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qdpe52QNlunT7bkh01NhmI6bX+E= 2024-03-04T17:48:17.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qdpe52QNlunT7bkh01NhmI6bX+E=', 'Content-Length': '0'} 2024-03-04T17:48:17.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c6e5fdd7af64034c-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.358 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:17.358 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:17.358 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.358 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.358 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.358 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.358 DEBUG:boto:Data: 2024-03-04T17:48:17.358 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.358 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.359 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.359 DEBUG:boto:Params: {} 2024-03-04T17:48:17.359 DEBUG:boto:Token: None 2024-03-04T17:48:17.359 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.359 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZSncIF4xEo+mg4Z7NejUNrwcPQk= 2024-03-04T17:48:17.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZSncIF4xEo+mg4Z7NejUNrwcPQk=', 'Content-Length': '0'} 2024-03-04T17:48:17.361 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a48d0069b7fad0f8-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.361 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:17.361 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:17.362 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.362 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.362 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.362 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.362 DEBUG:boto:Data: 2024-03-04T17:48:17.362 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.362 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.362 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.362 DEBUG:boto:Params: {} 2024-03-04T17:48:17.362 DEBUG:boto:Token: None 2024-03-04T17:48:17.363 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.363 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZSncIF4xEo+mg4Z7NejUNrwcPQk= 2024-03-04T17:48:17.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZSncIF4xEo+mg4Z7NejUNrwcPQk=', 'Content-Length': '0'} 2024-03-04T17:48:17.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000569cb093b7a82638-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:17.366 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:17.366 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:17.366 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.366 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.366 DEBUG:boto:Method: GET 2024-03-04T17:48:17.366 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.366 DEBUG:boto:Data: 2024-03-04T17:48:17.366 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.367 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.367 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.367 DEBUG:boto:Params: {} 2024-03-04T17:48:17.367 DEBUG:boto:Token: None 2024-03-04T17:48:17.367 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.367 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qdpe52QNlunT7bkh01NhmI6bX+E= 2024-03-04T17:48:17.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qdpe52QNlunT7bkh01NhmI6bX+E=', 'Content-Length': '0'} 2024-03-04T17:48:17.369 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a71870664e976811-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.370 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:17.370 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:17.370 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.370 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.370 DEBUG:boto:Method: GET 2024-03-04T17:48:17.370 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.370 DEBUG:boto:Data: 2024-03-04T17:48:17.370 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.370 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.371 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.371 DEBUG:boto:Params: {} 2024-03-04T17:48:17.371 DEBUG:boto:Token: None 2024-03-04T17:48:17.371 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.371 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qdpe52QNlunT7bkh01NhmI6bX+E= 2024-03-04T17:48:17.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qdpe52QNlunT7bkh01NhmI6bX+E=', 'Content-Length': '0'} 2024-03-04T17:48:17.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d3ef5ccabfb01d8f-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.374 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.374 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.374 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:17.374 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:17.375 DEBUG:boto:Method: GET 2024-03-04T17:48:17.375 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:48:17.375 DEBUG:boto:Data: 2024-03-04T17:48:17.375 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.375 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.375 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.375 DEBUG:boto:Params: {} 2024-03-04T17:48:17.375 DEBUG:boto:Token: None 2024-03-04T17:48:17.375 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/ 2024-03-04T17:48:17.375 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw= 2024-03-04T17:48:17.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw=', 'Content-Length': '0'} 2024-03-04T17:48:17.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dcec827ee32b3c9-0065e60961-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.420 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.421 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.421 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.421 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:17.421 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:17.422 DEBUG:boto:Method: GET 2024-03-04T17:48:17.422 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:48:17.422 DEBUG:boto:Data: 2024-03-04T17:48:17.422 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.422 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.422 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.422 DEBUG:boto:Params: {} 2024-03-04T17:48:17.423 DEBUG:boto:Token: None 2024-03-04T17:48:17.423 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/ 2024-03-04T17:48:17.423 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw= 2024-03-04T17:48:17.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw=', 'Content-Length': '0'} 2024-03-04T17:48:17.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064eb4d62149f3416-0065e60961-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.428 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.428 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:17.428 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:17.428 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.428 DEBUG:boto:Path: /ppfahn-20/dummy 2024-03-04T17:48:17.428 DEBUG:boto:Data: 2024-03-04T17:48:17.429 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.429 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.429 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.429 DEBUG:boto:Params: {} 2024-03-04T17:48:17.429 DEBUG:boto:Token: None 2024-03-04T17:48:17.429 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/dummy 2024-03-04T17:48:17.429 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RACvnEkDxZsC/IZ88CYmf7oLTHs= 2024-03-04T17:48:17.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RACvnEkDxZsC/IZ88CYmf7oLTHs=', 'Content-Length': '0'} 2024-03-04T17:48:17.433 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009adcf137534f8aef-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.433 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:17.433 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:17.433 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.434 DEBUG:boto:Path: /ppfahn-20/dummy 2024-03-04T17:48:17.434 DEBUG:boto:Data: 2024-03-04T17:48:17.434 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.434 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.434 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.434 DEBUG:boto:Params: {} 2024-03-04T17:48:17.434 DEBUG:boto:Token: None 2024-03-04T17:48:17.434 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/dummy 2024-03-04T17:48:17.434 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RACvnEkDxZsC/IZ88CYmf7oLTHs= 2024-03-04T17:48:17.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RACvnEkDxZsC/IZ88CYmf7oLTHs=', 'Content-Length': '0'} 2024-03-04T17:48:17.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cf02aec53638b8c5-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:17.438 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:17.439 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:17.439 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.439 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.439 DEBUG:boto:Method: GET 2024-03-04T17:48:17.439 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.439 DEBUG:boto:Data: 2024-03-04T17:48:17.439 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.439 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.439 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.439 DEBUG:boto:Params: {} 2024-03-04T17:48:17.439 DEBUG:boto:Token: None 2024-03-04T17:48:17.440 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.440 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qdpe52QNlunT7bkh01NhmI6bX+E= 2024-03-04T17:48:17.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qdpe52QNlunT7bkh01NhmI6bX+E=', 'Content-Length': '0'} 2024-03-04T17:48:17.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002953f569e8dadc54-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.443 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:17.443 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:17.443 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.443 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.443 DEBUG:boto:Method: GET 2024-03-04T17:48:17.443 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.443 DEBUG:boto:Data: 2024-03-04T17:48:17.443 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.443 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.444 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.444 DEBUG:boto:Params: {} 2024-03-04T17:48:17.444 DEBUG:boto:Token: None 2024-03-04T17:48:17.444 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:17.444 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qdpe52QNlunT7bkh01NhmI6bX+E= 2024-03-04T17:48:17.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qdpe52QNlunT7bkh01NhmI6bX+E=', 'Content-Length': '0'} 2024-03-04T17:48:17.447 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006bec6501aa06ade0-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.448 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.448 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.448 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:17.448 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:17.448 DEBUG:boto:Method: GET 2024-03-04T17:48:17.448 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:48:17.448 DEBUG:boto:Data: 2024-03-04T17:48:17.448 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.448 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.448 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.448 DEBUG:boto:Params: {} 2024-03-04T17:48:17.449 DEBUG:boto:Token: None 2024-03-04T17:48:17.449 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/ 2024-03-04T17:48:17.449 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw= 2024-03-04T17:48:17.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw=', 'Content-Length': '0'} 2024-03-04T17:48:17.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e952396805289186-0065e60961-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.453 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.453 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.453 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.454 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:17.454 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:17.454 DEBUG:boto:Method: GET 2024-03-04T17:48:17.454 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:48:17.454 DEBUG:boto:Data: 2024-03-04T17:48:17.454 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.454 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.454 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.454 DEBUG:boto:Params: {} 2024-03-04T17:48:17.454 DEBUG:boto:Token: None 2024-03-04T17:48:17.454 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/ 2024-03-04T17:48:17.455 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw= 2024-03-04T17:48:17.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw=', 'Content-Length': '0'} 2024-03-04T17:48:17.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dd52ba734f5b52b-0065e60961-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.459 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:17.460 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.460 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.460 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.460 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.460 DEBUG:boto:Method: GET 2024-03-04T17:48:17.460 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.461 DEBUG:boto:Data: 2024-03-04T17:48:17.461 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.461 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.461 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.461 DEBUG:boto:Params: {} 2024-03-04T17:48:17.461 DEBUG:boto:Token: None 2024-03-04T17:48:17.461 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.461 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LaPmu1AK4lyzT6XJB9aIXmZ002E= 2024-03-04T17:48:17.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LaPmu1AK4lyzT6XJB9aIXmZ002E=', 'Content-Length': '0'} 2024-03-04T17:48:17.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a640d319bac5c997-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.508 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.508 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.509 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.509 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.509 DEBUG:boto:Method: GET 2024-03-04T17:48:17.509 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.509 DEBUG:boto:Data: 2024-03-04T17:48:17.509 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.509 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.509 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.509 DEBUG:boto:Params: {} 2024-03-04T17:48:17.509 DEBUG:boto:Token: None 2024-03-04T17:48:17.510 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.510 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LaPmu1AK4lyzT6XJB9aIXmZ002E= 2024-03-04T17:48:17.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LaPmu1AK4lyzT6XJB9aIXmZ002E=', 'Content-Length': '0'} 2024-03-04T17:48:17.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019b6b0676b145071-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.513 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.513 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.514 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.514 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.514 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.514 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.514 DEBUG:boto:Data: 2024-03-04T17:48:17.514 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.514 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.514 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.514 DEBUG:boto:Params: {} 2024-03-04T17:48:17.514 DEBUG:boto:Token: None 2024-03-04T17:48:17.515 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.515 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iZt96bMm/SAHNxd8xNJ6w3grcPk= 2024-03-04T17:48:17.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iZt96bMm/SAHNxd8xNJ6w3grcPk=', 'Content-Length': '0'} 2024-03-04T17:48:17.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014a4c470c826f27d-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.518 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.518 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.518 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.518 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.518 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.519 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.519 DEBUG:boto:Data: 2024-03-04T17:48:17.519 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.519 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.519 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.519 DEBUG:boto:Params: {} 2024-03-04T17:48:17.519 DEBUG:boto:Token: None 2024-03-04T17:48:17.519 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.519 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iZt96bMm/SAHNxd8xNJ6w3grcPk= 2024-03-04T17:48:17.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iZt96bMm/SAHNxd8xNJ6w3grcPk=', 'Content-Length': '0'} 2024-03-04T17:48:17.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c029e6f0fd6501e-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:17.523 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.523 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.523 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.523 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.523 DEBUG:boto:Method: GET 2024-03-04T17:48:17.523 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.523 DEBUG:boto:Data: 2024-03-04T17:48:17.524 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.524 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.524 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.524 DEBUG:boto:Params: {} 2024-03-04T17:48:17.524 DEBUG:boto:Token: None 2024-03-04T17:48:17.524 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.524 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LaPmu1AK4lyzT6XJB9aIXmZ002E= 2024-03-04T17:48:17.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LaPmu1AK4lyzT6XJB9aIXmZ002E=', 'Content-Length': '0'} 2024-03-04T17:48:17.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005db98ed30bbaea5f-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.568 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.568 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.569 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.569 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.569 DEBUG:boto:Method: GET 2024-03-04T17:48:17.569 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.569 DEBUG:boto:Data: 2024-03-04T17:48:17.569 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.569 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.569 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.569 DEBUG:boto:Params: {} 2024-03-04T17:48:17.570 DEBUG:boto:Token: None 2024-03-04T17:48:17.570 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.570 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LaPmu1AK4lyzT6XJB9aIXmZ002E= 2024-03-04T17:48:17.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LaPmu1AK4lyzT6XJB9aIXmZ002E=', 'Content-Length': '0'} 2024-03-04T17:48:17.574 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002dee80d552df30a0-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.574 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.574 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.574 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:17.575 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:17.575 DEBUG:boto:Method: GET 2024-03-04T17:48:17.575 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:17.575 DEBUG:boto:Data: 2024-03-04T17:48:17.575 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.575 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.575 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.575 DEBUG:boto:Params: {} 2024-03-04T17:48:17.575 DEBUG:boto:Token: None 2024-03-04T17:48:17.576 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/ 2024-03-04T17:48:17.576 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw= 2024-03-04T17:48:17.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw=', 'Content-Length': '0'} 2024-03-04T17:48:17.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017c1ff9023db2b4a-0065e60961-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.624 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.625 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.625 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.625 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:17.625 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:17.625 DEBUG:boto:Method: GET 2024-03-04T17:48:17.625 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:17.625 DEBUG:boto:Data: 2024-03-04T17:48:17.625 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.626 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.626 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.626 DEBUG:boto:Params: {} 2024-03-04T17:48:17.626 DEBUG:boto:Token: None 2024-03-04T17:48:17.626 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/ 2024-03-04T17:48:17.626 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw= 2024-03-04T17:48:17.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw=', 'Content-Length': '0'} 2024-03-04T17:48:17.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7dfcd34ecb6de3d-0065e60961-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.631 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.631 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.631 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.631 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.631 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.632 DEBUG:boto:Data: 2024-03-04T17:48:17.632 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.632 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.632 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.632 DEBUG:boto:Params: {} 2024-03-04T17:48:17.632 DEBUG:boto:Token: None 2024-03-04T17:48:17.632 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.632 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SasvcJQxJnCUuBUwRke5lLgmUjw= 2024-03-04T17:48:17.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SasvcJQxJnCUuBUwRke5lLgmUjw=', 'Content-Length': '0'} 2024-03-04T17:48:17.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000256d4d56402c068b-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.637 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.637 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.637 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.637 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.637 DEBUG:boto:Data: 2024-03-04T17:48:17.637 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.637 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.637 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.637 DEBUG:boto:Params: {} 2024-03-04T17:48:17.638 DEBUG:boto:Token: None 2024-03-04T17:48:17.638 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.638 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SasvcJQxJnCUuBUwRke5lLgmUjw= 2024-03-04T17:48:17.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SasvcJQxJnCUuBUwRke5lLgmUjw=', 'Content-Length': '0'} 2024-03-04T17:48:17.642 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081995f1df203e780-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:17.643 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.643 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.643 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.643 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.643 DEBUG:boto:Method: GET 2024-03-04T17:48:17.643 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.643 DEBUG:boto:Data: 2024-03-04T17:48:17.643 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.644 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.644 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.644 DEBUG:boto:Params: {} 2024-03-04T17:48:17.644 DEBUG:boto:Token: None 2024-03-04T17:48:17.644 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.644 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LaPmu1AK4lyzT6XJB9aIXmZ002E= 2024-03-04T17:48:17.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LaPmu1AK4lyzT6XJB9aIXmZ002E=', 'Content-Length': '0'} 2024-03-04T17:48:17.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007228aa934f0c867a-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.692 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.693 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:17.693 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.693 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.693 DEBUG:boto:Method: GET 2024-03-04T17:48:17.693 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.693 DEBUG:boto:Data: 2024-03-04T17:48:17.693 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.693 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.694 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.694 DEBUG:boto:Params: {} 2024-03-04T17:48:17.694 DEBUG:boto:Token: None 2024-03-04T17:48:17.694 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:17.694 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LaPmu1AK4lyzT6XJB9aIXmZ002E= 2024-03-04T17:48:17.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LaPmu1AK4lyzT6XJB9aIXmZ002E=', 'Content-Length': '0'} 2024-03-04T17:48:17.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5b80e50368a1c5b-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.697 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.698 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.698 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:17.698 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:17.698 DEBUG:boto:Method: GET 2024-03-04T17:48:17.698 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:17.698 DEBUG:boto:Data: 2024-03-04T17:48:17.698 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.698 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.698 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.698 DEBUG:boto:Params: {} 2024-03-04T17:48:17.699 DEBUG:boto:Token: None 2024-03-04T17:48:17.699 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/ 2024-03-04T17:48:17.699 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw= 2024-03-04T17:48:17.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw=', 'Content-Length': '0'} 2024-03-04T17:48:17.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058fbe6ce1fedc510-0065e60961-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.744 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.744 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.745 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.745 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:17.745 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:17.745 DEBUG:boto:Method: GET 2024-03-04T17:48:17.745 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:17.745 DEBUG:boto:Data: 2024-03-04T17:48:17.745 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.745 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.745 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.745 DEBUG:boto:Params: {} 2024-03-04T17:48:17.746 DEBUG:boto:Token: None 2024-03-04T17:48:17.746 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/ 2024-03-04T17:48:17.746 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw= 2024-03-04T17:48:17.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw=', 'Content-Length': '0'} 2024-03-04T17:48:17.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000055ff606a8a2b88a-0065e60961-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.751 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:17.751 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.751 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.751 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.751 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.752 DEBUG:boto:Method: GET 2024-03-04T17:48:17.752 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.752 DEBUG:boto:Data: 2024-03-04T17:48:17.752 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.752 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.752 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.752 DEBUG:boto:Params: {} 2024-03-04T17:48:17.752 DEBUG:boto:Token: None 2024-03-04T17:48:17.752 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.753 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DlnX/I0YDZsACpG/gLtOrmrndwk= 2024-03-04T17:48:17.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DlnX/I0YDZsACpG/gLtOrmrndwk=', 'Content-Length': '0'} 2024-03-04T17:48:17.755 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db4d2a0ce0d64031-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.756 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.756 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.756 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.756 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.756 DEBUG:boto:Method: GET 2024-03-04T17:48:17.757 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.757 DEBUG:boto:Data: 2024-03-04T17:48:17.757 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.757 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.757 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.757 DEBUG:boto:Params: {} 2024-03-04T17:48:17.757 DEBUG:boto:Token: None 2024-03-04T17:48:17.757 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.757 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DlnX/I0YDZsACpG/gLtOrmrndwk= 2024-03-04T17:48:17.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DlnX/I0YDZsACpG/gLtOrmrndwk=', 'Content-Length': '0'} 2024-03-04T17:48:17.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000d26a2469235ce5-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.804 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.804 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.804 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.804 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.805 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.805 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.805 DEBUG:boto:Data: 2024-03-04T17:48:17.805 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.805 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.805 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.805 DEBUG:boto:Params: {} 2024-03-04T17:48:17.806 DEBUG:boto:Token: None 2024-03-04T17:48:17.806 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.806 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:33CyJCOFbcUfr1eWQGpuwL+lvuY= 2024-03-04T17:48:17.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:33CyJCOFbcUfr1eWQGpuwL+lvuY=', 'Content-Length': '0'} 2024-03-04T17:48:17.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073601bdf080cea0c-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.809 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.809 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.809 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.810 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.810 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.810 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.810 DEBUG:boto:Data: 2024-03-04T17:48:17.810 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.810 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.810 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.811 DEBUG:boto:Params: {} 2024-03-04T17:48:17.811 DEBUG:boto:Token: None 2024-03-04T17:48:17.811 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.811 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:33CyJCOFbcUfr1eWQGpuwL+lvuY= 2024-03-04T17:48:17.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:33CyJCOFbcUfr1eWQGpuwL+lvuY=', 'Content-Length': '0'} 2024-03-04T17:48:17.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009dec074b6df82bda-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:17.814 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.814 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.815 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.815 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.815 DEBUG:boto:Method: GET 2024-03-04T17:48:17.815 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.815 DEBUG:boto:Data: 2024-03-04T17:48:17.815 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.815 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.815 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.815 DEBUG:boto:Params: {} 2024-03-04T17:48:17.816 DEBUG:boto:Token: None 2024-03-04T17:48:17.816 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.816 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DlnX/I0YDZsACpG/gLtOrmrndwk= 2024-03-04T17:48:17.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DlnX/I0YDZsACpG/gLtOrmrndwk=', 'Content-Length': '0'} 2024-03-04T17:48:17.819 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f19b98c2763f042-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.823 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.823 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.823 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.823 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.823 DEBUG:boto:Method: GET 2024-03-04T17:48:17.823 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.824 DEBUG:boto:Data: 2024-03-04T17:48:17.824 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.824 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.824 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.824 DEBUG:boto:Params: {} 2024-03-04T17:48:17.824 DEBUG:boto:Token: None 2024-03-04T17:48:17.824 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.824 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DlnX/I0YDZsACpG/gLtOrmrndwk= 2024-03-04T17:48:17.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DlnX/I0YDZsACpG/gLtOrmrndwk=', 'Content-Length': '0'} 2024-03-04T17:48:17.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007429cf8079467c0a-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.868 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.868 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.868 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:17.869 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:17.869 DEBUG:boto:Method: GET 2024-03-04T17:48:17.869 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:17.869 DEBUG:boto:Data: 2024-03-04T17:48:17.869 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.869 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.869 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.869 DEBUG:boto:Params: {} 2024-03-04T17:48:17.869 DEBUG:boto:Token: None 2024-03-04T17:48:17.870 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/ 2024-03-04T17:48:17.870 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw= 2024-03-04T17:48:17.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw=', 'Content-Length': '0'} 2024-03-04T17:48:17.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079fabb5b905bc04f-0065e60961-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.874 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.875 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:17.875 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:17.875 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:17.875 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:17.876 DEBUG:boto:Method: GET 2024-03-04T17:48:17.876 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:17.876 DEBUG:boto:Data: 2024-03-04T17:48:17.876 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.877 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.877 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.877 DEBUG:boto:Params: {} 2024-03-04T17:48:17.879 DEBUG:boto:Token: None 2024-03-04T17:48:17.880 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/ 2024-03-04T17:48:17.880 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw= 2024-03-04T17:48:17.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0PXEMx3nLd9oW4DzCFsMdyBD/Fw=', 'Content-Length': '0'} 2024-03-04T17:48:17.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095a35a4c586fa4a5-0065e60961-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.891 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:17.891 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.892 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.892 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.892 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.892 DEBUG:boto:Data: 2024-03-04T17:48:17.892 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.892 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.892 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.892 DEBUG:boto:Params: {} 2024-03-04T17:48:17.892 DEBUG:boto:Token: None 2024-03-04T17:48:17.893 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.893 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Erl8YLMvqpi7GrJa6ywHXdJDn5w= 2024-03-04T17:48:17.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Erl8YLMvqpi7GrJa6ywHXdJDn5w=', 'Content-Length': '0'} 2024-03-04T17:48:17.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c54db74ccd467c7b-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.898 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.898 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.898 DEBUG:boto:Method: HEAD 2024-03-04T17:48:17.898 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.898 DEBUG:boto:Data: 2024-03-04T17:48:17.898 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.898 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.899 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.899 DEBUG:boto:Params: {} 2024-03-04T17:48:17.899 DEBUG:boto:Token: None 2024-03-04T17:48:17.899 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.899 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Erl8YLMvqpi7GrJa6ywHXdJDn5w= 2024-03-04T17:48:17.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Erl8YLMvqpi7GrJa6ywHXdJDn5w=', 'Content-Length': '0'} 2024-03-04T17:48:17.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b17e4922edf77b3-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:17.904 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.904 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.904 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.904 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.904 DEBUG:boto:Method: GET 2024-03-04T17:48:17.904 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.905 DEBUG:boto:Data: 2024-03-04T17:48:17.905 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.905 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:17.905 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.905 DEBUG:boto:Params: {} 2024-03-04T17:48:17.905 DEBUG:boto:Token: None 2024-03-04T17:48:17.905 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.905 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DlnX/I0YDZsACpG/gLtOrmrndwk= 2024-03-04T17:48:17.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DlnX/I0YDZsACpG/gLtOrmrndwk=', 'Content-Length': '0'} 2024-03-04T17:48:17.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004336955b5c3608d7-0065e60961-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:17.958 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.958 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:17.958 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.958 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.958 DEBUG:boto:Method: GET 2024-03-04T17:48:17.958 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.958 DEBUG:boto:Data: 2024-03-04T17:48:17.958 DEBUG:boto:Headers: {} 2024-03-04T17:48:17.958 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:17.958 DEBUG:boto:Port: 8000 2024-03-04T17:48:17.958 DEBUG:boto:Params: {} 2024-03-04T17:48:17.959 DEBUG:boto:Token: None 2024-03-04T17:48:17.959 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:17 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:17.959 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DlnX/I0YDZsACpG/gLtOrmrndwk= 2024-03-04T17:48:17.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DlnX/I0YDZsACpG/gLtOrmrndwk=', 'Content-Length': '0'} 2024-03-04T17:48:17.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ef927ed53756ac1-0065e60961-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.008 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.008 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.008 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:18.008 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:18.009 DEBUG:boto:Method: GET 2024-03-04T17:48:18.009 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:18.009 DEBUG:boto:Data: 2024-03-04T17:48:18.009 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.009 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:18.009 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.009 DEBUG:boto:Params: {} 2024-03-04T17:48:18.009 DEBUG:boto:Token: None 2024-03-04T17:48:18.009 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.010 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000741bd06b4bb300ea-0065e60962-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.014 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.015 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.015 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.015 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:18.015 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:18.015 DEBUG:boto:Method: GET 2024-03-04T17:48:18.015 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:18.015 DEBUG:boto:Data: 2024-03-04T17:48:18.015 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.015 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.016 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.016 DEBUG:boto:Params: {} 2024-03-04T17:48:18.016 DEBUG:boto:Token: None 2024-03-04T17:48:18.016 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.016 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012889b14c2d8e7bb-0065e60962-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.021 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:18.022 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.022 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.022 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.022 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.022 DEBUG:boto:Method: GET 2024-03-04T17:48:18.022 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.022 DEBUG:boto:Data: 2024-03-04T17:48:18.022 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.023 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:18.023 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.023 DEBUG:boto:Params: {} 2024-03-04T17:48:18.023 DEBUG:boto:Token: None 2024-03-04T17:48:18.023 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.023 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GAeAjXG54D4nXRRcgKQTwxTJ88Y= 2024-03-04T17:48:18.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GAeAjXG54D4nXRRcgKQTwxTJ88Y=', 'Content-Length': '0'} 2024-03-04T17:48:18.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e47c28fe94e1d191-0065e60962-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.068 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.068 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.068 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.068 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.069 DEBUG:boto:Method: GET 2024-03-04T17:48:18.069 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.069 DEBUG:boto:Data: 2024-03-04T17:48:18.069 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.069 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.069 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.069 DEBUG:boto:Params: {} 2024-03-04T17:48:18.069 DEBUG:boto:Token: None 2024-03-04T17:48:18.069 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.070 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GAeAjXG54D4nXRRcgKQTwxTJ88Y= 2024-03-04T17:48:18.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GAeAjXG54D4nXRRcgKQTwxTJ88Y=', 'Content-Length': '0'} 2024-03-04T17:48:18.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0faa51f6af59c54-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.116 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.116 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.116 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.116 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.117 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.117 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.117 DEBUG:boto:Data: 2024-03-04T17:48:18.117 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.117 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:18.117 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.117 DEBUG:boto:Params: {} 2024-03-04T17:48:18.117 DEBUG:boto:Token: None 2024-03-04T17:48:18.118 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.118 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:d/tDxFTNokK09HA/cnTB368Tigg= 2024-03-04T17:48:18.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:d/tDxFTNokK09HA/cnTB368Tigg=', 'Content-Length': '0'} 2024-03-04T17:48:18.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058f1718fc2e8e574-0065e60962-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.121 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.121 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.121 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.121 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.121 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.121 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.122 DEBUG:boto:Data: 2024-03-04T17:48:18.122 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.122 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.122 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.122 DEBUG:boto:Params: {} 2024-03-04T17:48:18.122 DEBUG:boto:Token: None 2024-03-04T17:48:18.122 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.122 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:d/tDxFTNokK09HA/cnTB368Tigg= 2024-03-04T17:48:18.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:d/tDxFTNokK09HA/cnTB368Tigg=', 'Content-Length': '0'} 2024-03-04T17:48:18.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d3c5059b08d5598-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:18.126 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.126 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.126 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.126 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.126 DEBUG:boto:Method: GET 2024-03-04T17:48:18.126 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.126 DEBUG:boto:Data: 2024-03-04T17:48:18.126 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.127 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:18.127 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.127 DEBUG:boto:Params: {} 2024-03-04T17:48:18.127 DEBUG:boto:Token: None 2024-03-04T17:48:18.127 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.127 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GAeAjXG54D4nXRRcgKQTwxTJ88Y= 2024-03-04T17:48:18.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GAeAjXG54D4nXRRcgKQTwxTJ88Y=', 'Content-Length': '0'} 2024-03-04T17:48:18.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0439e9fedbb30c6-0065e60962-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.172 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.172 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.172 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.172 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.172 DEBUG:boto:Method: GET 2024-03-04T17:48:18.173 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.173 DEBUG:boto:Data: 2024-03-04T17:48:18.173 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.173 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.173 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.173 DEBUG:boto:Params: {} 2024-03-04T17:48:18.173 DEBUG:boto:Token: None 2024-03-04T17:48:18.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.173 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GAeAjXG54D4nXRRcgKQTwxTJ88Y= 2024-03-04T17:48:18.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GAeAjXG54D4nXRRcgKQTwxTJ88Y=', 'Content-Length': '0'} 2024-03-04T17:48:18.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1883a98e75fac75-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.220 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.220 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.220 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:18.220 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:18.220 DEBUG:boto:Method: GET 2024-03-04T17:48:18.221 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:18.221 DEBUG:boto:Data: 2024-03-04T17:48:18.221 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.221 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:18.221 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.221 DEBUG:boto:Params: {} 2024-03-04T17:48:18.221 DEBUG:boto:Token: None 2024-03-04T17:48:18.221 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.221 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000067af8dfcce69766-0065e60962-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.227 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.228 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.228 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.228 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:18.228 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:18.228 DEBUG:boto:Method: GET 2024-03-04T17:48:18.228 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:18.229 DEBUG:boto:Data: 2024-03-04T17:48:18.229 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.229 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.229 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.229 DEBUG:boto:Params: {} 2024-03-04T17:48:18.229 DEBUG:boto:Token: None 2024-03-04T17:48:18.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.229 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095275f3d432064d0-0065e60962-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.235 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.235 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.235 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.236 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.236 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.236 DEBUG:boto:Data: 2024-03-04T17:48:18.236 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.236 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:18.236 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.236 DEBUG:boto:Params: {} 2024-03-04T17:48:18.236 DEBUG:boto:Token: None 2024-03-04T17:48:18.236 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.236 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PmNYA807oK+anwv8JdIxCh6EPPc= 2024-03-04T17:48:18.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PmNYA807oK+anwv8JdIxCh6EPPc=', 'Content-Length': '0'} 2024-03-04T17:48:18.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ea26c8bf87f658a-0065e60962-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.241 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.241 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.241 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.241 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.241 DEBUG:boto:Data: 2024-03-04T17:48:18.241 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.241 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.241 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.241 DEBUG:boto:Params: {} 2024-03-04T17:48:18.242 DEBUG:boto:Token: None 2024-03-04T17:48:18.242 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.242 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PmNYA807oK+anwv8JdIxCh6EPPc= 2024-03-04T17:48:18.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PmNYA807oK+anwv8JdIxCh6EPPc=', 'Content-Length': '0'} 2024-03-04T17:48:18.245 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002dfef582c35571b2-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:18.246 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.246 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.246 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.246 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.246 DEBUG:boto:Method: GET 2024-03-04T17:48:18.246 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.246 DEBUG:boto:Data: 2024-03-04T17:48:18.246 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.247 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:18.247 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.247 DEBUG:boto:Params: {} 2024-03-04T17:48:18.247 DEBUG:boto:Token: None 2024-03-04T17:48:18.247 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.247 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GAeAjXG54D4nXRRcgKQTwxTJ88Y= 2024-03-04T17:48:18.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GAeAjXG54D4nXRRcgKQTwxTJ88Y=', 'Content-Length': '0'} 2024-03-04T17:48:18.250 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7fa721c96ccffc3-0065e60962-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.296 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.296 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:18.296 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.296 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.297 DEBUG:boto:Method: GET 2024-03-04T17:48:18.297 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.297 DEBUG:boto:Data: 2024-03-04T17:48:18.297 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.297 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.297 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.297 DEBUG:boto:Params: {} 2024-03-04T17:48:18.297 DEBUG:boto:Token: None 2024-03-04T17:48:18.297 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:18.298 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GAeAjXG54D4nXRRcgKQTwxTJ88Y= 2024-03-04T17:48:18.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GAeAjXG54D4nXRRcgKQTwxTJ88Y=', 'Content-Length': '0'} 2024-03-04T17:48:18.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027ea092bcc58954f-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.344 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.344 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.344 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:18.344 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:18.345 DEBUG:boto:Method: GET 2024-03-04T17:48:18.345 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:18.345 DEBUG:boto:Data: 2024-03-04T17:48:18.345 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.345 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:18.345 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.345 DEBUG:boto:Params: {} 2024-03-04T17:48:18.345 DEBUG:boto:Token: None 2024-03-04T17:48:18.345 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.346 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dff8bec2078821f3-0065e60962-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.392 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.393 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.393 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.393 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:18.393 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:18.393 DEBUG:boto:Method: GET 2024-03-04T17:48:18.393 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:18.393 DEBUG:boto:Data: 2024-03-04T17:48:18.393 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.393 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.393 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.394 DEBUG:boto:Params: {} 2024-03-04T17:48:18.394 DEBUG:boto:Token: None 2024-03-04T17:48:18.394 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.394 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b75e565747768512-0065e60962-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.399 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.400 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-20 zones={test-zone2, test-zone1} 2024-03-04T17:48:18.400 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-20 zones={test-zone3, test-zone1} 2024-03-04T17:48:18.400 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.400 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.400 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.400 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:48:18.400 DEBUG:boto:Data: 2024-03-04T17:48:18.400 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.400 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.401 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.401 DEBUG:boto:Params: {} 2024-03-04T17:48:18.401 DEBUG:boto:Token: None 2024-03-04T17:48:18.401 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.401 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uvCuornsOZJ50zS+Uyiq+Xv+WVY= 2024-03-04T17:48:18.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uvCuornsOZJ50zS+Uyiq+Xv+WVY=', 'Content-Length': '0'} 2024-03-04T17:48:18.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000017ca9cbb8ec7a7f-0065e60962-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.405 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.405 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.406 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.406 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:48:18.406 DEBUG:boto:Data: 2024-03-04T17:48:18.406 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.406 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.406 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.406 DEBUG:boto:Params: {} 2024-03-04T17:48:18.406 DEBUG:boto:Token: None 2024-03-04T17:48:18.407 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.407 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uvCuornsOZJ50zS+Uyiq+Xv+WVY= 2024-03-04T17:48:18.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uvCuornsOZJ50zS+Uyiq+Xv+WVY=', 'Content-Length': '0'} 2024-03-04T17:48:18.411 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d7b1232ffe3bba1-0065e60962-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.411 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:48:18.411 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.411 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.411 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:18.411 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:18.411 DEBUG:boto:Method: GET 2024-03-04T17:48:18.412 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:18.412 DEBUG:boto:Data: 2024-03-04T17:48:18.412 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.412 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.412 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.412 DEBUG:boto:Params: {} 2024-03-04T17:48:18.412 DEBUG:boto:Token: None 2024-03-04T17:48:18.412 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/?versions 2024-03-04T17:48:18.412 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sVbuCT9d/MQUl24kvsoR7OKxuhY= 2024-03-04T17:48:18.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sVbuCT9d/MQUl24kvsoR7OKxuhY=', 'Content-Length': '0'} 2024-03-04T17:48:18.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fb9d2350249ef39-0065e60962-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.417 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.417 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:18.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:18.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:18.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:18.418 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:48:18.418 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.418 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.418 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:18.418 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:18.418 DEBUG:boto:Method: GET 2024-03-04T17:48:18.418 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:18.418 DEBUG:boto:Data: 2024-03-04T17:48:18.419 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.419 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.419 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.419 DEBUG:boto:Params: {} 2024-03-04T17:48:18.419 DEBUG:boto:Token: None 2024-03-04T17:48:18.419 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/?versions 2024-03-04T17:48:18.419 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sVbuCT9d/MQUl24kvsoR7OKxuhY= 2024-03-04T17:48:18.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sVbuCT9d/MQUl24kvsoR7OKxuhY=', 'Content-Length': '0'} 2024-03-04T17:48:18.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba7717aa578be33d-0065e60962-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.425 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.425 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:18.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:18.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:18.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:18.426 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.426 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.426 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:18.426 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:18.426 DEBUG:boto:Method: GET 2024-03-04T17:48:18.426 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:18.426 DEBUG:boto:Data: 2024-03-04T17:48:18.426 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.427 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.427 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.427 DEBUG:boto:Params: {} 2024-03-04T17:48:18.427 DEBUG:boto:Token: None 2024-03-04T17:48:18.427 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/?versions 2024-03-04T17:48:18.427 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sVbuCT9d/MQUl24kvsoR7OKxuhY= 2024-03-04T17:48:18.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sVbuCT9d/MQUl24kvsoR7OKxuhY=', 'Content-Length': '0'} 2024-03-04T17:48:18.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eca16250fc05b0b-0065e60962-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.432 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.432 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.433 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.433 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:18.433 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:18.433 DEBUG:boto:Method: GET 2024-03-04T17:48:18.433 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:18.433 DEBUG:boto:Data: 2024-03-04T17:48:18.433 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.433 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.433 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.433 DEBUG:boto:Params: {} 2024-03-04T17:48:18.433 DEBUG:boto:Token: None 2024-03-04T17:48:18.434 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/?versions 2024-03-04T17:48:18.434 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sVbuCT9d/MQUl24kvsoR7OKxuhY= 2024-03-04T17:48:18.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sVbuCT9d/MQUl24kvsoR7OKxuhY=', 'Content-Length': '0'} 2024-03-04T17:48:18.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b8fc9042b6eddba-0065e60962-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.439 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:18.439 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:18.439 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:18.439 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.439 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.440 DEBUG:boto:Method: GET 2024-03-04T17:48:18.440 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.440 DEBUG:boto:Data: 2024-03-04T17:48:18.440 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.440 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.440 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.440 DEBUG:boto:Params: {} 2024-03-04T17:48:18.440 DEBUG:boto:Token: None 2024-03-04T17:48:18.441 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.441 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4PT4lcq1p6OGEw+88AcnNW/UZiY= 2024-03-04T17:48:18.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4PT4lcq1p6OGEw+88AcnNW/UZiY=', 'Content-Length': '0'} 2024-03-04T17:48:18.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f52bc1ffc2efe19e-0065e60962-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.444 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:18.444 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:18.444 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.444 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.444 DEBUG:boto:Method: GET 2024-03-04T17:48:18.444 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.444 DEBUG:boto:Data: 2024-03-04T17:48:18.444 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.444 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.444 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.444 DEBUG:boto:Params: {} 2024-03-04T17:48:18.445 DEBUG:boto:Token: None 2024-03-04T17:48:18.445 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.445 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4PT4lcq1p6OGEw+88AcnNW/UZiY= 2024-03-04T17:48:18.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4PT4lcq1p6OGEw+88AcnNW/UZiY=', 'Content-Length': '0'} 2024-03-04T17:48:18.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a0bb299dbc685215-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.448 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:18.448 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:18.448 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.449 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.449 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.449 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.449 DEBUG:boto:Data: 2024-03-04T17:48:18.449 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.449 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.449 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.449 DEBUG:boto:Params: {} 2024-03-04T17:48:18.449 DEBUG:boto:Token: None 2024-03-04T17:48:18.449 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.450 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UZiScM+bSO9SUO3rRxMIaejYZe4= 2024-03-04T17:48:18.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UZiScM+bSO9SUO3rRxMIaejYZe4=', 'Content-Length': '0'} 2024-03-04T17:48:18.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e23d6c82d45206f-0065e60962-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.453 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:18.453 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:18.453 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.453 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.453 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.453 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.453 DEBUG:boto:Data: 2024-03-04T17:48:18.453 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.454 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.454 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.454 DEBUG:boto:Params: {} 2024-03-04T17:48:18.454 DEBUG:boto:Token: None 2024-03-04T17:48:18.454 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.454 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UZiScM+bSO9SUO3rRxMIaejYZe4= 2024-03-04T17:48:18.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UZiScM+bSO9SUO3rRxMIaejYZe4=', 'Content-Length': '0'} 2024-03-04T17:48:18.457 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9e561355d0fe9f5-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:18.457 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:18.458 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:18.458 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.458 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.458 DEBUG:boto:Method: GET 2024-03-04T17:48:18.458 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.458 DEBUG:boto:Data: 2024-03-04T17:48:18.458 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.458 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.458 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.458 DEBUG:boto:Params: {} 2024-03-04T17:48:18.459 DEBUG:boto:Token: None 2024-03-04T17:48:18.459 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.459 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4PT4lcq1p6OGEw+88AcnNW/UZiY= 2024-03-04T17:48:18.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4PT4lcq1p6OGEw+88AcnNW/UZiY=', 'Content-Length': '0'} 2024-03-04T17:48:18.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d4051c8c67199014-0065e60962-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.463 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:18.463 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:18.463 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.464 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.464 DEBUG:boto:Method: GET 2024-03-04T17:48:18.464 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.464 DEBUG:boto:Data: 2024-03-04T17:48:18.464 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.464 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.464 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.464 DEBUG:boto:Params: {} 2024-03-04T17:48:18.464 DEBUG:boto:Token: None 2024-03-04T17:48:18.464 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.465 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4PT4lcq1p6OGEw+88AcnNW/UZiY= 2024-03-04T17:48:18.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4PT4lcq1p6OGEw+88AcnNW/UZiY=', 'Content-Length': '0'} 2024-03-04T17:48:18.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c252265a3588f31-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.468 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.468 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.468 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:18.468 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:18.468 DEBUG:boto:Method: GET 2024-03-04T17:48:18.468 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:48:18.468 DEBUG:boto:Data: 2024-03-04T17:48:18.468 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.468 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.468 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.468 DEBUG:boto:Params: {} 2024-03-04T17:48:18.469 DEBUG:boto:Token: None 2024-03-04T17:48:18.469 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.469 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f710c8c6819b6c70-0065e60962-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.516 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.517 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.517 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.517 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:18.517 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:18.517 DEBUG:boto:Method: GET 2024-03-04T17:48:18.517 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:48:18.517 DEBUG:boto:Data: 2024-03-04T17:48:18.517 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.517 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.517 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.518 DEBUG:boto:Params: {} 2024-03-04T17:48:18.518 DEBUG:boto:Token: None 2024-03-04T17:48:18.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.518 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c54a51a08f4414c0-0065e60962-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.523 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.524 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:18.524 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:18.524 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.524 DEBUG:boto:Path: /ppfahn-20/dummy 2024-03-04T17:48:18.524 DEBUG:boto:Data: 2024-03-04T17:48:18.524 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.524 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.524 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.524 DEBUG:boto:Params: {} 2024-03-04T17:48:18.525 DEBUG:boto:Token: None 2024-03-04T17:48:18.525 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/dummy 2024-03-04T17:48:18.525 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7QVD7vkoRavKEtdddwYqQnf2SaY= 2024-03-04T17:48:18.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7QVD7vkoRavKEtdddwYqQnf2SaY=', 'Content-Length': '0'} 2024-03-04T17:48:18.528 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bcca8ff0ba38bd81-0065e60962-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.529 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:18.529 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:18.529 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.529 DEBUG:boto:Path: /ppfahn-20/dummy 2024-03-04T17:48:18.529 DEBUG:boto:Data: 2024-03-04T17:48:18.529 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.530 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.530 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.530 DEBUG:boto:Params: {} 2024-03-04T17:48:18.530 DEBUG:boto:Token: None 2024-03-04T17:48:18.530 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/dummy 2024-03-04T17:48:18.530 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7QVD7vkoRavKEtdddwYqQnf2SaY= 2024-03-04T17:48:18.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7QVD7vkoRavKEtdddwYqQnf2SaY=', 'Content-Length': '0'} 2024-03-04T17:48:18.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000085b6fb02c9592aeb-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:18.534 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:18.534 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:18.535 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.535 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.535 DEBUG:boto:Method: GET 2024-03-04T17:48:18.535 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.535 DEBUG:boto:Data: 2024-03-04T17:48:18.535 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.535 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.535 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.535 DEBUG:boto:Params: {} 2024-03-04T17:48:18.535 DEBUG:boto:Token: None 2024-03-04T17:48:18.536 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.536 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4PT4lcq1p6OGEw+88AcnNW/UZiY= 2024-03-04T17:48:18.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4PT4lcq1p6OGEw+88AcnNW/UZiY=', 'Content-Length': '0'} 2024-03-04T17:48:18.539 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a8d19f0410f9bd82-0065e60962-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.539 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:18.539 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:18.539 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.540 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.540 DEBUG:boto:Method: GET 2024-03-04T17:48:18.540 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.540 DEBUG:boto:Data: 2024-03-04T17:48:18.540 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.540 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.540 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.540 DEBUG:boto:Params: {} 2024-03-04T17:48:18.540 DEBUG:boto:Token: None 2024-03-04T17:48:18.541 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:18.541 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4PT4lcq1p6OGEw+88AcnNW/UZiY= 2024-03-04T17:48:18.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4PT4lcq1p6OGEw+88AcnNW/UZiY=', 'Content-Length': '0'} 2024-03-04T17:48:18.543 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e1a8885ff88403b4-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.544 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.544 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.544 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:18.544 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:18.544 DEBUG:boto:Method: GET 2024-03-04T17:48:18.544 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:48:18.544 DEBUG:boto:Data: 2024-03-04T17:48:18.544 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.545 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.545 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.545 DEBUG:boto:Params: {} 2024-03-04T17:48:18.545 DEBUG:boto:Token: None 2024-03-04T17:48:18.545 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.545 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d45b71822d9dcda-0065e60962-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.549 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.550 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.550 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.550 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:18.550 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:18.550 DEBUG:boto:Method: GET 2024-03-04T17:48:18.550 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:48:18.550 DEBUG:boto:Data: 2024-03-04T17:48:18.550 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.551 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.551 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.551 DEBUG:boto:Params: {} 2024-03-04T17:48:18.551 DEBUG:boto:Token: None 2024-03-04T17:48:18.551 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.551 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cd43e8cb15cfd73-0065e60962-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.555 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:18.556 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.556 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.556 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.556 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.556 DEBUG:boto:Method: GET 2024-03-04T17:48:18.556 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.557 DEBUG:boto:Data: 2024-03-04T17:48:18.557 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.557 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.557 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.557 DEBUG:boto:Params: {} 2024-03-04T17:48:18.557 DEBUG:boto:Token: None 2024-03-04T17:48:18.557 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.557 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8jDtEJmi31TcRslppvVvP7HxtS0= 2024-03-04T17:48:18.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8jDtEJmi31TcRslppvVvP7HxtS0=', 'Content-Length': '0'} 2024-03-04T17:48:18.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bc6d95208b08850-0065e60962-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.604 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.604 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.604 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.604 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.604 DEBUG:boto:Method: GET 2024-03-04T17:48:18.605 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.605 DEBUG:boto:Data: 2024-03-04T17:48:18.605 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.605 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.605 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.605 DEBUG:boto:Params: {} 2024-03-04T17:48:18.605 DEBUG:boto:Token: None 2024-03-04T17:48:18.605 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.605 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8jDtEJmi31TcRslppvVvP7HxtS0= 2024-03-04T17:48:18.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8jDtEJmi31TcRslppvVvP7HxtS0=', 'Content-Length': '0'} 2024-03-04T17:48:18.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe667b6f7acd93d0-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.609 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.609 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.609 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.609 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.609 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.609 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.609 DEBUG:boto:Data: 2024-03-04T17:48:18.609 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.609 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.610 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.610 DEBUG:boto:Params: {} 2024-03-04T17:48:18.610 DEBUG:boto:Token: None 2024-03-04T17:48:18.610 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.610 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xDACY2qnH0JvUfB9MBppshsbx70= 2024-03-04T17:48:18.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xDACY2qnH0JvUfB9MBppshsbx70=', 'Content-Length': '0'} 2024-03-04T17:48:18.613 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d69b2a7bef9b2876-0065e60962-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.614 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.614 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.614 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.614 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.614 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.614 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.614 DEBUG:boto:Data: 2024-03-04T17:48:18.614 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.614 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.615 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.615 DEBUG:boto:Params: {} 2024-03-04T17:48:18.615 DEBUG:boto:Token: None 2024-03-04T17:48:18.615 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.615 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xDACY2qnH0JvUfB9MBppshsbx70= 2024-03-04T17:48:18.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xDACY2qnH0JvUfB9MBppshsbx70=', 'Content-Length': '0'} 2024-03-04T17:48:18.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b42c889edecba0f6-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:18.618 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.618 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.618 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.619 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.619 DEBUG:boto:Method: GET 2024-03-04T17:48:18.619 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.619 DEBUG:boto:Data: 2024-03-04T17:48:18.619 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.619 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.619 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.619 DEBUG:boto:Params: {} 2024-03-04T17:48:18.619 DEBUG:boto:Token: None 2024-03-04T17:48:18.620 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.620 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8jDtEJmi31TcRslppvVvP7HxtS0= 2024-03-04T17:48:18.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8jDtEJmi31TcRslppvVvP7HxtS0=', 'Content-Length': '0'} 2024-03-04T17:48:18.623 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f33120a26fffec8-0065e60962-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.664 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.664 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.664 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.664 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.664 DEBUG:boto:Method: GET 2024-03-04T17:48:18.664 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.665 DEBUG:boto:Data: 2024-03-04T17:48:18.665 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.665 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.665 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.665 DEBUG:boto:Params: {} 2024-03-04T17:48:18.665 DEBUG:boto:Token: None 2024-03-04T17:48:18.665 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.665 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8jDtEJmi31TcRslppvVvP7HxtS0= 2024-03-04T17:48:18.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8jDtEJmi31TcRslppvVvP7HxtS0=', 'Content-Length': '0'} 2024-03-04T17:48:18.669 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066b37ca988ae7715-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.669 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.669 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.669 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:18.669 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:18.669 DEBUG:boto:Method: GET 2024-03-04T17:48:18.669 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:18.670 DEBUG:boto:Data: 2024-03-04T17:48:18.670 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.670 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.670 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.670 DEBUG:boto:Params: {} 2024-03-04T17:48:18.670 DEBUG:boto:Token: None 2024-03-04T17:48:18.670 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.670 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000716dffd593b7f0ce-0065e60962-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.720 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.720 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.721 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.721 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:18.721 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:18.721 DEBUG:boto:Method: GET 2024-03-04T17:48:18.721 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:18.721 DEBUG:boto:Data: 2024-03-04T17:48:18.721 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.721 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.721 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.721 DEBUG:boto:Params: {} 2024-03-04T17:48:18.722 DEBUG:boto:Token: None 2024-03-04T17:48:18.722 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.722 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086adb38ad1181b38-0065e60962-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.727 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.727 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.727 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.727 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.727 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.727 DEBUG:boto:Data: 2024-03-04T17:48:18.727 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.728 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.728 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.728 DEBUG:boto:Params: {} 2024-03-04T17:48:18.728 DEBUG:boto:Token: None 2024-03-04T17:48:18.728 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.728 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2mtbUNARM95P1t3+yMpPNWHvOog= 2024-03-04T17:48:18.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2mtbUNARM95P1t3+yMpPNWHvOog=', 'Content-Length': '0'} 2024-03-04T17:48:18.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f17fa8b62258905-0065e60962-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.732 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.732 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.732 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.733 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.733 DEBUG:boto:Data: 2024-03-04T17:48:18.733 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.733 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.733 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.733 DEBUG:boto:Params: {} 2024-03-04T17:48:18.733 DEBUG:boto:Token: None 2024-03-04T17:48:18.733 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.733 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2mtbUNARM95P1t3+yMpPNWHvOog= 2024-03-04T17:48:18.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2mtbUNARM95P1t3+yMpPNWHvOog=', 'Content-Length': '0'} 2024-03-04T17:48:18.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d0f9dc304401b7a-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:18.737 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.738 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.738 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.738 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.738 DEBUG:boto:Method: GET 2024-03-04T17:48:18.738 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.738 DEBUG:boto:Data: 2024-03-04T17:48:18.738 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.738 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.738 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.738 DEBUG:boto:Params: {} 2024-03-04T17:48:18.739 DEBUG:boto:Token: None 2024-03-04T17:48:18.739 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.739 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8jDtEJmi31TcRslppvVvP7HxtS0= 2024-03-04T17:48:18.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8jDtEJmi31TcRslppvVvP7HxtS0=', 'Content-Length': '0'} 2024-03-04T17:48:18.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e6fd3df05c0c984-0065e60962-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.784 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.784 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:18.784 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.784 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.784 DEBUG:boto:Method: GET 2024-03-04T17:48:18.784 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.785 DEBUG:boto:Data: 2024-03-04T17:48:18.785 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.785 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.785 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.785 DEBUG:boto:Params: {} 2024-03-04T17:48:18.785 DEBUG:boto:Token: None 2024-03-04T17:48:18.785 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:18.785 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8jDtEJmi31TcRslppvVvP7HxtS0= 2024-03-04T17:48:18.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8jDtEJmi31TcRslppvVvP7HxtS0=', 'Content-Length': '0'} 2024-03-04T17:48:18.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c78a2dcae3637ad8-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.789 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.789 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.789 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:18.789 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:18.789 DEBUG:boto:Method: GET 2024-03-04T17:48:18.789 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:18.789 DEBUG:boto:Data: 2024-03-04T17:48:18.789 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.789 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.790 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.790 DEBUG:boto:Params: {} 2024-03-04T17:48:18.790 DEBUG:boto:Token: None 2024-03-04T17:48:18.790 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.790 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000620f45daa92aad7c-0065e60962-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.836 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.836 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:18.837 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:18.837 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:18.837 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:18.837 DEBUG:boto:Method: GET 2024-03-04T17:48:18.837 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:18.837 DEBUG:boto:Data: 2024-03-04T17:48:18.837 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.837 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.837 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.838 DEBUG:boto:Params: {} 2024-03-04T17:48:18.838 DEBUG:boto:Token: None 2024-03-04T17:48:18.838 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/ 2024-03-04T17:48:18.838 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU= 2024-03-04T17:48:18.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/rL8cNOC8+aUdJTYs3V+8GdpCU=', 'Content-Length': '0'} 2024-03-04T17:48:18.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f27c28fa20df9120-0065e60962-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.843 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:18.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:18.843 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:18.843 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:18.843 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.844 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.844 DEBUG:boto:Method: GET 2024-03-04T17:48:18.844 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.844 DEBUG:boto:Data: 2024-03-04T17:48:18.844 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.844 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.844 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.844 DEBUG:boto:Params: {} 2024-03-04T17:48:18.844 DEBUG:boto:Token: None 2024-03-04T17:48:18.844 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.845 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QPmn9gl0EjzpbVlBvO+F+8m2GOk= 2024-03-04T17:48:18.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QPmn9gl0EjzpbVlBvO+F+8m2GOk=', 'Content-Length': '0'} 2024-03-04T17:48:18.847 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9cf7e3dc397b83d-0065e60962-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.848 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:18.848 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:18.848 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.848 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.848 DEBUG:boto:Method: GET 2024-03-04T17:48:18.848 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.848 DEBUG:boto:Data: 2024-03-04T17:48:18.848 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.848 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.849 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.849 DEBUG:boto:Params: {} 2024-03-04T17:48:18.849 DEBUG:boto:Token: None 2024-03-04T17:48:18.849 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.849 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QPmn9gl0EjzpbVlBvO+F+8m2GOk= 2024-03-04T17:48:18.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QPmn9gl0EjzpbVlBvO+F+8m2GOk=', 'Content-Length': '0'} 2024-03-04T17:48:18.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fd91cf02e917f47-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.900 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:18.900 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:18.900 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.901 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.901 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.901 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.901 DEBUG:boto:Data: 2024-03-04T17:48:18.901 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.901 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.901 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.901 DEBUG:boto:Params: {} 2024-03-04T17:48:18.901 DEBUG:boto:Token: None 2024-03-04T17:48:18.902 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.902 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ve42TrOrWJHpJbWPc7L74r4XEto= 2024-03-04T17:48:18.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ve42TrOrWJHpJbWPc7L74r4XEto=', 'Content-Length': '0'} 2024-03-04T17:48:18.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2f188ceb2d6f21c-0065e60962-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.905 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:18.905 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:18.905 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.905 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.906 DEBUG:boto:Method: HEAD 2024-03-04T17:48:18.906 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.906 DEBUG:boto:Data: 2024-03-04T17:48:18.906 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.906 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.906 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.906 DEBUG:boto:Params: {} 2024-03-04T17:48:18.906 DEBUG:boto:Token: None 2024-03-04T17:48:18.906 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.906 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ve42TrOrWJHpJbWPc7L74r4XEto= 2024-03-04T17:48:18.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ve42TrOrWJHpJbWPc7L74r4XEto=', 'Content-Length': '0'} 2024-03-04T17:48:18.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd6ce50efab3b2d3-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:18.910 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:18.910 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:18.910 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.910 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.910 DEBUG:boto:Method: GET 2024-03-04T17:48:18.910 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.910 DEBUG:boto:Data: 2024-03-04T17:48:18.910 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.910 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:18.911 DEBUG:boto:Port: 8001 2024-03-04T17:48:18.911 DEBUG:boto:Params: {} 2024-03-04T17:48:18.911 DEBUG:boto:Token: None 2024-03-04T17:48:18.911 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.911 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QPmn9gl0EjzpbVlBvO+F+8m2GOk= 2024-03-04T17:48:18.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QPmn9gl0EjzpbVlBvO+F+8m2GOk=', 'Content-Length': '0'} 2024-03-04T17:48:18.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc1e82191d67c816-0065e60962-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:18.956 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:18.956 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:18.956 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.956 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.957 DEBUG:boto:Method: GET 2024-03-04T17:48:18.957 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.957 DEBUG:boto:Data: 2024-03-04T17:48:18.957 DEBUG:boto:Headers: {} 2024-03-04T17:48:18.957 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:18.957 DEBUG:boto:Port: 8000 2024-03-04T17:48:18.957 DEBUG:boto:Params: {} 2024-03-04T17:48:18.957 DEBUG:boto:Token: None 2024-03-04T17:48:18.957 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:18 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:18.957 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QPmn9gl0EjzpbVlBvO+F+8m2GOk= 2024-03-04T17:48:18.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QPmn9gl0EjzpbVlBvO+F+8m2GOk=', 'Content-Length': '0'} 2024-03-04T17:48:18.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004efbe2ed39f0371e-0065e60962-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.004 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.004 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.004 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:19.004 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:19.005 DEBUG:boto:Method: GET 2024-03-04T17:48:19.005 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:19.005 DEBUG:boto:Data: 2024-03-04T17:48:19.005 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.005 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.005 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.005 DEBUG:boto:Params: {} 2024-03-04T17:48:19.005 DEBUG:boto:Token: None 2024-03-04T17:48:19.005 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.006 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e82bc521828b63c-0065e60963-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.052 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.053 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.053 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.053 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:19.053 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:19.053 DEBUG:boto:Method: GET 2024-03-04T17:48:19.053 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:19.053 DEBUG:boto:Data: 2024-03-04T17:48:19.053 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.053 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:19.054 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.054 DEBUG:boto:Params: {} 2024-03-04T17:48:19.054 DEBUG:boto:Token: None 2024-03-04T17:48:19.054 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.054 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fae12e4ebfcda34f-0065e60963-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.059 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.059 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:19.059 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:19.060 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.060 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:48:19.060 DEBUG:boto:Data: 2024-03-04T17:48:19.060 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.060 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.060 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.060 DEBUG:boto:Params: {} 2024-03-04T17:48:19.060 DEBUG:boto:Token: None 2024-03-04T17:48:19.060 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:48:19.061 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:x7MU8rq4YP6gbmEcneClR7PBPbw= 2024-03-04T17:48:19.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:x7MU8rq4YP6gbmEcneClR7PBPbw=', 'Content-Length': '0'} 2024-03-04T17:48:19.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff370cf8f19102e5-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.065 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:19.065 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:19.065 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.065 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:48:19.065 DEBUG:boto:Data: 2024-03-04T17:48:19.065 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.065 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:19.066 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.066 DEBUG:boto:Params: {} 2024-03-04T17:48:19.066 DEBUG:boto:Token: None 2024-03-04T17:48:19.066 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:48:19.066 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:x7MU8rq4YP6gbmEcneClR7PBPbw= 2024-03-04T17:48:19.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:x7MU8rq4YP6gbmEcneClR7PBPbw=', 'Content-Length': '0'} 2024-03-04T17:48:19.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4a0bb0ab6301cc5-0065e60963-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:19.070 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:19.070 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:19.071 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:19.071 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:19.071 DEBUG:boto:Method: GET 2024-03-04T17:48:19.071 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:19.071 DEBUG:boto:Data: 2024-03-04T17:48:19.071 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.071 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.071 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.071 DEBUG:boto:Params: {} 2024-03-04T17:48:19.071 DEBUG:boto:Token: None 2024-03-04T17:48:19.071 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:19.072 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9ASteM4EXY7EhBBZ+rSAgG7MQ6Q= 2024-03-04T17:48:19.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9ASteM4EXY7EhBBZ+rSAgG7MQ6Q=', 'Content-Length': '0'} 2024-03-04T17:48:19.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9fa21a8c18a01ce-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.120 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:19.120 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:19.120 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:19.120 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:19.120 DEBUG:boto:Method: GET 2024-03-04T17:48:19.121 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:19.121 DEBUG:boto:Data: 2024-03-04T17:48:19.121 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.121 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:19.121 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.121 DEBUG:boto:Params: {} 2024-03-04T17:48:19.121 DEBUG:boto:Token: None 2024-03-04T17:48:19.121 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:19.121 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9ASteM4EXY7EhBBZ+rSAgG7MQ6Q= 2024-03-04T17:48:19.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9ASteM4EXY7EhBBZ+rSAgG7MQ6Q=', 'Content-Length': '0'} 2024-03-04T17:48:19.124 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0e1c0a6e6da8c05-0065e60963-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.168 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.168 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.168 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:19.168 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:19.168 DEBUG:boto:Method: GET 2024-03-04T17:48:19.169 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:19.169 DEBUG:boto:Data: 2024-03-04T17:48:19.169 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.169 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.169 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.169 DEBUG:boto:Params: {} 2024-03-04T17:48:19.169 DEBUG:boto:Token: None 2024-03-04T17:48:19.169 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.169 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b9fccb861f54d78-0065e60963-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.216 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.216 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.217 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.217 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:19.217 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:19.217 DEBUG:boto:Method: GET 2024-03-04T17:48:19.217 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:19.217 DEBUG:boto:Data: 2024-03-04T17:48:19.217 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.217 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:19.217 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.217 DEBUG:boto:Params: {} 2024-03-04T17:48:19.218 DEBUG:boto:Token: None 2024-03-04T17:48:19.218 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.218 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9c59d6f92b30098-0065e60963-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.223 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:19.223 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.223 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.224 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.224 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.224 DEBUG:boto:Method: GET 2024-03-04T17:48:19.224 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.224 DEBUG:boto:Data: 2024-03-04T17:48:19.224 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.224 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.224 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.224 DEBUG:boto:Params: {} 2024-03-04T17:48:19.224 DEBUG:boto:Token: None 2024-03-04T17:48:19.225 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.225 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GKU9GkZud4rW8SEpCPOV6v5G0pY= 2024-03-04T17:48:19.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GKU9GkZud4rW8SEpCPOV6v5G0pY=', 'Content-Length': '0'} 2024-03-04T17:48:19.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000901ef1c67eacd640-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.228 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.228 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.228 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.228 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.228 DEBUG:boto:Method: GET 2024-03-04T17:48:19.228 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.229 DEBUG:boto:Data: 2024-03-04T17:48:19.229 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.229 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:19.229 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.229 DEBUG:boto:Params: {} 2024-03-04T17:48:19.229 DEBUG:boto:Token: None 2024-03-04T17:48:19.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.229 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GKU9GkZud4rW8SEpCPOV6v5G0pY= 2024-03-04T17:48:19.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GKU9GkZud4rW8SEpCPOV6v5G0pY=', 'Content-Length': '0'} 2024-03-04T17:48:19.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b3bcf0dba3fcab2-0065e60963-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.276 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.276 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.276 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.276 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.276 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.277 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.277 DEBUG:boto:Data: 2024-03-04T17:48:19.277 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.277 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.277 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.277 DEBUG:boto:Params: {} 2024-03-04T17:48:19.277 DEBUG:boto:Token: None 2024-03-04T17:48:19.277 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.277 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:azupAo8c7d9FIbqCnVjEYNBZuuI= 2024-03-04T17:48:19.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:azupAo8c7d9FIbqCnVjEYNBZuuI=', 'Content-Length': '0'} 2024-03-04T17:48:19.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffe070ec50ad8276-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.281 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.281 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.281 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.281 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.281 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.281 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.281 DEBUG:boto:Data: 2024-03-04T17:48:19.281 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.281 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:19.281 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.282 DEBUG:boto:Params: {} 2024-03-04T17:48:19.282 DEBUG:boto:Token: None 2024-03-04T17:48:19.282 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.282 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:azupAo8c7d9FIbqCnVjEYNBZuuI= 2024-03-04T17:48:19.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:azupAo8c7d9FIbqCnVjEYNBZuuI=', 'Content-Length': '0'} 2024-03-04T17:48:19.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff9c78c8f50d9d6d-0065e60963-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:19.285 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.285 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.285 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.285 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.285 DEBUG:boto:Method: GET 2024-03-04T17:48:19.286 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.286 DEBUG:boto:Data: 2024-03-04T17:48:19.286 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.286 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.286 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.286 DEBUG:boto:Params: {} 2024-03-04T17:48:19.286 DEBUG:boto:Token: None 2024-03-04T17:48:19.286 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.286 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GKU9GkZud4rW8SEpCPOV6v5G0pY= 2024-03-04T17:48:19.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GKU9GkZud4rW8SEpCPOV6v5G0pY=', 'Content-Length': '0'} 2024-03-04T17:48:19.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa5b56909bb70bda-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.292 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.292 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.292 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.293 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.293 DEBUG:boto:Method: GET 2024-03-04T17:48:19.293 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.293 DEBUG:boto:Data: 2024-03-04T17:48:19.293 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.293 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:19.293 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.293 DEBUG:boto:Params: {} 2024-03-04T17:48:19.294 DEBUG:boto:Token: None 2024-03-04T17:48:19.294 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.294 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GKU9GkZud4rW8SEpCPOV6v5G0pY= 2024-03-04T17:48:19.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GKU9GkZud4rW8SEpCPOV6v5G0pY=', 'Content-Length': '0'} 2024-03-04T17:48:19.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053e983773fc783b6-0065e60963-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.344 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.344 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.345 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:19.345 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:19.345 DEBUG:boto:Method: GET 2024-03-04T17:48:19.345 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:19.345 DEBUG:boto:Data: 2024-03-04T17:48:19.345 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.345 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.345 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.345 DEBUG:boto:Params: {} 2024-03-04T17:48:19.345 DEBUG:boto:Token: None 2024-03-04T17:48:19.346 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.346 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000730373d983498796-0065e60963-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.351 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.352 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.352 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.352 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:19.352 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:19.353 DEBUG:boto:Method: GET 2024-03-04T17:48:19.353 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:19.353 DEBUG:boto:Data: 2024-03-04T17:48:19.353 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.353 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:19.353 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.353 DEBUG:boto:Params: {} 2024-03-04T17:48:19.353 DEBUG:boto:Token: None 2024-03-04T17:48:19.353 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.354 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f84e4e95d8e6c72-0065e60963-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.359 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.359 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.359 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.360 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.360 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.360 DEBUG:boto:Data: 2024-03-04T17:48:19.360 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.360 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.360 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.360 DEBUG:boto:Params: {} 2024-03-04T17:48:19.361 DEBUG:boto:Token: None 2024-03-04T17:48:19.361 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.361 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NdjpTU7A3aUgrLl+E7qlpTZnoNM= 2024-03-04T17:48:19.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NdjpTU7A3aUgrLl+E7qlpTZnoNM=', 'Content-Length': '0'} 2024-03-04T17:48:19.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004839dea4fdaa1b60-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.365 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.365 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.365 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.366 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.366 DEBUG:boto:Data: 2024-03-04T17:48:19.366 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.366 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:19.366 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.366 DEBUG:boto:Params: {} 2024-03-04T17:48:19.366 DEBUG:boto:Token: None 2024-03-04T17:48:19.367 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.367 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NdjpTU7A3aUgrLl+E7qlpTZnoNM= 2024-03-04T17:48:19.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NdjpTU7A3aUgrLl+E7qlpTZnoNM=', 'Content-Length': '0'} 2024-03-04T17:48:19.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000416438392fc145a-0065e60963-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:19.371 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.372 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.372 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.372 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.372 DEBUG:boto:Method: GET 2024-03-04T17:48:19.372 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.373 DEBUG:boto:Data: 2024-03-04T17:48:19.373 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.373 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.373 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.373 DEBUG:boto:Params: {} 2024-03-04T17:48:19.374 DEBUG:boto:Token: None 2024-03-04T17:48:19.374 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.374 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GKU9GkZud4rW8SEpCPOV6v5G0pY= 2024-03-04T17:48:19.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GKU9GkZud4rW8SEpCPOV6v5G0pY=', 'Content-Length': '0'} 2024-03-04T17:48:19.377 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038351ccd6d1838da-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.424 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.424 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:19.424 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.425 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.425 DEBUG:boto:Method: GET 2024-03-04T17:48:19.425 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.425 DEBUG:boto:Data: 2024-03-04T17:48:19.425 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.425 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:19.425 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.425 DEBUG:boto:Params: {} 2024-03-04T17:48:19.425 DEBUG:boto:Token: None 2024-03-04T17:48:19.425 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:19.426 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GKU9GkZud4rW8SEpCPOV6v5G0pY= 2024-03-04T17:48:19.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GKU9GkZud4rW8SEpCPOV6v5G0pY=', 'Content-Length': '0'} 2024-03-04T17:48:19.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8d6df1502479991-0065e60963-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.472 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.472 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.472 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:19.472 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:19.472 DEBUG:boto:Method: GET 2024-03-04T17:48:19.473 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:19.473 DEBUG:boto:Data: 2024-03-04T17:48:19.473 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.473 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.473 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.473 DEBUG:boto:Params: {} 2024-03-04T17:48:19.474 DEBUG:boto:Token: None 2024-03-04T17:48:19.474 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.474 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d67422656b43908-0065e60963-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.479 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.479 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.479 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.480 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:19.480 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:19.480 DEBUG:boto:Method: GET 2024-03-04T17:48:19.480 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:19.480 DEBUG:boto:Data: 2024-03-04T17:48:19.480 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.481 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:19.481 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.481 DEBUG:boto:Params: {} 2024-03-04T17:48:19.481 DEBUG:boto:Token: None 2024-03-04T17:48:19.481 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.482 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c58f871d8dbde55-0065e60963-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.486 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.487 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-20 zones={test-zone3, test-zone1} 2024-03-04T17:48:19.487 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-20 zones={test-zone3, test-zone2} 2024-03-04T17:48:19.487 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.487 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.488 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.488 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:48:19.488 DEBUG:boto:Data: 2024-03-04T17:48:19.488 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.488 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.488 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.488 DEBUG:boto:Params: {} 2024-03-04T17:48:19.488 DEBUG:boto:Token: None 2024-03-04T17:48:19.488 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.488 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:26AHB3RPYod6NQqJyEaLr5xUMqQ= 2024-03-04T17:48:19.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:26AHB3RPYod6NQqJyEaLr5xUMqQ=', 'Content-Length': '0'} 2024-03-04T17:48:19.492 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096f78a5d4f5e3dec-0065e60963-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.493 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.493 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.493 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.493 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:48:19.493 DEBUG:boto:Data: 2024-03-04T17:48:19.493 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.494 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.494 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.494 DEBUG:boto:Params: {} 2024-03-04T17:48:19.494 DEBUG:boto:Token: None 2024-03-04T17:48:19.494 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.495 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:26AHB3RPYod6NQqJyEaLr5xUMqQ= 2024-03-04T17:48:19.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:26AHB3RPYod6NQqJyEaLr5xUMqQ=', 'Content-Length': '0'} 2024-03-04T17:48:19.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddcc16bd0ca4ff46-0065e60963-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.499 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:48:19.499 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.499 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.499 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:19.499 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:19.500 DEBUG:boto:Method: GET 2024-03-04T17:48:19.500 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:19.500 DEBUG:boto:Data: 2024-03-04T17:48:19.500 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.500 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.500 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.500 DEBUG:boto:Params: {} 2024-03-04T17:48:19.501 DEBUG:boto:Token: None 2024-03-04T17:48:19.501 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/?versions 2024-03-04T17:48:19.501 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3mFx2WywyLlXawPpof4btYE13kY= 2024-03-04T17:48:19.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3mFx2WywyLlXawPpof4btYE13kY=', 'Content-Length': '0'} 2024-03-04T17:48:19.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f8654c0fa34fd6d-0065e60963-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.506 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.506 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:19.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:19.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:19.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:19.507 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:48:19.507 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.507 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.507 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:19.507 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:19.507 DEBUG:boto:Method: GET 2024-03-04T17:48:19.507 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:19.507 DEBUG:boto:Data: 2024-03-04T17:48:19.507 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.508 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.508 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.508 DEBUG:boto:Params: {} 2024-03-04T17:48:19.508 DEBUG:boto:Token: None 2024-03-04T17:48:19.508 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/?versions 2024-03-04T17:48:19.508 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3mFx2WywyLlXawPpof4btYE13kY= 2024-03-04T17:48:19.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3mFx2WywyLlXawPpof4btYE13kY=', 'Content-Length': '0'} 2024-03-04T17:48:19.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025ef577370f00c18-0065e60963-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.513 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.513 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:19.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:19.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:19.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:19.514 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.514 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.514 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:19.514 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:19.514 DEBUG:boto:Method: GET 2024-03-04T17:48:19.514 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:19.515 DEBUG:boto:Data: 2024-03-04T17:48:19.515 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.515 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.515 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.515 DEBUG:boto:Params: {} 2024-03-04T17:48:19.515 DEBUG:boto:Token: None 2024-03-04T17:48:19.515 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/?versions 2024-03-04T17:48:19.515 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3mFx2WywyLlXawPpof4btYE13kY= 2024-03-04T17:48:19.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3mFx2WywyLlXawPpof4btYE13kY=', 'Content-Length': '0'} 2024-03-04T17:48:19.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddab9107ded2782d-0065e60963-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.564 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.565 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.565 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.565 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:19.565 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:19.566 DEBUG:boto:Method: GET 2024-03-04T17:48:19.566 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:48:19.566 DEBUG:boto:Data: 2024-03-04T17:48:19.566 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.566 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.566 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.566 DEBUG:boto:Params: {} 2024-03-04T17:48:19.567 DEBUG:boto:Token: None 2024-03-04T17:48:19.567 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/?versions 2024-03-04T17:48:19.567 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3mFx2WywyLlXawPpof4btYE13kY= 2024-03-04T17:48:19.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3mFx2WywyLlXawPpof4btYE13kY=', 'Content-Length': '0'} 2024-03-04T17:48:19.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c0e9a82754bcc6d-0065e60963-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.572 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:19.573 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:19.573 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:19.573 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.573 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.573 DEBUG:boto:Method: GET 2024-03-04T17:48:19.573 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.573 DEBUG:boto:Data: 2024-03-04T17:48:19.573 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.573 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.574 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.574 DEBUG:boto:Params: {} 2024-03-04T17:48:19.574 DEBUG:boto:Token: None 2024-03-04T17:48:19.574 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.574 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6y6y0A1yJBooG5zkkR1PxnMYfWI= 2024-03-04T17:48:19.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6y6y0A1yJBooG5zkkR1PxnMYfWI=', 'Content-Length': '0'} 2024-03-04T17:48:19.577 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a19c5012342b0e29-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.577 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:19.578 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:19.578 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.578 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.578 DEBUG:boto:Method: GET 2024-03-04T17:48:19.578 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.578 DEBUG:boto:Data: 2024-03-04T17:48:19.578 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.578 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.578 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.578 DEBUG:boto:Params: {} 2024-03-04T17:48:19.579 DEBUG:boto:Token: None 2024-03-04T17:48:19.579 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.579 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6y6y0A1yJBooG5zkkR1PxnMYfWI= 2024-03-04T17:48:19.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6y6y0A1yJBooG5zkkR1PxnMYfWI=', 'Content-Length': '0'} 2024-03-04T17:48:19.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003f66dd3e76f3b3e5-0065e60963-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.582 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:19.582 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:19.582 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.582 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.583 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.583 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.583 DEBUG:boto:Data: 2024-03-04T17:48:19.583 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.583 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.583 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.583 DEBUG:boto:Params: {} 2024-03-04T17:48:19.583 DEBUG:boto:Token: None 2024-03-04T17:48:19.583 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.583 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EVSKlD5gdikf54CEeyOorFKTeks= 2024-03-04T17:48:19.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EVSKlD5gdikf54CEeyOorFKTeks=', 'Content-Length': '0'} 2024-03-04T17:48:19.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c4feea53639b8f8b-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.587 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:19.587 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:19.587 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.587 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.587 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.587 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.587 DEBUG:boto:Data: 2024-03-04T17:48:19.587 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.588 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.588 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.588 DEBUG:boto:Params: {} 2024-03-04T17:48:19.588 DEBUG:boto:Token: None 2024-03-04T17:48:19.588 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.588 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EVSKlD5gdikf54CEeyOorFKTeks= 2024-03-04T17:48:19.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EVSKlD5gdikf54CEeyOorFKTeks=', 'Content-Length': '0'} 2024-03-04T17:48:19.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c14401742a6a1e42-0065e60963-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:19.592 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:19.592 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:19.592 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.592 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.592 DEBUG:boto:Method: GET 2024-03-04T17:48:19.592 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.592 DEBUG:boto:Data: 2024-03-04T17:48:19.592 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.592 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.592 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.593 DEBUG:boto:Params: {} 2024-03-04T17:48:19.593 DEBUG:boto:Token: None 2024-03-04T17:48:19.593 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.593 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6y6y0A1yJBooG5zkkR1PxnMYfWI= 2024-03-04T17:48:19.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6y6y0A1yJBooG5zkkR1PxnMYfWI=', 'Content-Length': '0'} 2024-03-04T17:48:19.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000002b0e564cc8500f5-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.596 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:19.596 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:19.596 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.596 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.596 DEBUG:boto:Method: GET 2024-03-04T17:48:19.596 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.596 DEBUG:boto:Data: 2024-03-04T17:48:19.596 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.596 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.597 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.597 DEBUG:boto:Params: {} 2024-03-04T17:48:19.597 DEBUG:boto:Token: None 2024-03-04T17:48:19.597 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.597 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6y6y0A1yJBooG5zkkR1PxnMYfWI= 2024-03-04T17:48:19.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6y6y0A1yJBooG5zkkR1PxnMYfWI=', 'Content-Length': '0'} 2024-03-04T17:48:19.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023d9ad434096066f-0065e60963-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.601 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.601 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.601 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:19.601 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:19.601 DEBUG:boto:Method: GET 2024-03-04T17:48:19.601 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:48:19.601 DEBUG:boto:Data: 2024-03-04T17:48:19.601 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.601 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.601 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.601 DEBUG:boto:Params: {} 2024-03-04T17:48:19.602 DEBUG:boto:Token: None 2024-03-04T17:48:19.602 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.602 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094928eb1716477bf-0065e60963-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.606 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.607 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.607 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.607 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:19.607 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:19.607 DEBUG:boto:Method: GET 2024-03-04T17:48:19.607 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:48:19.607 DEBUG:boto:Data: 2024-03-04T17:48:19.607 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.608 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.608 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.608 DEBUG:boto:Params: {} 2024-03-04T17:48:19.608 DEBUG:boto:Token: None 2024-03-04T17:48:19.608 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.608 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002af6a40da13f03c4-0065e60963-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.656 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.656 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:19.657 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:19.657 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.657 DEBUG:boto:Path: /ppfahn-20/dummy 2024-03-04T17:48:19.657 DEBUG:boto:Data: 2024-03-04T17:48:19.657 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.657 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.657 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.657 DEBUG:boto:Params: {} 2024-03-04T17:48:19.657 DEBUG:boto:Token: None 2024-03-04T17:48:19.658 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/dummy 2024-03-04T17:48:19.658 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:af1+ycnP+7WqGB9Ew2hVqbVKnDs= 2024-03-04T17:48:19.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:af1+ycnP+7WqGB9Ew2hVqbVKnDs=', 'Content-Length': '0'} 2024-03-04T17:48:19.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003b7e147336e9c435-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.664 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:19.664 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:19.664 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.664 DEBUG:boto:Path: /ppfahn-20/dummy 2024-03-04T17:48:19.664 DEBUG:boto:Data: 2024-03-04T17:48:19.664 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.664 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.664 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.664 DEBUG:boto:Params: {} 2024-03-04T17:48:19.665 DEBUG:boto:Token: None 2024-03-04T17:48:19.665 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/dummy 2024-03-04T17:48:19.665 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:af1+ycnP+7WqGB9Ew2hVqbVKnDs= 2024-03-04T17:48:19.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:af1+ycnP+7WqGB9Ew2hVqbVKnDs=', 'Content-Length': '0'} 2024-03-04T17:48:19.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2861253d6255ea7-0065e60963-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:19.669 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:19.669 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:19.669 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.669 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.669 DEBUG:boto:Method: GET 2024-03-04T17:48:19.669 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.669 DEBUG:boto:Data: 2024-03-04T17:48:19.669 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.670 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.670 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.670 DEBUG:boto:Params: {} 2024-03-04T17:48:19.670 DEBUG:boto:Token: None 2024-03-04T17:48:19.670 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.670 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6y6y0A1yJBooG5zkkR1PxnMYfWI= 2024-03-04T17:48:19.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6y6y0A1yJBooG5zkkR1PxnMYfWI=', 'Content-Length': '0'} 2024-03-04T17:48:19.673 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000549f71760d278b81-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.673 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:48:19.673 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:48:19.673 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.673 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.673 DEBUG:boto:Method: GET 2024-03-04T17:48:19.674 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.674 DEBUG:boto:Data: 2024-03-04T17:48:19.674 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.674 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.674 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.674 DEBUG:boto:Params: {} 2024-03-04T17:48:19.674 DEBUG:boto:Token: None 2024-03-04T17:48:19.674 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:48:19.674 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6y6y0A1yJBooG5zkkR1PxnMYfWI= 2024-03-04T17:48:19.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6y6y0A1yJBooG5zkkR1PxnMYfWI=', 'Content-Length': '0'} 2024-03-04T17:48:19.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000039fd784c77d4aa64-0065e60963-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.677 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.678 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.678 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:19.678 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:19.678 DEBUG:boto:Method: GET 2024-03-04T17:48:19.678 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:48:19.678 DEBUG:boto:Data: 2024-03-04T17:48:19.678 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.678 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.678 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.678 DEBUG:boto:Params: {} 2024-03-04T17:48:19.678 DEBUG:boto:Token: None 2024-03-04T17:48:19.679 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.679 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d2069f9e199df61-0065e60963-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.724 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.725 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.725 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.725 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:19.725 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:48:19.725 DEBUG:boto:Method: GET 2024-03-04T17:48:19.725 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:48:19.725 DEBUG:boto:Data: 2024-03-04T17:48:19.725 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.725 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.725 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.726 DEBUG:boto:Params: {} 2024-03-04T17:48:19.726 DEBUG:boto:Token: None 2024-03-04T17:48:19.726 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.726 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c529195185d960c8-0065e60963-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.730 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:19.731 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.731 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.731 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.731 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.731 DEBUG:boto:Method: GET 2024-03-04T17:48:19.731 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.732 DEBUG:boto:Data: 2024-03-04T17:48:19.732 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.732 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.732 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.732 DEBUG:boto:Params: {} 2024-03-04T17:48:19.732 DEBUG:boto:Token: None 2024-03-04T17:48:19.732 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.732 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eLMDE3mUyQDh6S+1YIgu5d3UYh4= 2024-03-04T17:48:19.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eLMDE3mUyQDh6S+1YIgu5d3UYh4=', 'Content-Length': '0'} 2024-03-04T17:48:19.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c262c19bb9a1e153-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.776 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.776 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.776 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.776 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.776 DEBUG:boto:Method: GET 2024-03-04T17:48:19.777 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.777 DEBUG:boto:Data: 2024-03-04T17:48:19.777 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.777 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.777 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.777 DEBUG:boto:Params: {} 2024-03-04T17:48:19.777 DEBUG:boto:Token: None 2024-03-04T17:48:19.777 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.777 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eLMDE3mUyQDh6S+1YIgu5d3UYh4= 2024-03-04T17:48:19.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eLMDE3mUyQDh6S+1YIgu5d3UYh4=', 'Content-Length': '0'} 2024-03-04T17:48:19.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd6f6349c205677c-0065e60963-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.781 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.781 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.781 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.782 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.782 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.782 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.782 DEBUG:boto:Data: 2024-03-04T17:48:19.782 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.782 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.782 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.782 DEBUG:boto:Params: {} 2024-03-04T17:48:19.782 DEBUG:boto:Token: None 2024-03-04T17:48:19.783 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.783 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pm3lYpkr9i0Pk9M42BwM7XI7lns= 2024-03-04T17:48:19.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pm3lYpkr9i0Pk9M42BwM7XI7lns=', 'Content-Length': '0'} 2024-03-04T17:48:19.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b192a3395e0e79b-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.786 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.786 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.786 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.786 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.786 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.786 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.786 DEBUG:boto:Data: 2024-03-04T17:48:19.787 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.787 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.787 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.787 DEBUG:boto:Params: {} 2024-03-04T17:48:19.787 DEBUG:boto:Token: None 2024-03-04T17:48:19.787 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.787 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pm3lYpkr9i0Pk9M42BwM7XI7lns= 2024-03-04T17:48:19.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pm3lYpkr9i0Pk9M42BwM7XI7lns=', 'Content-Length': '0'} 2024-03-04T17:48:19.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f9ee78a946c8845-0065e60963-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:19.791 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.791 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.791 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.791 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.791 DEBUG:boto:Method: GET 2024-03-04T17:48:19.791 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.791 DEBUG:boto:Data: 2024-03-04T17:48:19.791 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.791 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.792 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.792 DEBUG:boto:Params: {} 2024-03-04T17:48:19.792 DEBUG:boto:Token: None 2024-03-04T17:48:19.792 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.792 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eLMDE3mUyQDh6S+1YIgu5d3UYh4= 2024-03-04T17:48:19.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eLMDE3mUyQDh6S+1YIgu5d3UYh4=', 'Content-Length': '0'} 2024-03-04T17:48:19.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057a1ccb1a085604a-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.836 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.836 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.836 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.836 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.837 DEBUG:boto:Method: GET 2024-03-04T17:48:19.837 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.837 DEBUG:boto:Data: 2024-03-04T17:48:19.837 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.837 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.837 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.837 DEBUG:boto:Params: {} 2024-03-04T17:48:19.837 DEBUG:boto:Token: None 2024-03-04T17:48:19.837 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.837 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eLMDE3mUyQDh6S+1YIgu5d3UYh4= 2024-03-04T17:48:19.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eLMDE3mUyQDh6S+1YIgu5d3UYh4=', 'Content-Length': '0'} 2024-03-04T17:48:19.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd007b590145957e-0065e60963-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.884 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.884 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.885 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:19.885 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:19.885 DEBUG:boto:Method: GET 2024-03-04T17:48:19.885 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:19.885 DEBUG:boto:Data: 2024-03-04T17:48:19.885 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.885 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.885 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.885 DEBUG:boto:Params: {} 2024-03-04T17:48:19.886 DEBUG:boto:Token: None 2024-03-04T17:48:19.886 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.886 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d90dd2b4d0ee9b4-0065e60963-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.891 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.891 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:19.891 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:19.891 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:19.891 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:19.892 DEBUG:boto:Method: GET 2024-03-04T17:48:19.892 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:19.892 DEBUG:boto:Data: 2024-03-04T17:48:19.892 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.892 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.892 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.892 DEBUG:boto:Params: {} 2024-03-04T17:48:19.892 DEBUG:boto:Token: None 2024-03-04T17:48:19.892 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/ 2024-03-04T17:48:19.893 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc= 2024-03-04T17:48:19.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nDUy9tWMFk1SssHZQyv4PbbhPnc=', 'Content-Length': '0'} 2024-03-04T17:48:19.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046c72e5f624f53a7-0065e60963-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.897 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:19.897 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.897 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.898 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.898 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.898 DEBUG:boto:Data: 2024-03-04T17:48:19.898 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.898 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.898 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.898 DEBUG:boto:Params: {} 2024-03-04T17:48:19.898 DEBUG:boto:Token: None 2024-03-04T17:48:19.898 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.899 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:35fM4gjGpju5K8G8hzf9EQ6oclw= 2024-03-04T17:48:19.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:35fM4gjGpju5K8G8hzf9EQ6oclw=', 'Content-Length': '0'} 2024-03-04T17:48:19.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f947aaefbb45bcf-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.903 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.903 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.903 DEBUG:boto:Method: HEAD 2024-03-04T17:48:19.903 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.903 DEBUG:boto:Data: 2024-03-04T17:48:19.903 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.903 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.903 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.904 DEBUG:boto:Params: {} 2024-03-04T17:48:19.904 DEBUG:boto:Token: None 2024-03-04T17:48:19.904 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.904 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:35fM4gjGpju5K8G8hzf9EQ6oclw= 2024-03-04T17:48:19.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:35fM4gjGpju5K8G8hzf9EQ6oclw=', 'Content-Length': '0'} 2024-03-04T17:48:19.908 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa41e4db0f2c0b23-0065e60963-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:19.908 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.908 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.908 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.908 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.908 DEBUG:boto:Method: GET 2024-03-04T17:48:19.909 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.909 DEBUG:boto:Data: 2024-03-04T17:48:19.909 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.909 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:19.909 DEBUG:boto:Port: 8001 2024-03-04T17:48:19.909 DEBUG:boto:Params: {} 2024-03-04T17:48:19.909 DEBUG:boto:Token: None 2024-03-04T17:48:19.909 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.909 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eLMDE3mUyQDh6S+1YIgu5d3UYh4= 2024-03-04T17:48:19.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eLMDE3mUyQDh6S+1YIgu5d3UYh4=', 'Content-Length': '0'} 2024-03-04T17:48:19.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cb119d86b6ca7c1-0065e60963-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:19.956 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.956 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:19.956 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.956 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.957 DEBUG:boto:Method: GET 2024-03-04T17:48:19.957 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.957 DEBUG:boto:Data: 2024-03-04T17:48:19.957 DEBUG:boto:Headers: {} 2024-03-04T17:48:19.957 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:19.957 DEBUG:boto:Port: 8000 2024-03-04T17:48:19.957 DEBUG:boto:Params: {} 2024-03-04T17:48:19.957 DEBUG:boto:Token: None 2024-03-04T17:48:19.957 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:19 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:48:19.958 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eLMDE3mUyQDh6S+1YIgu5d3UYh4= 2024-03-04T17:48:19.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:19 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eLMDE3mUyQDh6S+1YIgu5d3UYh4=', 'Content-Length': '0'} 2024-03-04T17:48:19.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aeae177b9459bc90-0065e60963-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.004 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:20.004 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:20.004 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:20.005 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:20.005 DEBUG:boto:Method: GET 2024-03-04T17:48:20.005 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:20.005 DEBUG:boto:Data: 2024-03-04T17:48:20.005 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.005 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.005 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.005 DEBUG:boto:Params: {} 2024-03-04T17:48:20.005 DEBUG:boto:Token: None 2024-03-04T17:48:20.005 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/ 2024-03-04T17:48:20.006 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo= 2024-03-04T17:48:20.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo=', 'Content-Length': '0'} 2024-03-04T17:48:20.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020fde391f4d7e1b1-0065e60964-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.052 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:20.052 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:20.053 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:20.053 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:20.053 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:20.053 DEBUG:boto:Method: GET 2024-03-04T17:48:20.053 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:48:20.053 DEBUG:boto:Data: 2024-03-04T17:48:20.053 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.053 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.053 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.053 DEBUG:boto:Params: {} 2024-03-04T17:48:20.054 DEBUG:boto:Token: None 2024-03-04T17:48:20.054 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/ 2024-03-04T17:48:20.054 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo= 2024-03-04T17:48:20.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo=', 'Content-Length': '0'} 2024-03-04T17:48:20.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000926b4eeabc6972a9-0065e60964-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.100 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:20.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:20.101 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.101 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.101 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.101 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.102 DEBUG:boto:Method: GET 2024-03-04T17:48:20.102 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.102 DEBUG:boto:Data: 2024-03-04T17:48:20.102 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.102 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.102 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.102 DEBUG:boto:Params: {} 2024-03-04T17:48:20.102 DEBUG:boto:Token: None 2024-03-04T17:48:20.102 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.102 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zh5HAsRD+pJYzBvn7LekiqBZHfI= 2024-03-04T17:48:20.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zh5HAsRD+pJYzBvn7LekiqBZHfI=', 'Content-Length': '0'} 2024-03-04T17:48:20.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c7c681ba53e3407-0065e60964-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.106 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.106 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.106 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.106 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.106 DEBUG:boto:Method: GET 2024-03-04T17:48:20.106 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.106 DEBUG:boto:Data: 2024-03-04T17:48:20.106 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.107 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.107 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.107 DEBUG:boto:Params: {} 2024-03-04T17:48:20.107 DEBUG:boto:Token: None 2024-03-04T17:48:20.107 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.107 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zh5HAsRD+pJYzBvn7LekiqBZHfI= 2024-03-04T17:48:20.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zh5HAsRD+pJYzBvn7LekiqBZHfI=', 'Content-Length': '0'} 2024-03-04T17:48:20.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006024cab39e5e1d3f-0065e60964-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.110 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.111 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.111 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.111 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.111 DEBUG:boto:Method: HEAD 2024-03-04T17:48:20.111 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.111 DEBUG:boto:Data: 2024-03-04T17:48:20.111 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.111 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.111 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.111 DEBUG:boto:Params: {} 2024-03-04T17:48:20.112 DEBUG:boto:Token: None 2024-03-04T17:48:20.112 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.112 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7l2BG5Y0yyFm6TwNKYh+G6bC/HU= 2024-03-04T17:48:20.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7l2BG5Y0yyFm6TwNKYh+G6bC/HU=', 'Content-Length': '0'} 2024-03-04T17:48:20.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000472d5639463b6714-0065e60964-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.115 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.115 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.115 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.115 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.116 DEBUG:boto:Method: HEAD 2024-03-04T17:48:20.116 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.116 DEBUG:boto:Data: 2024-03-04T17:48:20.116 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.116 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.116 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.116 DEBUG:boto:Params: {} 2024-03-04T17:48:20.116 DEBUG:boto:Token: None 2024-03-04T17:48:20.116 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.117 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7l2BG5Y0yyFm6TwNKYh+G6bC/HU= 2024-03-04T17:48:20.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7l2BG5Y0yyFm6TwNKYh+G6bC/HU=', 'Content-Length': '0'} 2024-03-04T17:48:20.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b8c935226068dba-0065e60964-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:20.120 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.121 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.121 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.121 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.121 DEBUG:boto:Method: GET 2024-03-04T17:48:20.121 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.121 DEBUG:boto:Data: 2024-03-04T17:48:20.121 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.121 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.121 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.121 DEBUG:boto:Params: {} 2024-03-04T17:48:20.122 DEBUG:boto:Token: None 2024-03-04T17:48:20.122 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.122 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zh5HAsRD+pJYzBvn7LekiqBZHfI= 2024-03-04T17:48:20.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zh5HAsRD+pJYzBvn7LekiqBZHfI=', 'Content-Length': '0'} 2024-03-04T17:48:20.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000887fa854480bfbd5-0065e60964-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.172 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.172 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.172 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.172 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.172 DEBUG:boto:Method: GET 2024-03-04T17:48:20.172 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.173 DEBUG:boto:Data: 2024-03-04T17:48:20.173 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.173 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.173 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.173 DEBUG:boto:Params: {} 2024-03-04T17:48:20.173 DEBUG:boto:Token: None 2024-03-04T17:48:20.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.173 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zh5HAsRD+pJYzBvn7LekiqBZHfI= 2024-03-04T17:48:20.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zh5HAsRD+pJYzBvn7LekiqBZHfI=', 'Content-Length': '0'} 2024-03-04T17:48:20.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009aefb86b74acbc0a-0065e60964-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.177 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:20.177 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:20.177 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:20.177 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:20.177 DEBUG:boto:Method: GET 2024-03-04T17:48:20.177 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:20.177 DEBUG:boto:Data: 2024-03-04T17:48:20.177 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.177 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.178 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.178 DEBUG:boto:Params: {} 2024-03-04T17:48:20.178 DEBUG:boto:Token: None 2024-03-04T17:48:20.178 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/ 2024-03-04T17:48:20.178 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo= 2024-03-04T17:48:20.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo=', 'Content-Length': '0'} 2024-03-04T17:48:20.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf8bceb66933f5eb-0065e60964-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.182 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:20.183 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:20.183 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:20.183 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:20.183 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:20.183 DEBUG:boto:Method: GET 2024-03-04T17:48:20.183 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:20.183 DEBUG:boto:Data: 2024-03-04T17:48:20.183 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.184 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.184 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.184 DEBUG:boto:Params: {} 2024-03-04T17:48:20.184 DEBUG:boto:Token: None 2024-03-04T17:48:20.184 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/ 2024-03-04T17:48:20.184 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo= 2024-03-04T17:48:20.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo=', 'Content-Length': '0'} 2024-03-04T17:48:20.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e98f5bcbdf73114-0065e60964-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.236 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:20.236 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.237 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.237 DEBUG:boto:Method: HEAD 2024-03-04T17:48:20.237 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.237 DEBUG:boto:Data: 2024-03-04T17:48:20.237 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.237 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.237 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.237 DEBUG:boto:Params: {} 2024-03-04T17:48:20.237 DEBUG:boto:Token: None 2024-03-04T17:48:20.238 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.238 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DsnWxu9wQr8jiNGfFLjcQxdsv+M= 2024-03-04T17:48:20.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DsnWxu9wQr8jiNGfFLjcQxdsv+M=', 'Content-Length': '0'} 2024-03-04T17:48:20.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000899e1e2931260b8d-0065e60964-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.242 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.242 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.242 DEBUG:boto:Method: HEAD 2024-03-04T17:48:20.243 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.243 DEBUG:boto:Data: 2024-03-04T17:48:20.243 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.243 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.243 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.243 DEBUG:boto:Params: {} 2024-03-04T17:48:20.243 DEBUG:boto:Token: None 2024-03-04T17:48:20.243 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.243 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DsnWxu9wQr8jiNGfFLjcQxdsv+M= 2024-03-04T17:48:20.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DsnWxu9wQr8jiNGfFLjcQxdsv+M=', 'Content-Length': '0'} 2024-03-04T17:48:20.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029a5e87a6fec90ed-0065e60964-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:20.248 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.248 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.248 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.248 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.248 DEBUG:boto:Method: GET 2024-03-04T17:48:20.248 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.248 DEBUG:boto:Data: 2024-03-04T17:48:20.248 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.248 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.248 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.249 DEBUG:boto:Params: {} 2024-03-04T17:48:20.249 DEBUG:boto:Token: None 2024-03-04T17:48:20.249 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.249 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zh5HAsRD+pJYzBvn7LekiqBZHfI= 2024-03-04T17:48:20.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zh5HAsRD+pJYzBvn7LekiqBZHfI=', 'Content-Length': '0'} 2024-03-04T17:48:20.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b35e1aed011e246-0065e60964-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.296 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.296 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:20.296 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.296 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.296 DEBUG:boto:Method: GET 2024-03-04T17:48:20.297 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.297 DEBUG:boto:Data: 2024-03-04T17:48:20.297 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.297 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.297 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.297 DEBUG:boto:Params: {} 2024-03-04T17:48:20.297 DEBUG:boto:Token: None 2024-03-04T17:48:20.297 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:48:20.297 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zh5HAsRD+pJYzBvn7LekiqBZHfI= 2024-03-04T17:48:20.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zh5HAsRD+pJYzBvn7LekiqBZHfI=', 'Content-Length': '0'} 2024-03-04T17:48:20.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6e3da15d678c316-0065e60964-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.301 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:20.301 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:20.301 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:20.301 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:20.301 DEBUG:boto:Method: GET 2024-03-04T17:48:20.302 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:20.302 DEBUG:boto:Data: 2024-03-04T17:48:20.302 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.302 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.302 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.302 DEBUG:boto:Params: {} 2024-03-04T17:48:20.302 DEBUG:boto:Token: None 2024-03-04T17:48:20.302 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/ 2024-03-04T17:48:20.302 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo= 2024-03-04T17:48:20.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo=', 'Content-Length': '0'} 2024-03-04T17:48:20.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006739f936014a5dbc-0065e60964-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.307 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:20.307 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:20.307 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:20.308 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:20.308 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:20.308 DEBUG:boto:Method: GET 2024-03-04T17:48:20.308 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:48:20.308 DEBUG:boto:Data: 2024-03-04T17:48:20.308 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.308 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.309 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.309 DEBUG:boto:Params: {} 2024-03-04T17:48:20.309 DEBUG:boto:Token: None 2024-03-04T17:48:20.309 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/ 2024-03-04T17:48:20.309 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo= 2024-03-04T17:48:20.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo=', 'Content-Length': '0'} 2024-03-04T17:48:20.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000670941ef856c4fa9-0065e60964-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.314 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:20.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:20.314 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.314 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.314 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.314 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.314 DEBUG:boto:Method: GET 2024-03-04T17:48:20.315 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.315 DEBUG:boto:Data: 2024-03-04T17:48:20.315 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.315 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.315 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.315 DEBUG:boto:Params: {} 2024-03-04T17:48:20.315 DEBUG:boto:Token: None 2024-03-04T17:48:20.315 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.315 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Flv3PkYqDoIdR7HQ2LLKn+kV77M= 2024-03-04T17:48:20.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Flv3PkYqDoIdR7HQ2LLKn+kV77M=', 'Content-Length': '0'} 2024-03-04T17:48:20.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdfeb30c4142fdc8-0065e60964-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.319 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.319 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.319 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.319 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.319 DEBUG:boto:Method: GET 2024-03-04T17:48:20.319 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.319 DEBUG:boto:Data: 2024-03-04T17:48:20.320 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.320 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.320 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.320 DEBUG:boto:Params: {} 2024-03-04T17:48:20.320 DEBUG:boto:Token: None 2024-03-04T17:48:20.320 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.320 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Flv3PkYqDoIdR7HQ2LLKn+kV77M= 2024-03-04T17:48:20.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Flv3PkYqDoIdR7HQ2LLKn+kV77M=', 'Content-Length': '0'} 2024-03-04T17:48:20.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049c50baca7eff7a0-0065e60964-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.368 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.368 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.368 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.368 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.368 DEBUG:boto:Method: HEAD 2024-03-04T17:48:20.369 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.369 DEBUG:boto:Data: 2024-03-04T17:48:20.369 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.369 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.369 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.369 DEBUG:boto:Params: {} 2024-03-04T17:48:20.369 DEBUG:boto:Token: None 2024-03-04T17:48:20.369 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.370 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ao7PEkrTGBPuHKj6ThQuUqnTmk= 2024-03-04T17:48:20.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ao7PEkrTGBPuHKj6ThQuUqnTmk=', 'Content-Length': '0'} 2024-03-04T17:48:20.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f41416dad58f7fa-0065e60964-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.373 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.373 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.373 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.374 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.374 DEBUG:boto:Method: HEAD 2024-03-04T17:48:20.374 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.374 DEBUG:boto:Data: 2024-03-04T17:48:20.374 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.374 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.375 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.375 DEBUG:boto:Params: {} 2024-03-04T17:48:20.375 DEBUG:boto:Token: None 2024-03-04T17:48:20.375 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.375 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ao7PEkrTGBPuHKj6ThQuUqnTmk= 2024-03-04T17:48:20.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ao7PEkrTGBPuHKj6ThQuUqnTmk=', 'Content-Length': '0'} 2024-03-04T17:48:20.378 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000384e3c96cf9f85b3-0065e60964-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:20.379 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.379 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.379 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.379 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.379 DEBUG:boto:Method: GET 2024-03-04T17:48:20.379 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.379 DEBUG:boto:Data: 2024-03-04T17:48:20.380 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.380 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.380 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.380 DEBUG:boto:Params: {} 2024-03-04T17:48:20.380 DEBUG:boto:Token: None 2024-03-04T17:48:20.380 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.380 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Flv3PkYqDoIdR7HQ2LLKn+kV77M= 2024-03-04T17:48:20.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Flv3PkYqDoIdR7HQ2LLKn+kV77M=', 'Content-Length': '0'} 2024-03-04T17:48:20.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057de2bd296a6c312-0065e60964-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.428 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.428 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.428 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.428 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.429 DEBUG:boto:Method: GET 2024-03-04T17:48:20.429 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.429 DEBUG:boto:Data: 2024-03-04T17:48:20.429 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.429 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.429 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.429 DEBUG:boto:Params: {} 2024-03-04T17:48:20.429 DEBUG:boto:Token: None 2024-03-04T17:48:20.429 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.430 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Flv3PkYqDoIdR7HQ2LLKn+kV77M= 2024-03-04T17:48:20.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Flv3PkYqDoIdR7HQ2LLKn+kV77M=', 'Content-Length': '0'} 2024-03-04T17:48:20.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d1f215659104c9e-0065e60964-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.480 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:20.480 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:20.480 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:20.481 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:20.481 DEBUG:boto:Method: GET 2024-03-04T17:48:20.481 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:20.481 DEBUG:boto:Data: 2024-03-04T17:48:20.481 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.481 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.481 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.481 DEBUG:boto:Params: {} 2024-03-04T17:48:20.481 DEBUG:boto:Token: None 2024-03-04T17:48:20.482 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/ 2024-03-04T17:48:20.482 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo= 2024-03-04T17:48:20.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo=', 'Content-Length': '0'} 2024-03-04T17:48:20.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b18ad8f55f3ed9b-0065e60964-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.487 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:20.487 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:20.488 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:20.488 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:20.488 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:20.488 DEBUG:boto:Method: GET 2024-03-04T17:48:20.488 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:20.488 DEBUG:boto:Data: 2024-03-04T17:48:20.488 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.488 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.488 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.488 DEBUG:boto:Params: {} 2024-03-04T17:48:20.489 DEBUG:boto:Token: None 2024-03-04T17:48:20.489 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/ 2024-03-04T17:48:20.489 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo= 2024-03-04T17:48:20.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo=', 'Content-Length': '0'} 2024-03-04T17:48:20.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be2e5c4732a2136f-0065e60964-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.493 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:20.494 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.494 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.494 DEBUG:boto:Method: HEAD 2024-03-04T17:48:20.494 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.494 DEBUG:boto:Data: 2024-03-04T17:48:20.494 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.494 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.494 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.494 DEBUG:boto:Params: {} 2024-03-04T17:48:20.494 DEBUG:boto:Token: None 2024-03-04T17:48:20.495 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.495 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SyzVTSsE9zl1TnoQQMqDD7ZcttE= 2024-03-04T17:48:20.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SyzVTSsE9zl1TnoQQMqDD7ZcttE=', 'Content-Length': '0'} 2024-03-04T17:48:20.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049f1b3ae0ed14cbd-0065e60964-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.499 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.499 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.499 DEBUG:boto:Method: HEAD 2024-03-04T17:48:20.499 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.499 DEBUG:boto:Data: 2024-03-04T17:48:20.499 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.499 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.499 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.499 DEBUG:boto:Params: {} 2024-03-04T17:48:20.499 DEBUG:boto:Token: None 2024-03-04T17:48:20.500 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.500 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SyzVTSsE9zl1TnoQQMqDD7ZcttE= 2024-03-04T17:48:20.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SyzVTSsE9zl1TnoQQMqDD7ZcttE=', 'Content-Length': '0'} 2024-03-04T17:48:20.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcfaabc5a65f6c64-0065e60964-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:20.504 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.504 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.504 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.504 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.504 DEBUG:boto:Method: GET 2024-03-04T17:48:20.504 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.504 DEBUG:boto:Data: 2024-03-04T17:48:20.504 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.504 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.504 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.504 DEBUG:boto:Params: {} 2024-03-04T17:48:20.505 DEBUG:boto:Token: None 2024-03-04T17:48:20.505 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.505 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Flv3PkYqDoIdR7HQ2LLKn+kV77M= 2024-03-04T17:48:20.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Flv3PkYqDoIdR7HQ2LLKn+kV77M=', 'Content-Length': '0'} 2024-03-04T17:48:20.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000227a5c4a949973b2-0065e60964-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.508 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.508 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:20.508 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.508 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.508 DEBUG:boto:Method: GET 2024-03-04T17:48:20.508 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.509 DEBUG:boto:Data: 2024-03-04T17:48:20.509 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.509 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.509 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.509 DEBUG:boto:Params: {} 2024-03-04T17:48:20.509 DEBUG:boto:Token: None 2024-03-04T17:48:20.509 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:48:20.509 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Flv3PkYqDoIdR7HQ2LLKn+kV77M= 2024-03-04T17:48:20.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Flv3PkYqDoIdR7HQ2LLKn+kV77M=', 'Content-Length': '0'} 2024-03-04T17:48:20.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a5ea982101be8de-0065e60964-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.556 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:20.556 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:20.556 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:20.557 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:20.557 DEBUG:boto:Method: GET 2024-03-04T17:48:20.557 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:20.557 DEBUG:boto:Data: 2024-03-04T17:48:20.557 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.557 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:20.557 DEBUG:boto:Port: 8001 2024-03-04T17:48:20.557 DEBUG:boto:Params: {} 2024-03-04T17:48:20.557 DEBUG:boto:Token: None 2024-03-04T17:48:20.557 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/ 2024-03-04T17:48:20.558 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo= 2024-03-04T17:48:20.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo=', 'Content-Length': '0'} 2024-03-04T17:48:20.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc1f458be1a49337-0065e60964-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.563 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:20.563 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:20.563 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:20.563 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:20.563 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:20.563 DEBUG:boto:Method: GET 2024-03-04T17:48:20.564 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:48:20.564 DEBUG:boto:Data: 2024-03-04T17:48:20.564 DEBUG:boto:Headers: {} 2024-03-04T17:48:20.564 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:20.564 DEBUG:boto:Port: 8000 2024-03-04T17:48:20.564 DEBUG:boto:Params: {} 2024-03-04T17:48:20.564 DEBUG:boto:Token: None 2024-03-04T17:48:20.564 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:20 GMT /ppfahn-20/ 2024-03-04T17:48:20.564 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo= 2024-03-04T17:48:20.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:20 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IMPJsVy8FD66T4vYsCH5IuSMZGo=', 'Content-Length': '0'} 2024-03-04T17:48:20.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000753f5142cae486e0-0065e60964-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:20.568 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:20.569 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-20 zones={test-zone3, test-zone2} 2024-03-04T17:48:20.569 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-21 2024-03-04T17:48:20.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:20.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:20.569 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:20.619 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:20.619 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:20.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:20.684+0000 7efff3369980 1 bucket sync caught up with source: 2024-03-04T17:48:20.685 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000010.56.6, , 00000000003.35.10, , 00000000008.22.10, , , , , ] 2024-03-04T17:48:20.686 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000010.56.6, , 00000000003.35.10, , 00000000008.22.10, , , , , ] 2024-03-04T17:48:20.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:20.684+0000 7efff3369980 0 bucket checkpoint complete 2024-03-04T17:48:20.707 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-21 2024-03-04T17:48:20.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:20.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:20.708 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:20.758 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:20.758 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:20.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:20.821+0000 7f38be81e980 1 bucket sync caught up with source: 2024-03-04T17:48:20.818 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000010.56.6, , 00000000003.35.10, , 00000000008.22.10, , , , , ] 2024-03-04T17:48:20.818 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000010.56.6, , 00000000003.35.10, , 00000000008.22.10, , , , , ] 2024-03-04T17:48:20.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:20.821+0000 7f38be81e980 0 bucket checkpoint complete 2024-03-04T17:48:20.839 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-21 2024-03-04T17:48:20.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:20.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:20.840 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:20.880 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:20.881 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:20.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:20.941+0000 7f2734a52980 1 bucket sync caught up with source: 2024-03-04T17:48:20.940 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.42.10, , 00000000010.49.6, , 00000000008.32.10, , , , , ] 2024-03-04T17:48:20.940 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.42.10, , 00000000010.49.6, , 00000000008.32.10, , , , , ] 2024-03-04T17:48:20.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:20.941+0000 7f2734a52980 0 bucket checkpoint complete 2024-03-04T17:48:20.962 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-21 2024-03-04T17:48:20.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:20.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:20.962 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:21.046 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:21.046 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:21.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:21.133+0000 7f61247fd980 1 bucket sync caught up with source: 2024-03-04T17:48:21.129 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.42.10, , 00000000010.49.6, , 00000000008.32.10, , , , , ] 2024-03-04T17:48:21.129 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.42.10, , 00000000010.49.6, , 00000000008.32.10, , , , , ] 2024-03-04T17:48:21.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:21.133+0000 7f61247fd980 0 bucket checkpoint complete 2024-03-04T17:48:21.149 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-21 2024-03-04T17:48:21.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:21.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:21.150 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:21.255 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:21.255 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:21.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:21.313+0000 7f1f7b0e6980 1 bucket sync caught up with source: 2024-03-04T17:48:21.312 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.107.10, , 00000000003.68.10, , 00000000016.91.10, , , , , ] 2024-03-04T17:48:21.312 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.107.10, , 00000000003.68.10, , 00000000016.91.10, , , , , ] 2024-03-04T17:48:21.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:21.313+0000 7f1f7b0e6980 0 bucket checkpoint complete 2024-03-04T17:48:21.332 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-21 2024-03-04T17:48:21.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:21.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:21.333 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:21.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:21.444+0000 7f235ac31980 1 bucket sync caught up with source: 2024-03-04T17:48:21.445 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000003.107.10, , 00000000003.68.10, , 00000000016.91.10, , , , , ] 2024-03-04T17:48:21.446 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000003.107.10, , 00000000003.68.10, , 00000000016.91.10, , , , , ] 2024-03-04T17:48:21.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:21.444+0000 7f235ac31980 0 bucket checkpoint complete 2024-03-04T17:48:21.468 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-21 zones={test-zone2, test-zone1} 2024-03-04T17:48:21.468 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.468 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.468 DEBUG:boto:Method: HEAD 2024-03-04T17:48:21.468 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:48:21.468 DEBUG:boto:Data: 2024-03-04T17:48:21.468 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.468 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.468 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.469 DEBUG:boto:Params: {} 2024-03-04T17:48:21.469 DEBUG:boto:Token: None 2024-03-04T17:48:21.469 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/ 2024-03-04T17:48:21.469 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qguCO+YA/ARtGvo6T1616W0WQgM= 2024-03-04T17:48:21.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qguCO+YA/ARtGvo6T1616W0WQgM=', 'Content-Length': '0'} 2024-03-04T17:48:21.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2471fff6df2c901-0065e60965-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.471 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.472 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.472 DEBUG:boto:Method: HEAD 2024-03-04T17:48:21.472 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:48:21.472 DEBUG:boto:Data: 2024-03-04T17:48:21.472 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.472 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.472 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.472 DEBUG:boto:Params: {} 2024-03-04T17:48:21.472 DEBUG:boto:Token: None 2024-03-04T17:48:21.472 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/ 2024-03-04T17:48:21.473 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qguCO+YA/ARtGvo6T1616W0WQgM= 2024-03-04T17:48:21.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qguCO+YA/ARtGvo6T1616W0WQgM=', 'Content-Length': '0'} 2024-03-04T17:48:21.477 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f275d99de4708c82-0065e60965-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.477 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:48:21.477 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.477 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.477 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:21.477 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:21.477 DEBUG:boto:Method: GET 2024-03-04T17:48:21.477 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:21.477 DEBUG:boto:Data: 2024-03-04T17:48:21.477 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.478 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.478 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.478 DEBUG:boto:Params: {} 2024-03-04T17:48:21.478 DEBUG:boto:Token: None 2024-03-04T17:48:21.478 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/?versions 2024-03-04T17:48:21.478 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ahmt5sV7ozot3+nCBS/oGHFmwBI= 2024-03-04T17:48:21.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ahmt5sV7ozot3+nCBS/oGHFmwBI=', 'Content-Length': '0'} 2024-03-04T17:48:21.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4dd38355bb91b35-0065e60965-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.480 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:21.481 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:21.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:21.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:21.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:21.481 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:48:21.482 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.482 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.482 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:21.482 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:21.482 DEBUG:boto:Method: GET 2024-03-04T17:48:21.482 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:21.482 DEBUG:boto:Data: 2024-03-04T17:48:21.482 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.482 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.482 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.482 DEBUG:boto:Params: {} 2024-03-04T17:48:21.482 DEBUG:boto:Token: None 2024-03-04T17:48:21.483 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/?versions 2024-03-04T17:48:21.483 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ahmt5sV7ozot3+nCBS/oGHFmwBI= 2024-03-04T17:48:21.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ahmt5sV7ozot3+nCBS/oGHFmwBI=', 'Content-Length': '0'} 2024-03-04T17:48:21.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000390eda0917de5f6c-0065e60965-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.487 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:21.488 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:21.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:21.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:21.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:21.488 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.488 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.488 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:21.488 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:21.489 DEBUG:boto:Method: GET 2024-03-04T17:48:21.489 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:21.489 DEBUG:boto:Data: 2024-03-04T17:48:21.489 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.489 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.489 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.489 DEBUG:boto:Params: {} 2024-03-04T17:48:21.489 DEBUG:boto:Token: None 2024-03-04T17:48:21.489 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/?versions 2024-03-04T17:48:21.489 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ahmt5sV7ozot3+nCBS/oGHFmwBI= 2024-03-04T17:48:21.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ahmt5sV7ozot3+nCBS/oGHFmwBI=', 'Content-Length': '0'} 2024-03-04T17:48:21.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d40c4f8518437570-0065e60965-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.492 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:21.492 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.493 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.493 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:21.493 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:21.493 DEBUG:boto:Method: GET 2024-03-04T17:48:21.493 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:21.493 DEBUG:boto:Data: 2024-03-04T17:48:21.493 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.493 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.493 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.493 DEBUG:boto:Params: {} 2024-03-04T17:48:21.494 DEBUG:boto:Token: None 2024-03-04T17:48:21.494 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/?versions 2024-03-04T17:48:21.494 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ahmt5sV7ozot3+nCBS/oGHFmwBI= 2024-03-04T17:48:21.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ahmt5sV7ozot3+nCBS/oGHFmwBI=', 'Content-Length': '0'} 2024-03-04T17:48:21.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021529f11e9489508-0065e60965-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.498 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:21.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:21.499 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:21.499 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:21.499 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.499 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.499 DEBUG:boto:Method: GET 2024-03-04T17:48:21.499 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.500 DEBUG:boto:Data: 2024-03-04T17:48:21.500 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.500 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.500 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.500 DEBUG:boto:Params: {} 2024-03-04T17:48:21.500 DEBUG:boto:Token: None 2024-03-04T17:48:21.500 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.500 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+s1v036LvjQ8URdwxVhW4FReZgQ= 2024-03-04T17:48:21.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+s1v036LvjQ8URdwxVhW4FReZgQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c0d88f488ff1299-0065e60965-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.502 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:21.502 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:21.502 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.502 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.503 DEBUG:boto:Method: GET 2024-03-04T17:48:21.503 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.503 DEBUG:boto:Data: 2024-03-04T17:48:21.503 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.503 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.503 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.503 DEBUG:boto:Params: {} 2024-03-04T17:48:21.503 DEBUG:boto:Token: None 2024-03-04T17:48:21.503 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.503 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+s1v036LvjQ8URdwxVhW4FReZgQ= 2024-03-04T17:48:21.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+s1v036LvjQ8URdwxVhW4FReZgQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.507 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1a6d9c7bbaf0f3a-0065e60965-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.508 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:21.508 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:21.508 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.508 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.508 DEBUG:boto:Method: HEAD 2024-03-04T17:48:21.508 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.508 DEBUG:boto:Data: 2024-03-04T17:48:21.508 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.508 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.508 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.509 DEBUG:boto:Params: {} 2024-03-04T17:48:21.509 DEBUG:boto:Token: None 2024-03-04T17:48:21.509 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.509 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KOGk2YgQ4MK3V0R44WjDz54/RRM= 2024-03-04T17:48:21.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KOGk2YgQ4MK3V0R44WjDz54/RRM=', 'Content-Length': '0'} 2024-03-04T17:48:21.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a78fd17707c7baf-0065e60965-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.511 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:21.511 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:21.511 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.511 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.511 DEBUG:boto:Method: HEAD 2024-03-04T17:48:21.512 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.512 DEBUG:boto:Data: 2024-03-04T17:48:21.512 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.512 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.512 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.512 DEBUG:boto:Params: {} 2024-03-04T17:48:21.512 DEBUG:boto:Token: None 2024-03-04T17:48:21.512 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.512 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KOGk2YgQ4MK3V0R44WjDz54/RRM= 2024-03-04T17:48:21.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KOGk2YgQ4MK3V0R44WjDz54/RRM=', 'Content-Length': '0'} 2024-03-04T17:48:21.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007cdd74156a6c5709-0065e60965-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:21.515 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:21.515 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:21.516 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.516 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.516 DEBUG:boto:Method: GET 2024-03-04T17:48:21.516 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.516 DEBUG:boto:Data: 2024-03-04T17:48:21.516 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.516 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.516 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.516 DEBUG:boto:Params: {} 2024-03-04T17:48:21.516 DEBUG:boto:Token: None 2024-03-04T17:48:21.517 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.517 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+s1v036LvjQ8URdwxVhW4FReZgQ= 2024-03-04T17:48:21.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+s1v036LvjQ8URdwxVhW4FReZgQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f782cb99fa027888-0065e60965-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.519 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:21.519 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:21.519 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.519 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.519 DEBUG:boto:Method: GET 2024-03-04T17:48:21.519 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.519 DEBUG:boto:Data: 2024-03-04T17:48:21.519 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.519 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.519 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.520 DEBUG:boto:Params: {} 2024-03-04T17:48:21.520 DEBUG:boto:Token: None 2024-03-04T17:48:21.520 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.520 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+s1v036LvjQ8URdwxVhW4FReZgQ= 2024-03-04T17:48:21.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+s1v036LvjQ8URdwxVhW4FReZgQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.523 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca844fbabf0edba0-0065e60965-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.523 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.523 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.523 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:21.523 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:21.524 DEBUG:boto:Method: GET 2024-03-04T17:48:21.524 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:48:21.524 DEBUG:boto:Data: 2024-03-04T17:48:21.524 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.524 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.524 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.524 DEBUG:boto:Params: {} 2024-03-04T17:48:21.524 DEBUG:boto:Token: None 2024-03-04T17:48:21.524 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/ 2024-03-04T17:48:21.524 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ= 2024-03-04T17:48:21.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e11d81aeabe780fc-0065e60965-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.572 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:21.573 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.573 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.573 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:21.573 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:21.573 DEBUG:boto:Method: GET 2024-03-04T17:48:21.573 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:48:21.573 DEBUG:boto:Data: 2024-03-04T17:48:21.573 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.573 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.573 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.573 DEBUG:boto:Params: {} 2024-03-04T17:48:21.574 DEBUG:boto:Token: None 2024-03-04T17:48:21.574 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/ 2024-03-04T17:48:21.574 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ= 2024-03-04T17:48:21.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007de3aab5642c0256-0065e60965-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.578 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:21.579 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:21.579 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:21.579 DEBUG:boto:Method: HEAD 2024-03-04T17:48:21.579 DEBUG:boto:Path: /ppfahn-21/dummy 2024-03-04T17:48:21.579 DEBUG:boto:Data: 2024-03-04T17:48:21.579 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.579 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.579 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.579 DEBUG:boto:Params: {} 2024-03-04T17:48:21.580 DEBUG:boto:Token: None 2024-03-04T17:48:21.580 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/dummy 2024-03-04T17:48:21.580 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xHBeq542aJDmF2/bLK96Qlhl3q8= 2024-03-04T17:48:21.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xHBeq542aJDmF2/bLK96Qlhl3q8=', 'Content-Length': '0'} 2024-03-04T17:48:21.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005fc7c411e9154c0b-0065e60965-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.584 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:21.584 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:21.584 DEBUG:boto:Method: HEAD 2024-03-04T17:48:21.584 DEBUG:boto:Path: /ppfahn-21/dummy 2024-03-04T17:48:21.584 DEBUG:boto:Data: 2024-03-04T17:48:21.584 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.584 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.584 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.585 DEBUG:boto:Params: {} 2024-03-04T17:48:21.585 DEBUG:boto:Token: None 2024-03-04T17:48:21.585 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/dummy 2024-03-04T17:48:21.585 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xHBeq542aJDmF2/bLK96Qlhl3q8= 2024-03-04T17:48:21.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xHBeq542aJDmF2/bLK96Qlhl3q8=', 'Content-Length': '0'} 2024-03-04T17:48:21.588 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000192e8dc59a016711-0065e60965-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:21.589 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:21.589 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:21.589 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.589 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.589 DEBUG:boto:Method: GET 2024-03-04T17:48:21.589 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.589 DEBUG:boto:Data: 2024-03-04T17:48:21.589 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.590 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.590 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.590 DEBUG:boto:Params: {} 2024-03-04T17:48:21.590 DEBUG:boto:Token: None 2024-03-04T17:48:21.590 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.590 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+s1v036LvjQ8URdwxVhW4FReZgQ= 2024-03-04T17:48:21.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+s1v036LvjQ8URdwxVhW4FReZgQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.593 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d51e079df8bbe2d-0065e60965-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.593 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:21.594 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:21.594 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.594 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.594 DEBUG:boto:Method: GET 2024-03-04T17:48:21.594 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.594 DEBUG:boto:Data: 2024-03-04T17:48:21.595 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.595 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.595 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.595 DEBUG:boto:Params: {} 2024-03-04T17:48:21.595 DEBUG:boto:Token: None 2024-03-04T17:48:21.595 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:21.595 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+s1v036LvjQ8URdwxVhW4FReZgQ= 2024-03-04T17:48:21.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+s1v036LvjQ8URdwxVhW4FReZgQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000631c6e64b2b70474-0065e60965-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.598 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.598 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.598 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:21.598 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:21.599 DEBUG:boto:Method: GET 2024-03-04T17:48:21.599 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:48:21.599 DEBUG:boto:Data: 2024-03-04T17:48:21.599 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.599 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.599 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.599 DEBUG:boto:Params: {} 2024-03-04T17:48:21.599 DEBUG:boto:Token: None 2024-03-04T17:48:21.599 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/ 2024-03-04T17:48:21.599 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ= 2024-03-04T17:48:21.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079afb8e01867d5cf-0065e60965-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.605 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:21.605 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.605 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.605 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:21.605 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:21.605 DEBUG:boto:Method: GET 2024-03-04T17:48:21.605 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:48:21.606 DEBUG:boto:Data: 2024-03-04T17:48:21.606 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.606 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.606 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.606 DEBUG:boto:Params: {} 2024-03-04T17:48:21.606 DEBUG:boto:Token: None 2024-03-04T17:48:21.607 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/ 2024-03-04T17:48:21.607 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ= 2024-03-04T17:48:21.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b61f30789ad7467-0065e60965-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.656 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:21.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:21.657 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.657 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.657 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.657 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.657 DEBUG:boto:Method: GET 2024-03-04T17:48:21.657 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.657 DEBUG:boto:Data: 2024-03-04T17:48:21.657 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.657 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.657 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.658 DEBUG:boto:Params: {} 2024-03-04T17:48:21.658 DEBUG:boto:Token: None 2024-03-04T17:48:21.658 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.658 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6cah3TuBlKPSiIvOfSrSMLsOpyY= 2024-03-04T17:48:21.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6cah3TuBlKPSiIvOfSrSMLsOpyY=', 'Content-Length': '0'} 2024-03-04T17:48:21.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e8bf33351e7f7f5-0065e60965-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.704 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.704 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.704 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.704 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.704 DEBUG:boto:Method: GET 2024-03-04T17:48:21.704 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.705 DEBUG:boto:Data: 2024-03-04T17:48:21.705 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.705 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.705 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.705 DEBUG:boto:Params: {} 2024-03-04T17:48:21.705 DEBUG:boto:Token: None 2024-03-04T17:48:21.705 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.705 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6cah3TuBlKPSiIvOfSrSMLsOpyY= 2024-03-04T17:48:21.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6cah3TuBlKPSiIvOfSrSMLsOpyY=', 'Content-Length': '0'} 2024-03-04T17:48:21.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c768b07e36bb458-0065e60965-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.709 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.709 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.709 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.709 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.709 DEBUG:boto:Method: HEAD 2024-03-04T17:48:21.709 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.709 DEBUG:boto:Data: 2024-03-04T17:48:21.709 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.709 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.709 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.709 DEBUG:boto:Params: {} 2024-03-04T17:48:21.710 DEBUG:boto:Token: None 2024-03-04T17:48:21.710 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.710 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eHybPoV2AdAszV4gHf3I0YyuH4w= 2024-03-04T17:48:21.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eHybPoV2AdAszV4gHf3I0YyuH4w=', 'Content-Length': '0'} 2024-03-04T17:48:21.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c4bb5fe0c054cbf-0065e60965-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.715 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.715 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.716 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.716 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.716 DEBUG:boto:Method: HEAD 2024-03-04T17:48:21.716 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.716 DEBUG:boto:Data: 2024-03-04T17:48:21.716 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.716 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.716 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.716 DEBUG:boto:Params: {} 2024-03-04T17:48:21.716 DEBUG:boto:Token: None 2024-03-04T17:48:21.717 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.717 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eHybPoV2AdAszV4gHf3I0YyuH4w= 2024-03-04T17:48:21.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eHybPoV2AdAszV4gHf3I0YyuH4w=', 'Content-Length': '0'} 2024-03-04T17:48:21.719 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0b6c3d9db109197-0065e60965-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:21.720 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.720 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.720 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.720 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.720 DEBUG:boto:Method: GET 2024-03-04T17:48:21.720 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.720 DEBUG:boto:Data: 2024-03-04T17:48:21.720 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.720 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.720 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.720 DEBUG:boto:Params: {} 2024-03-04T17:48:21.721 DEBUG:boto:Token: None 2024-03-04T17:48:21.721 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.721 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6cah3TuBlKPSiIvOfSrSMLsOpyY= 2024-03-04T17:48:21.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6cah3TuBlKPSiIvOfSrSMLsOpyY=', 'Content-Length': '0'} 2024-03-04T17:48:21.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfc8a7b3ddd50c10-0065e60965-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.768 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.768 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.768 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.768 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.768 DEBUG:boto:Method: GET 2024-03-04T17:48:21.769 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.769 DEBUG:boto:Data: 2024-03-04T17:48:21.769 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.769 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.769 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.769 DEBUG:boto:Params: {} 2024-03-04T17:48:21.769 DEBUG:boto:Token: None 2024-03-04T17:48:21.769 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.769 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6cah3TuBlKPSiIvOfSrSMLsOpyY= 2024-03-04T17:48:21.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6cah3TuBlKPSiIvOfSrSMLsOpyY=', 'Content-Length': '0'} 2024-03-04T17:48:21.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e4f2cb4e6c43e1e-0065e60965-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.773 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.773 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.773 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:21.773 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:21.773 DEBUG:boto:Method: GET 2024-03-04T17:48:21.773 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:21.773 DEBUG:boto:Data: 2024-03-04T17:48:21.773 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.773 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.774 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.774 DEBUG:boto:Params: {} 2024-03-04T17:48:21.774 DEBUG:boto:Token: None 2024-03-04T17:48:21.774 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/ 2024-03-04T17:48:21.774 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ= 2024-03-04T17:48:21.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6429d82a5de06f3-0065e60965-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.779 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:21.779 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.779 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.779 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:21.779 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:21.780 DEBUG:boto:Method: GET 2024-03-04T17:48:21.780 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:21.780 DEBUG:boto:Data: 2024-03-04T17:48:21.780 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.780 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.780 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.780 DEBUG:boto:Params: {} 2024-03-04T17:48:21.780 DEBUG:boto:Token: None 2024-03-04T17:48:21.780 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/ 2024-03-04T17:48:21.781 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ= 2024-03-04T17:48:21.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a035edf2e85c911e-0065e60965-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.785 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:21.785 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.786 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.786 DEBUG:boto:Method: HEAD 2024-03-04T17:48:21.786 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.786 DEBUG:boto:Data: 2024-03-04T17:48:21.786 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.786 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.786 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.786 DEBUG:boto:Params: {} 2024-03-04T17:48:21.786 DEBUG:boto:Token: None 2024-03-04T17:48:21.787 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.787 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kmtJ1xpB5ZWpZGp0TaXySTayTgY= 2024-03-04T17:48:21.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kmtJ1xpB5ZWpZGp0TaXySTayTgY=', 'Content-Length': '0'} 2024-03-04T17:48:21.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbf3adfc867c550f-0065e60965-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.791 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.791 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.791 DEBUG:boto:Method: HEAD 2024-03-04T17:48:21.791 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.791 DEBUG:boto:Data: 2024-03-04T17:48:21.791 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.791 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.791 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.792 DEBUG:boto:Params: {} 2024-03-04T17:48:21.792 DEBUG:boto:Token: None 2024-03-04T17:48:21.792 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.792 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kmtJ1xpB5ZWpZGp0TaXySTayTgY= 2024-03-04T17:48:21.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kmtJ1xpB5ZWpZGp0TaXySTayTgY=', 'Content-Length': '0'} 2024-03-04T17:48:21.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc950a1de745a902-0065e60965-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:21.796 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.796 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.796 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.796 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.796 DEBUG:boto:Method: GET 2024-03-04T17:48:21.796 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.796 DEBUG:boto:Data: 2024-03-04T17:48:21.796 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.796 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.797 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.797 DEBUG:boto:Params: {} 2024-03-04T17:48:21.797 DEBUG:boto:Token: None 2024-03-04T17:48:21.797 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.797 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6cah3TuBlKPSiIvOfSrSMLsOpyY= 2024-03-04T17:48:21.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6cah3TuBlKPSiIvOfSrSMLsOpyY=', 'Content-Length': '0'} 2024-03-04T17:48:21.800 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b06d28b65b65ea69-0065e60965-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.840 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.840 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:21.840 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.840 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.841 DEBUG:boto:Method: GET 2024-03-04T17:48:21.841 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.841 DEBUG:boto:Data: 2024-03-04T17:48:21.841 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.841 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.841 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.841 DEBUG:boto:Params: {} 2024-03-04T17:48:21.841 DEBUG:boto:Token: None 2024-03-04T17:48:21.841 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:21.841 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6cah3TuBlKPSiIvOfSrSMLsOpyY= 2024-03-04T17:48:21.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6cah3TuBlKPSiIvOfSrSMLsOpyY=', 'Content-Length': '0'} 2024-03-04T17:48:21.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f115104d77722853-0065e60965-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.845 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.845 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.845 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:21.845 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:21.845 DEBUG:boto:Method: GET 2024-03-04T17:48:21.845 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:21.845 DEBUG:boto:Data: 2024-03-04T17:48:21.845 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.845 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.846 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.846 DEBUG:boto:Params: {} 2024-03-04T17:48:21.846 DEBUG:boto:Token: None 2024-03-04T17:48:21.846 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/ 2024-03-04T17:48:21.846 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ= 2024-03-04T17:48:21.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeab4fd227614fa6-0065e60965-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.850 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:21.851 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:21.851 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:21.851 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:21.851 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:21.851 DEBUG:boto:Method: GET 2024-03-04T17:48:21.851 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:21.851 DEBUG:boto:Data: 2024-03-04T17:48:21.851 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.851 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.852 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.852 DEBUG:boto:Params: {} 2024-03-04T17:48:21.852 DEBUG:boto:Token: None 2024-03-04T17:48:21.852 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/ 2024-03-04T17:48:21.852 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ= 2024-03-04T17:48:21.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UE1RSdJEclBujiyUVk4QET7lmPQ=', 'Content-Length': '0'} 2024-03-04T17:48:21.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000006887c91bb35d2b-0065e60965-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.857 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:21.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:21.857 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:21.857 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:21.857 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.858 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.858 DEBUG:boto:Method: GET 2024-03-04T17:48:21.858 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.858 DEBUG:boto:Data: 2024-03-04T17:48:21.858 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.858 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.858 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.858 DEBUG:boto:Params: {} 2024-03-04T17:48:21.858 DEBUG:boto:Token: None 2024-03-04T17:48:21.858 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.859 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SDPtK77/U58et7tc4wxQBAUaRjY= 2024-03-04T17:48:21.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SDPtK77/U58et7tc4wxQBAUaRjY=', 'Content-Length': '0'} 2024-03-04T17:48:21.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6d665f6d02a4b41-0065e60965-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.862 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:21.862 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:21.862 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.862 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.862 DEBUG:boto:Method: GET 2024-03-04T17:48:21.862 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.862 DEBUG:boto:Data: 2024-03-04T17:48:21.862 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.862 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.862 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.862 DEBUG:boto:Params: {} 2024-03-04T17:48:21.863 DEBUG:boto:Token: None 2024-03-04T17:48:21.863 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.863 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SDPtK77/U58et7tc4wxQBAUaRjY= 2024-03-04T17:48:21.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SDPtK77/U58et7tc4wxQBAUaRjY=', 'Content-Length': '0'} 2024-03-04T17:48:21.866 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c23ed2c8cc08df30-0065e60965-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.912 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:21.912 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:21.912 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.912 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.912 DEBUG:boto:Method: HEAD 2024-03-04T17:48:21.913 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.913 DEBUG:boto:Data: 2024-03-04T17:48:21.913 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.913 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.913 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.913 DEBUG:boto:Params: {} 2024-03-04T17:48:21.913 DEBUG:boto:Token: None 2024-03-04T17:48:21.913 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.913 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5oQENiq4+le4L32k9Ks/KeJ4IHw= 2024-03-04T17:48:21.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5oQENiq4+le4L32k9Ks/KeJ4IHw=', 'Content-Length': '0'} 2024-03-04T17:48:21.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a6e081371f0ace5-0065e60965-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.917 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:21.917 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:21.917 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.917 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.917 DEBUG:boto:Method: HEAD 2024-03-04T17:48:21.917 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.917 DEBUG:boto:Data: 2024-03-04T17:48:21.917 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.917 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.917 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.917 DEBUG:boto:Params: {} 2024-03-04T17:48:21.918 DEBUG:boto:Token: None 2024-03-04T17:48:21.918 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.918 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5oQENiq4+le4L32k9Ks/KeJ4IHw= 2024-03-04T17:48:21.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5oQENiq4+le4L32k9Ks/KeJ4IHw=', 'Content-Length': '0'} 2024-03-04T17:48:21.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d24a740e62ec5fe2-0065e60965-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:21.923 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:21.923 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:21.924 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.924 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.924 DEBUG:boto:Method: GET 2024-03-04T17:48:21.924 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.924 DEBUG:boto:Data: 2024-03-04T17:48:21.924 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.924 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:21.924 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.924 DEBUG:boto:Params: {} 2024-03-04T17:48:21.924 DEBUG:boto:Token: None 2024-03-04T17:48:21.924 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.925 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SDPtK77/U58et7tc4wxQBAUaRjY= 2024-03-04T17:48:21.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SDPtK77/U58et7tc4wxQBAUaRjY=', 'Content-Length': '0'} 2024-03-04T17:48:21.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000931d5f5a8e4b421a-0065e60965-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:21.968 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:21.968 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:21.968 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.969 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.969 DEBUG:boto:Method: GET 2024-03-04T17:48:21.969 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.969 DEBUG:boto:Data: 2024-03-04T17:48:21.969 DEBUG:boto:Headers: {} 2024-03-04T17:48:21.969 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:21.969 DEBUG:boto:Port: 8000 2024-03-04T17:48:21.969 DEBUG:boto:Params: {} 2024-03-04T17:48:21.969 DEBUG:boto:Token: None 2024-03-04T17:48:21.969 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:21 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:21.970 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SDPtK77/U58et7tc4wxQBAUaRjY= 2024-03-04T17:48:21.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SDPtK77/U58et7tc4wxQBAUaRjY=', 'Content-Length': '0'} 2024-03-04T17:48:21.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002aad489d02400374-0065e60965-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.016 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.016 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.016 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:22.017 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:22.017 DEBUG:boto:Method: GET 2024-03-04T17:48:22.017 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:22.017 DEBUG:boto:Data: 2024-03-04T17:48:22.017 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.017 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:22.017 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.017 DEBUG:boto:Params: {} 2024-03-04T17:48:22.018 DEBUG:boto:Token: None 2024-03-04T17:48:22.018 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.018 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000232070a5c105e7aa-0065e60966-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.023 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.023 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.023 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.023 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:22.023 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:22.023 DEBUG:boto:Method: GET 2024-03-04T17:48:22.024 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:22.024 DEBUG:boto:Data: 2024-03-04T17:48:22.024 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.024 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.024 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.024 DEBUG:boto:Params: {} 2024-03-04T17:48:22.024 DEBUG:boto:Token: None 2024-03-04T17:48:22.024 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.024 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b84969b4d652f78a-0065e60966-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.029 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.030 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.030 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.030 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.030 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.030 DEBUG:boto:Data: 2024-03-04T17:48:22.030 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.030 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:22.030 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.030 DEBUG:boto:Params: {} 2024-03-04T17:48:22.030 DEBUG:boto:Token: None 2024-03-04T17:48:22.031 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.031 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JrEqj6FZI27B3/1DZLAhTB7w3b0= 2024-03-04T17:48:22.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JrEqj6FZI27B3/1DZLAhTB7w3b0=', 'Content-Length': '0'} 2024-03-04T17:48:22.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004320bc87a7f1211e-0065e60966-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.035 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.035 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.035 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.035 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.035 DEBUG:boto:Data: 2024-03-04T17:48:22.035 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.036 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.036 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.036 DEBUG:boto:Params: {} 2024-03-04T17:48:22.036 DEBUG:boto:Token: None 2024-03-04T17:48:22.036 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.036 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JrEqj6FZI27B3/1DZLAhTB7w3b0= 2024-03-04T17:48:22.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JrEqj6FZI27B3/1DZLAhTB7w3b0=', 'Content-Length': '0'} 2024-03-04T17:48:22.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080e5cd4cda7dcd5e-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:22.040 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.041 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.041 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.041 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.041 DEBUG:boto:Method: GET 2024-03-04T17:48:22.041 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.041 DEBUG:boto:Data: 2024-03-04T17:48:22.041 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.041 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:22.041 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.041 DEBUG:boto:Params: {} 2024-03-04T17:48:22.041 DEBUG:boto:Token: None 2024-03-04T17:48:22.042 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.042 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7mm/iCrBKp/gEJi9JBpW96Mb974= 2024-03-04T17:48:22.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7mm/iCrBKp/gEJi9JBpW96Mb974=', 'Content-Length': '0'} 2024-03-04T17:48:22.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b949966eb7431455-0065e60966-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.045 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.045 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.045 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.045 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.045 DEBUG:boto:Method: GET 2024-03-04T17:48:22.045 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.045 DEBUG:boto:Data: 2024-03-04T17:48:22.045 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.045 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.046 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.046 DEBUG:boto:Params: {} 2024-03-04T17:48:22.046 DEBUG:boto:Token: None 2024-03-04T17:48:22.046 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.046 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7mm/iCrBKp/gEJi9JBpW96Mb974= 2024-03-04T17:48:22.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7mm/iCrBKp/gEJi9JBpW96Mb974=', 'Content-Length': '0'} 2024-03-04T17:48:22.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000120f272af96ef67-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.092 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.092 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.092 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:22.093 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:22.093 DEBUG:boto:Method: GET 2024-03-04T17:48:22.093 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:22.093 DEBUG:boto:Data: 2024-03-04T17:48:22.093 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.093 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:22.093 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.093 DEBUG:boto:Params: {} 2024-03-04T17:48:22.093 DEBUG:boto:Token: None 2024-03-04T17:48:22.093 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.094 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000123389eaa279319a-0065e60966-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.098 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.098 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.098 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.099 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:22.099 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:22.099 DEBUG:boto:Method: GET 2024-03-04T17:48:22.099 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:22.099 DEBUG:boto:Data: 2024-03-04T17:48:22.099 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.099 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.099 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.099 DEBUG:boto:Params: {} 2024-03-04T17:48:22.099 DEBUG:boto:Token: None 2024-03-04T17:48:22.100 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.100 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecddd150fa3eb983-0065e60966-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.104 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:22.105 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.105 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.105 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.105 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.105 DEBUG:boto:Method: GET 2024-03-04T17:48:22.105 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.106 DEBUG:boto:Data: 2024-03-04T17:48:22.106 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.106 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:22.106 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.106 DEBUG:boto:Params: {} 2024-03-04T17:48:22.106 DEBUG:boto:Token: None 2024-03-04T17:48:22.106 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.106 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KQSb40D0as2UamOdtkDtn78yEtA= 2024-03-04T17:48:22.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KQSb40D0as2UamOdtkDtn78yEtA=', 'Content-Length': '0'} 2024-03-04T17:48:22.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c4e4ca0ffda5149-0065e60966-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.152 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.152 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.152 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.152 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.152 DEBUG:boto:Method: GET 2024-03-04T17:48:22.153 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.153 DEBUG:boto:Data: 2024-03-04T17:48:22.153 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.153 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.153 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.153 DEBUG:boto:Params: {} 2024-03-04T17:48:22.153 DEBUG:boto:Token: None 2024-03-04T17:48:22.153 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.153 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KQSb40D0as2UamOdtkDtn78yEtA= 2024-03-04T17:48:22.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KQSb40D0as2UamOdtkDtn78yEtA=', 'Content-Length': '0'} 2024-03-04T17:48:22.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008210e907402441fa-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.200 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.200 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.200 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.200 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.201 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.201 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.201 DEBUG:boto:Data: 2024-03-04T17:48:22.201 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.201 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:22.201 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.201 DEBUG:boto:Params: {} 2024-03-04T17:48:22.201 DEBUG:boto:Token: None 2024-03-04T17:48:22.201 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.202 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Lv/lic+WoNl4z8UlKFChT8ekKW8= 2024-03-04T17:48:22.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Lv/lic+WoNl4z8UlKFChT8ekKW8=', 'Content-Length': '0'} 2024-03-04T17:48:22.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d34c9a1cc704cc9c-0065e60966-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.205 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.205 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.205 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.205 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.205 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.205 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.205 DEBUG:boto:Data: 2024-03-04T17:48:22.205 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.205 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.206 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.206 DEBUG:boto:Params: {} 2024-03-04T17:48:22.206 DEBUG:boto:Token: None 2024-03-04T17:48:22.206 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.206 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Lv/lic+WoNl4z8UlKFChT8ekKW8= 2024-03-04T17:48:22.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Lv/lic+WoNl4z8UlKFChT8ekKW8=', 'Content-Length': '0'} 2024-03-04T17:48:22.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000992b0a5accbff7f5-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:22.210 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.210 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.210 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.210 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.210 DEBUG:boto:Method: GET 2024-03-04T17:48:22.210 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.210 DEBUG:boto:Data: 2024-03-04T17:48:22.210 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.210 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:22.210 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.210 DEBUG:boto:Params: {} 2024-03-04T17:48:22.211 DEBUG:boto:Token: None 2024-03-04T17:48:22.211 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.211 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KQSb40D0as2UamOdtkDtn78yEtA= 2024-03-04T17:48:22.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KQSb40D0as2UamOdtkDtn78yEtA=', 'Content-Length': '0'} 2024-03-04T17:48:22.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f0e4c4aab2e8b50-0065e60966-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.256 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.256 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.256 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.256 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.257 DEBUG:boto:Method: GET 2024-03-04T17:48:22.257 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.257 DEBUG:boto:Data: 2024-03-04T17:48:22.257 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.257 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.257 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.257 DEBUG:boto:Params: {} 2024-03-04T17:48:22.257 DEBUG:boto:Token: None 2024-03-04T17:48:22.257 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.257 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KQSb40D0as2UamOdtkDtn78yEtA= 2024-03-04T17:48:22.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KQSb40D0as2UamOdtkDtn78yEtA=', 'Content-Length': '0'} 2024-03-04T17:48:22.260 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c01d3ec4ff96497e-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.304 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.304 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.304 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:22.304 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:22.304 DEBUG:boto:Method: GET 2024-03-04T17:48:22.305 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:22.305 DEBUG:boto:Data: 2024-03-04T17:48:22.305 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.305 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:22.305 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.305 DEBUG:boto:Params: {} 2024-03-04T17:48:22.305 DEBUG:boto:Token: None 2024-03-04T17:48:22.305 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.305 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f30711b15e1cbd3a-0065e60966-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.310 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.311 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.311 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.311 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:22.311 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:22.311 DEBUG:boto:Method: GET 2024-03-04T17:48:22.311 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:22.311 DEBUG:boto:Data: 2024-03-04T17:48:22.311 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.311 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.312 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.312 DEBUG:boto:Params: {} 2024-03-04T17:48:22.312 DEBUG:boto:Token: None 2024-03-04T17:48:22.312 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.312 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000233d5ae3153f160f-0065e60966-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.317 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.318 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.318 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.318 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.318 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.318 DEBUG:boto:Data: 2024-03-04T17:48:22.318 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.318 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:22.318 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.319 DEBUG:boto:Params: {} 2024-03-04T17:48:22.319 DEBUG:boto:Token: None 2024-03-04T17:48:22.319 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.319 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:N0496qPBmvbR96kZqFX/DF5nMdM= 2024-03-04T17:48:22.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:N0496qPBmvbR96kZqFX/DF5nMdM=', 'Content-Length': '0'} 2024-03-04T17:48:22.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a20434d4fc4dbd2-0065e60966-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.323 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.323 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.323 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.324 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.324 DEBUG:boto:Data: 2024-03-04T17:48:22.324 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.324 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.324 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.324 DEBUG:boto:Params: {} 2024-03-04T17:48:22.324 DEBUG:boto:Token: None 2024-03-04T17:48:22.324 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.324 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:N0496qPBmvbR96kZqFX/DF5nMdM= 2024-03-04T17:48:22.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:N0496qPBmvbR96kZqFX/DF5nMdM=', 'Content-Length': '0'} 2024-03-04T17:48:22.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057873ce701433026-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:22.331 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.331 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.331 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.331 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.331 DEBUG:boto:Method: GET 2024-03-04T17:48:22.331 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.331 DEBUG:boto:Data: 2024-03-04T17:48:22.331 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.331 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:22.331 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.331 DEBUG:boto:Params: {} 2024-03-04T17:48:22.332 DEBUG:boto:Token: None 2024-03-04T17:48:22.332 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.332 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KQSb40D0as2UamOdtkDtn78yEtA= 2024-03-04T17:48:22.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KQSb40D0as2UamOdtkDtn78yEtA=', 'Content-Length': '0'} 2024-03-04T17:48:22.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000429cf56c92f6c1c9-0065e60966-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.376 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.376 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:22.376 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.376 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.377 DEBUG:boto:Method: GET 2024-03-04T17:48:22.377 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.377 DEBUG:boto:Data: 2024-03-04T17:48:22.377 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.377 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.377 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.377 DEBUG:boto:Params: {} 2024-03-04T17:48:22.377 DEBUG:boto:Token: None 2024-03-04T17:48:22.377 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:22.377 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KQSb40D0as2UamOdtkDtn78yEtA= 2024-03-04T17:48:22.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KQSb40D0as2UamOdtkDtn78yEtA=', 'Content-Length': '0'} 2024-03-04T17:48:22.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f14af6d81bffac13-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.424 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.424 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.424 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:22.424 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:22.425 DEBUG:boto:Method: GET 2024-03-04T17:48:22.425 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:22.425 DEBUG:boto:Data: 2024-03-04T17:48:22.425 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.425 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:22.425 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.425 DEBUG:boto:Params: {} 2024-03-04T17:48:22.425 DEBUG:boto:Token: None 2024-03-04T17:48:22.425 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.425 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002004762f09c4dab9-0065e60966-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.430 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.431 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.431 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.431 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:22.431 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:22.431 DEBUG:boto:Method: GET 2024-03-04T17:48:22.431 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:22.431 DEBUG:boto:Data: 2024-03-04T17:48:22.432 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.432 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.432 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.432 DEBUG:boto:Params: {} 2024-03-04T17:48:22.432 DEBUG:boto:Token: None 2024-03-04T17:48:22.432 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.432 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9afc8b3e0878fcf-0065e60966-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.437 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.437 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-21 zones={test-zone2, test-zone1} 2024-03-04T17:48:22.437 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-21 zones={test-zone3, test-zone1} 2024-03-04T17:48:22.437 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.437 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.437 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.438 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:48:22.438 DEBUG:boto:Data: 2024-03-04T17:48:22.438 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.438 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.438 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.438 DEBUG:boto:Params: {} 2024-03-04T17:48:22.438 DEBUG:boto:Token: None 2024-03-04T17:48:22.438 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.438 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:c3nWjotdVe5TeXWybgzcKEPgbZo= 2024-03-04T17:48:22.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:c3nWjotdVe5TeXWybgzcKEPgbZo=', 'Content-Length': '0'} 2024-03-04T17:48:22.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c75388a9dd179a70-0065e60966-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.443 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.443 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.443 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.443 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:48:22.443 DEBUG:boto:Data: 2024-03-04T17:48:22.443 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.443 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.443 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.443 DEBUG:boto:Params: {} 2024-03-04T17:48:22.443 DEBUG:boto:Token: None 2024-03-04T17:48:22.444 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.444 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:c3nWjotdVe5TeXWybgzcKEPgbZo= 2024-03-04T17:48:22.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:c3nWjotdVe5TeXWybgzcKEPgbZo=', 'Content-Length': '0'} 2024-03-04T17:48:22.448 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a912af17ef029f8d-0065e60966-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.448 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:48:22.448 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.448 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.448 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:22.448 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:22.448 DEBUG:boto:Method: GET 2024-03-04T17:48:22.449 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:22.449 DEBUG:boto:Data: 2024-03-04T17:48:22.449 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.449 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.449 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.449 DEBUG:boto:Params: {} 2024-03-04T17:48:22.449 DEBUG:boto:Token: None 2024-03-04T17:48:22.449 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/?versions 2024-03-04T17:48:22.449 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4NcevuorWP479jzCA/9/zBHGWA= 2024-03-04T17:48:22.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4NcevuorWP479jzCA/9/zBHGWA=', 'Content-Length': '0'} 2024-03-04T17:48:22.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f09ec706cdd41b0-0065e60966-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.454 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.454 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:22.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:22.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:22.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:22.455 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:48:22.455 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.455 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.455 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:22.455 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:22.455 DEBUG:boto:Method: GET 2024-03-04T17:48:22.455 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:22.455 DEBUG:boto:Data: 2024-03-04T17:48:22.455 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.455 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.456 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.456 DEBUG:boto:Params: {} 2024-03-04T17:48:22.456 DEBUG:boto:Token: None 2024-03-04T17:48:22.456 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/?versions 2024-03-04T17:48:22.456 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4NcevuorWP479jzCA/9/zBHGWA= 2024-03-04T17:48:22.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4NcevuorWP479jzCA/9/zBHGWA=', 'Content-Length': '0'} 2024-03-04T17:48:22.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfb8fc43149247dc-0065e60966-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.461 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.461 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:22.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:22.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:22.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:22.462 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.462 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.462 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:22.462 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:22.462 DEBUG:boto:Method: GET 2024-03-04T17:48:22.462 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:22.462 DEBUG:boto:Data: 2024-03-04T17:48:22.463 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.463 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.463 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.463 DEBUG:boto:Params: {} 2024-03-04T17:48:22.463 DEBUG:boto:Token: None 2024-03-04T17:48:22.463 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/?versions 2024-03-04T17:48:22.463 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4NcevuorWP479jzCA/9/zBHGWA= 2024-03-04T17:48:22.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4NcevuorWP479jzCA/9/zBHGWA=', 'Content-Length': '0'} 2024-03-04T17:48:22.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041f850b4a314fbf0-0065e60966-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.468 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.469 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.469 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.469 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:22.469 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:22.469 DEBUG:boto:Method: GET 2024-03-04T17:48:22.469 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:22.469 DEBUG:boto:Data: 2024-03-04T17:48:22.470 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.470 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.470 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.470 DEBUG:boto:Params: {} 2024-03-04T17:48:22.470 DEBUG:boto:Token: None 2024-03-04T17:48:22.470 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/?versions 2024-03-04T17:48:22.470 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4NcevuorWP479jzCA/9/zBHGWA= 2024-03-04T17:48:22.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4NcevuorWP479jzCA/9/zBHGWA=', 'Content-Length': '0'} 2024-03-04T17:48:22.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da1eb05ec12308af-0065e60966-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.475 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:22.475 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:22.475 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:22.475 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.476 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.476 DEBUG:boto:Method: GET 2024-03-04T17:48:22.476 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.476 DEBUG:boto:Data: 2024-03-04T17:48:22.476 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.476 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.476 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.476 DEBUG:boto:Params: {} 2024-03-04T17:48:22.476 DEBUG:boto:Token: None 2024-03-04T17:48:22.477 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.477 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E4ahF6EvAVxZUo473TYtUhsg6y8= 2024-03-04T17:48:22.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E4ahF6EvAVxZUo473TYtUhsg6y8=', 'Content-Length': '0'} 2024-03-04T17:48:22.480 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e170e9427203ec74-0065e60966-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.480 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:22.480 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:22.480 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.480 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.480 DEBUG:boto:Method: GET 2024-03-04T17:48:22.481 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.481 DEBUG:boto:Data: 2024-03-04T17:48:22.481 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.481 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.481 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.481 DEBUG:boto:Params: {} 2024-03-04T17:48:22.481 DEBUG:boto:Token: None 2024-03-04T17:48:22.481 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.481 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E4ahF6EvAVxZUo473TYtUhsg6y8= 2024-03-04T17:48:22.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E4ahF6EvAVxZUo473TYtUhsg6y8=', 'Content-Length': '0'} 2024-03-04T17:48:22.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023e0ad83d7bd1a9b-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.485 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:22.485 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:22.485 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.485 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.485 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.485 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.485 DEBUG:boto:Data: 2024-03-04T17:48:22.485 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.486 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.486 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.486 DEBUG:boto:Params: {} 2024-03-04T17:48:22.486 DEBUG:boto:Token: None 2024-03-04T17:48:22.486 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.486 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6wKMh6e7TsnTqzzaJv/LjfLxcwE= 2024-03-04T17:48:22.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6wKMh6e7TsnTqzzaJv/LjfLxcwE=', 'Content-Length': '0'} 2024-03-04T17:48:22.489 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003cf44a8620c12033-0065e60966-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.489 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:22.489 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:22.489 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.490 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.490 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.490 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.490 DEBUG:boto:Data: 2024-03-04T17:48:22.490 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.490 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.490 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.490 DEBUG:boto:Params: {} 2024-03-04T17:48:22.490 DEBUG:boto:Token: None 2024-03-04T17:48:22.490 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.491 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6wKMh6e7TsnTqzzaJv/LjfLxcwE= 2024-03-04T17:48:22.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6wKMh6e7TsnTqzzaJv/LjfLxcwE=', 'Content-Length': '0'} 2024-03-04T17:48:22.493 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000003c84b83168a2265-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:22.494 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:22.494 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:22.494 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.494 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.494 DEBUG:boto:Method: GET 2024-03-04T17:48:22.494 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.494 DEBUG:boto:Data: 2024-03-04T17:48:22.494 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.495 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.495 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.495 DEBUG:boto:Params: {} 2024-03-04T17:48:22.495 DEBUG:boto:Token: None 2024-03-04T17:48:22.495 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.495 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E4ahF6EvAVxZUo473TYtUhsg6y8= 2024-03-04T17:48:22.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E4ahF6EvAVxZUo473TYtUhsg6y8=', 'Content-Length': '0'} 2024-03-04T17:48:22.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000febfb4bdbceb505d-0065e60966-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.498 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:22.498 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:22.498 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.498 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.498 DEBUG:boto:Method: GET 2024-03-04T17:48:22.498 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.499 DEBUG:boto:Data: 2024-03-04T17:48:22.499 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.499 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.499 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.499 DEBUG:boto:Params: {} 2024-03-04T17:48:22.499 DEBUG:boto:Token: None 2024-03-04T17:48:22.499 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.499 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E4ahF6EvAVxZUo473TYtUhsg6y8= 2024-03-04T17:48:22.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E4ahF6EvAVxZUo473TYtUhsg6y8=', 'Content-Length': '0'} 2024-03-04T17:48:22.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000106d78304425e205-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.502 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.502 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.502 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:22.503 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:22.503 DEBUG:boto:Method: GET 2024-03-04T17:48:22.503 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:48:22.503 DEBUG:boto:Data: 2024-03-04T17:48:22.503 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.503 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.503 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.503 DEBUG:boto:Params: {} 2024-03-04T17:48:22.503 DEBUG:boto:Token: None 2024-03-04T17:48:22.504 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.504 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e0771a87d517b40-0065e60966-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.552 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.552 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.552 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.553 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:22.553 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:22.553 DEBUG:boto:Method: GET 2024-03-04T17:48:22.553 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:48:22.553 DEBUG:boto:Data: 2024-03-04T17:48:22.553 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.553 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.553 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.553 DEBUG:boto:Params: {} 2024-03-04T17:48:22.553 DEBUG:boto:Token: None 2024-03-04T17:48:22.553 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.554 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee396a5e3e0df51-0065e60966-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.558 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.558 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:22.559 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:22.559 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.559 DEBUG:boto:Path: /ppfahn-21/dummy 2024-03-04T17:48:22.559 DEBUG:boto:Data: 2024-03-04T17:48:22.559 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.559 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.559 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.559 DEBUG:boto:Params: {} 2024-03-04T17:48:22.559 DEBUG:boto:Token: None 2024-03-04T17:48:22.560 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/dummy 2024-03-04T17:48:22.560 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6GLP6I6I8a64p056Q/3WMUadaN8= 2024-03-04T17:48:22.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6GLP6I6I8a64p056Q/3WMUadaN8=', 'Content-Length': '0'} 2024-03-04T17:48:22.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc63e4e233033177-0065e60966-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.564 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:22.564 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:22.564 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.564 DEBUG:boto:Path: /ppfahn-21/dummy 2024-03-04T17:48:22.564 DEBUG:boto:Data: 2024-03-04T17:48:22.564 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.565 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.565 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.565 DEBUG:boto:Params: {} 2024-03-04T17:48:22.565 DEBUG:boto:Token: None 2024-03-04T17:48:22.565 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/dummy 2024-03-04T17:48:22.565 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6GLP6I6I8a64p056Q/3WMUadaN8= 2024-03-04T17:48:22.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6GLP6I6I8a64p056Q/3WMUadaN8=', 'Content-Length': '0'} 2024-03-04T17:48:22.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000074168abf9785d92c-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:22.569 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:22.569 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:22.569 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.569 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.569 DEBUG:boto:Method: GET 2024-03-04T17:48:22.569 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.570 DEBUG:boto:Data: 2024-03-04T17:48:22.570 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.570 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.570 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.570 DEBUG:boto:Params: {} 2024-03-04T17:48:22.570 DEBUG:boto:Token: None 2024-03-04T17:48:22.570 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.570 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E4ahF6EvAVxZUo473TYtUhsg6y8= 2024-03-04T17:48:22.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E4ahF6EvAVxZUo473TYtUhsg6y8=', 'Content-Length': '0'} 2024-03-04T17:48:22.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008796b48a4ff52482-0065e60966-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.573 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:22.574 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:22.574 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.574 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.574 DEBUG:boto:Method: GET 2024-03-04T17:48:22.574 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.574 DEBUG:boto:Data: 2024-03-04T17:48:22.574 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.574 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.574 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.574 DEBUG:boto:Params: {} 2024-03-04T17:48:22.575 DEBUG:boto:Token: None 2024-03-04T17:48:22.575 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:22.575 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E4ahF6EvAVxZUo473TYtUhsg6y8= 2024-03-04T17:48:22.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E4ahF6EvAVxZUo473TYtUhsg6y8=', 'Content-Length': '0'} 2024-03-04T17:48:22.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b8b5b63ce0caaf49-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.578 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.578 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.579 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:22.579 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:22.579 DEBUG:boto:Method: GET 2024-03-04T17:48:22.579 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:48:22.579 DEBUG:boto:Data: 2024-03-04T17:48:22.579 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.579 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.580 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.580 DEBUG:boto:Params: {} 2024-03-04T17:48:22.580 DEBUG:boto:Token: None 2024-03-04T17:48:22.580 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.580 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4285653f7d95a63-0065e60966-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.586 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.586 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.586 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.587 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:22.587 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:22.587 DEBUG:boto:Method: GET 2024-03-04T17:48:22.587 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:48:22.587 DEBUG:boto:Data: 2024-03-04T17:48:22.587 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.587 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.587 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.587 DEBUG:boto:Params: {} 2024-03-04T17:48:22.587 DEBUG:boto:Token: None 2024-03-04T17:48:22.588 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.588 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084d819d3fbb14716-0065e60966-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.640 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:22.641 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.641 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.641 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.641 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.641 DEBUG:boto:Method: GET 2024-03-04T17:48:22.641 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.642 DEBUG:boto:Data: 2024-03-04T17:48:22.642 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.642 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.642 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.642 DEBUG:boto:Params: {} 2024-03-04T17:48:22.642 DEBUG:boto:Token: None 2024-03-04T17:48:22.642 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.642 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wi2bXXmdDYj2IYJbjDjsk6ey48s= 2024-03-04T17:48:22.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wi2bXXmdDYj2IYJbjDjsk6ey48s=', 'Content-Length': '0'} 2024-03-04T17:48:22.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff54271b5e5a387c-0065e60966-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.646 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.646 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.646 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.646 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.646 DEBUG:boto:Method: GET 2024-03-04T17:48:22.646 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.646 DEBUG:boto:Data: 2024-03-04T17:48:22.646 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.647 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.647 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.647 DEBUG:boto:Params: {} 2024-03-04T17:48:22.647 DEBUG:boto:Token: None 2024-03-04T17:48:22.647 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.647 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wi2bXXmdDYj2IYJbjDjsk6ey48s= 2024-03-04T17:48:22.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wi2bXXmdDYj2IYJbjDjsk6ey48s=', 'Content-Length': '0'} 2024-03-04T17:48:22.650 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa6d5e155ac605b9-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.650 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.651 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.651 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.651 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.651 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.651 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.651 DEBUG:boto:Data: 2024-03-04T17:48:22.651 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.651 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.651 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.651 DEBUG:boto:Params: {} 2024-03-04T17:48:22.652 DEBUG:boto:Token: None 2024-03-04T17:48:22.652 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.652 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3U6KF9Oru0SaAHld9kjt8s3L238= 2024-03-04T17:48:22.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3U6KF9Oru0SaAHld9kjt8s3L238=', 'Content-Length': '0'} 2024-03-04T17:48:22.655 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058f80265a0ec4557-0065e60966-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.655 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.655 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.655 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.655 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.655 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.656 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.656 DEBUG:boto:Data: 2024-03-04T17:48:22.656 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.656 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.656 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.656 DEBUG:boto:Params: {} 2024-03-04T17:48:22.656 DEBUG:boto:Token: None 2024-03-04T17:48:22.656 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.656 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3U6KF9Oru0SaAHld9kjt8s3L238= 2024-03-04T17:48:22.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3U6KF9Oru0SaAHld9kjt8s3L238=', 'Content-Length': '0'} 2024-03-04T17:48:22.659 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b42d7d405c40fc26-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:22.660 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.660 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.660 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.660 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.660 DEBUG:boto:Method: GET 2024-03-04T17:48:22.660 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.660 DEBUG:boto:Data: 2024-03-04T17:48:22.660 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.660 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.661 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.661 DEBUG:boto:Params: {} 2024-03-04T17:48:22.661 DEBUG:boto:Token: None 2024-03-04T17:48:22.661 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.661 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wi2bXXmdDYj2IYJbjDjsk6ey48s= 2024-03-04T17:48:22.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wi2bXXmdDYj2IYJbjDjsk6ey48s=', 'Content-Length': '0'} 2024-03-04T17:48:22.664 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000670d6e9e297e859e-0065e60966-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.708 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.708 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.708 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.708 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.708 DEBUG:boto:Method: GET 2024-03-04T17:48:22.709 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.709 DEBUG:boto:Data: 2024-03-04T17:48:22.709 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.709 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.709 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.709 DEBUG:boto:Params: {} 2024-03-04T17:48:22.709 DEBUG:boto:Token: None 2024-03-04T17:48:22.709 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.709 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wi2bXXmdDYj2IYJbjDjsk6ey48s= 2024-03-04T17:48:22.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wi2bXXmdDYj2IYJbjDjsk6ey48s=', 'Content-Length': '0'} 2024-03-04T17:48:22.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d7eb4ba9d1db806-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.713 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.713 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.713 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:22.713 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:22.713 DEBUG:boto:Method: GET 2024-03-04T17:48:22.713 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:22.713 DEBUG:boto:Data: 2024-03-04T17:48:22.714 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.714 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.714 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.714 DEBUG:boto:Params: {} 2024-03-04T17:48:22.714 DEBUG:boto:Token: None 2024-03-04T17:48:22.714 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.714 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091b66b5c77adbf3e-0065e60966-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.719 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.719 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.720 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.720 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:22.720 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:22.720 DEBUG:boto:Method: GET 2024-03-04T17:48:22.720 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:22.720 DEBUG:boto:Data: 2024-03-04T17:48:22.720 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.720 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.720 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.720 DEBUG:boto:Params: {} 2024-03-04T17:48:22.721 DEBUG:boto:Token: None 2024-03-04T17:48:22.721 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.721 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f80feb91da2f016-0065e60966-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.772 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.772 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.773 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.773 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.773 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.773 DEBUG:boto:Data: 2024-03-04T17:48:22.773 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.773 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.773 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.773 DEBUG:boto:Params: {} 2024-03-04T17:48:22.773 DEBUG:boto:Token: None 2024-03-04T17:48:22.774 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.774 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BekyGGRkZ5SFm7GSY2OELrrK5Z0= 2024-03-04T17:48:22.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BekyGGRkZ5SFm7GSY2OELrrK5Z0=', 'Content-Length': '0'} 2024-03-04T17:48:22.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068bf3d012380d8f9-0065e60966-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.778 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.778 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.778 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.778 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.778 DEBUG:boto:Data: 2024-03-04T17:48:22.778 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.778 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.778 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.778 DEBUG:boto:Params: {} 2024-03-04T17:48:22.779 DEBUG:boto:Token: None 2024-03-04T17:48:22.779 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.779 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BekyGGRkZ5SFm7GSY2OELrrK5Z0= 2024-03-04T17:48:22.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BekyGGRkZ5SFm7GSY2OELrrK5Z0=', 'Content-Length': '0'} 2024-03-04T17:48:22.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059ea5496e7071b79-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:22.783 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.783 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.783 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.783 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.783 DEBUG:boto:Method: GET 2024-03-04T17:48:22.783 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.783 DEBUG:boto:Data: 2024-03-04T17:48:22.783 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.784 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.784 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.784 DEBUG:boto:Params: {} 2024-03-04T17:48:22.784 DEBUG:boto:Token: None 2024-03-04T17:48:22.784 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.784 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wi2bXXmdDYj2IYJbjDjsk6ey48s= 2024-03-04T17:48:22.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wi2bXXmdDYj2IYJbjDjsk6ey48s=', 'Content-Length': '0'} 2024-03-04T17:48:22.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6450fed0720b040-0065e60966-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.828 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.828 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:22.828 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.828 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.829 DEBUG:boto:Method: GET 2024-03-04T17:48:22.829 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.829 DEBUG:boto:Data: 2024-03-04T17:48:22.829 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.829 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.829 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.829 DEBUG:boto:Params: {} 2024-03-04T17:48:22.829 DEBUG:boto:Token: None 2024-03-04T17:48:22.829 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:22.830 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wi2bXXmdDYj2IYJbjDjsk6ey48s= 2024-03-04T17:48:22.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wi2bXXmdDYj2IYJbjDjsk6ey48s=', 'Content-Length': '0'} 2024-03-04T17:48:22.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2ee7514dedaaddc-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.833 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.833 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.833 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:22.833 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:22.833 DEBUG:boto:Method: GET 2024-03-04T17:48:22.833 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:22.833 DEBUG:boto:Data: 2024-03-04T17:48:22.834 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.834 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.834 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.834 DEBUG:boto:Params: {} 2024-03-04T17:48:22.834 DEBUG:boto:Token: None 2024-03-04T17:48:22.834 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.834 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056f78cdfd0a6b88d-0065e60966-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.880 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.880 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:22.881 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:22.881 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:22.881 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:22.881 DEBUG:boto:Method: GET 2024-03-04T17:48:22.881 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:22.881 DEBUG:boto:Data: 2024-03-04T17:48:22.881 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.881 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.881 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.881 DEBUG:boto:Params: {} 2024-03-04T17:48:22.882 DEBUG:boto:Token: None 2024-03-04T17:48:22.882 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/ 2024-03-04T17:48:22.882 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY= 2024-03-04T17:48:22.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7XrxBsVeI7atAA/qmczycyIEuXY=', 'Content-Length': '0'} 2024-03-04T17:48:22.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072044a902766fad1-0065e60966-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.887 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:22.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:22.887 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.887 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.888 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.888 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.888 DEBUG:boto:Method: GET 2024-03-04T17:48:22.888 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.888 DEBUG:boto:Data: 2024-03-04T17:48:22.888 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.888 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.888 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.888 DEBUG:boto:Params: {} 2024-03-04T17:48:22.888 DEBUG:boto:Token: None 2024-03-04T17:48:22.889 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.889 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7mm/iCrBKp/gEJi9JBpW96Mb974= 2024-03-04T17:48:22.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7mm/iCrBKp/gEJi9JBpW96Mb974=', 'Content-Length': '0'} 2024-03-04T17:48:22.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f3cbbf69bbe1c17-0065e60966-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.892 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.892 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.892 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.892 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.892 DEBUG:boto:Method: GET 2024-03-04T17:48:22.893 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.893 DEBUG:boto:Data: 2024-03-04T17:48:22.893 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.893 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.893 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.893 DEBUG:boto:Params: {} 2024-03-04T17:48:22.893 DEBUG:boto:Token: None 2024-03-04T17:48:22.893 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.893 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7mm/iCrBKp/gEJi9JBpW96Mb974= 2024-03-04T17:48:22.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7mm/iCrBKp/gEJi9JBpW96Mb974=', 'Content-Length': '0'} 2024-03-04T17:48:22.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009556bb7b8d04c8ec-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.944 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.944 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.944 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.944 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.944 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.945 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.945 DEBUG:boto:Data: 2024-03-04T17:48:22.945 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.945 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.945 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.945 DEBUG:boto:Params: {} 2024-03-04T17:48:22.945 DEBUG:boto:Token: None 2024-03-04T17:48:22.945 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.946 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hfOczOBcgNtiYS7AgQOraRd/HbQ= 2024-03-04T17:48:22.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hfOczOBcgNtiYS7AgQOraRd/HbQ=', 'Content-Length': '0'} 2024-03-04T17:48:22.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c95ee956c6da158b-0065e60966-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.949 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.949 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.949 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.949 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.949 DEBUG:boto:Method: HEAD 2024-03-04T17:48:22.949 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.949 DEBUG:boto:Data: 2024-03-04T17:48:22.949 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.949 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:22.949 DEBUG:boto:Port: 8000 2024-03-04T17:48:22.950 DEBUG:boto:Params: {} 2024-03-04T17:48:22.950 DEBUG:boto:Token: None 2024-03-04T17:48:22.950 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.950 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hfOczOBcgNtiYS7AgQOraRd/HbQ= 2024-03-04T17:48:22.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hfOczOBcgNtiYS7AgQOraRd/HbQ=', 'Content-Length': '0'} 2024-03-04T17:48:22.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb44397deb27dbe1-0065e60966-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:22.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:22.953 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.953 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:22.953 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.953 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.954 DEBUG:boto:Method: GET 2024-03-04T17:48:22.954 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.954 DEBUG:boto:Data: 2024-03-04T17:48:22.954 DEBUG:boto:Headers: {} 2024-03-04T17:48:22.954 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:22.954 DEBUG:boto:Port: 8001 2024-03-04T17:48:22.954 DEBUG:boto:Params: {} 2024-03-04T17:48:22.954 DEBUG:boto:Token: None 2024-03-04T17:48:22.954 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:22 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:22.954 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7mm/iCrBKp/gEJi9JBpW96Mb974= 2024-03-04T17:48:22.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7mm/iCrBKp/gEJi9JBpW96Mb974=', 'Content-Length': '0'} 2024-03-04T17:48:22.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6cb1f4bc7bebf40-0065e60966-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.004 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.004 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.005 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:23.005 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:23.005 DEBUG:boto:Method: GET 2024-03-04T17:48:23.005 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:23.005 DEBUG:boto:Data: 2024-03-04T17:48:23.005 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.005 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:23.006 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.006 DEBUG:boto:Params: {} 2024-03-04T17:48:23.006 DEBUG:boto:Token: None 2024-03-04T17:48:23.006 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:23.006 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mAyToTClcSqrJfc2fg5Z3VRsIFc= 2024-03-04T17:48:23.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mAyToTClcSqrJfc2fg5Z3VRsIFc=', 'Content-Length': '0'} 2024-03-04T17:48:23.009 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066504898a85cb6c5-0065e60967-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.056 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.056 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.056 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:23.056 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:23.057 DEBUG:boto:Method: GET 2024-03-04T17:48:23.057 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:23.057 DEBUG:boto:Data: 2024-03-04T17:48:23.057 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.057 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.057 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.057 DEBUG:boto:Params: {} 2024-03-04T17:48:23.057 DEBUG:boto:Token: None 2024-03-04T17:48:23.057 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.058 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM= 2024-03-04T17:48:23.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM=', 'Content-Length': '0'} 2024-03-04T17:48:23.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0ad6bd2d050e668-0065e60967-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.108 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.108 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.109 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.109 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:23.109 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:23.109 DEBUG:boto:Method: GET 2024-03-04T17:48:23.109 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:23.109 DEBUG:boto:Data: 2024-03-04T17:48:23.109 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.109 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:23.109 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.110 DEBUG:boto:Params: {} 2024-03-04T17:48:23.110 DEBUG:boto:Token: None 2024-03-04T17:48:23.110 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.110 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM= 2024-03-04T17:48:23.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM=', 'Content-Length': '0'} 2024-03-04T17:48:23.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f15983c51251afa4-0065e60967-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.115 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.115 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.115 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.115 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.115 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.115 DEBUG:boto:Data: 2024-03-04T17:48:23.115 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.116 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.116 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.116 DEBUG:boto:Params: {} 2024-03-04T17:48:23.116 DEBUG:boto:Token: None 2024-03-04T17:48:23.116 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.116 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cUOmZCnK9cS8bthJ995r4OCZITI= 2024-03-04T17:48:23.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cUOmZCnK9cS8bthJ995r4OCZITI=', 'Content-Length': '0'} 2024-03-04T17:48:23.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000adf4280e3918917-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.121 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.121 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.121 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.121 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.122 DEBUG:boto:Data: 2024-03-04T17:48:23.122 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.122 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:23.122 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.122 DEBUG:boto:Params: {} 2024-03-04T17:48:23.123 DEBUG:boto:Token: None 2024-03-04T17:48:23.123 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.123 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cUOmZCnK9cS8bthJ995r4OCZITI= 2024-03-04T17:48:23.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cUOmZCnK9cS8bthJ995r4OCZITI=', 'Content-Length': '0'} 2024-03-04T17:48:23.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6259c8665fada67-0065e60967-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:23.128 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.128 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.129 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:23.129 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:23.129 DEBUG:boto:Method: GET 2024-03-04T17:48:23.129 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:23.129 DEBUG:boto:Data: 2024-03-04T17:48:23.130 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.130 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.130 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.130 DEBUG:boto:Params: {} 2024-03-04T17:48:23.131 DEBUG:boto:Token: None 2024-03-04T17:48:23.131 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:23.131 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mAyToTClcSqrJfc2fg5Z3VRsIFc= 2024-03-04T17:48:23.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mAyToTClcSqrJfc2fg5Z3VRsIFc=', 'Content-Length': '0'} 2024-03-04T17:48:23.134 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f385642526d1ad9e-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.176 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.176 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:23.176 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:23.177 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:23.177 DEBUG:boto:Method: GET 2024-03-04T17:48:23.177 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:23.177 DEBUG:boto:Data: 2024-03-04T17:48:23.177 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.177 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:23.177 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.177 DEBUG:boto:Params: {} 2024-03-04T17:48:23.177 DEBUG:boto:Token: None 2024-03-04T17:48:23.178 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:23.178 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mAyToTClcSqrJfc2fg5Z3VRsIFc= 2024-03-04T17:48:23.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mAyToTClcSqrJfc2fg5Z3VRsIFc=', 'Content-Length': '0'} 2024-03-04T17:48:23.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2a689efbcbd2859-0065e60967-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.224 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.224 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.225 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:23.225 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:23.225 DEBUG:boto:Method: GET 2024-03-04T17:48:23.225 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:23.225 DEBUG:boto:Data: 2024-03-04T17:48:23.225 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.225 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.225 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.225 DEBUG:boto:Params: {} 2024-03-04T17:48:23.226 DEBUG:boto:Token: None 2024-03-04T17:48:23.226 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.226 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM= 2024-03-04T17:48:23.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM=', 'Content-Length': '0'} 2024-03-04T17:48:23.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d329868949e8b6c-0065e60967-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.276 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.276 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.277 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.277 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:23.277 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:23.277 DEBUG:boto:Method: GET 2024-03-04T17:48:23.277 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:23.277 DEBUG:boto:Data: 2024-03-04T17:48:23.277 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.277 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:23.277 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.277 DEBUG:boto:Params: {} 2024-03-04T17:48:23.278 DEBUG:boto:Token: None 2024-03-04T17:48:23.278 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.278 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM= 2024-03-04T17:48:23.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM=', 'Content-Length': '0'} 2024-03-04T17:48:23.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082d085f284d42152-0065e60967-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.324 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:23.324 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.325 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.325 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.325 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.325 DEBUG:boto:Method: GET 2024-03-04T17:48:23.325 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.325 DEBUG:boto:Data: 2024-03-04T17:48:23.325 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.325 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.325 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.325 DEBUG:boto:Params: {} 2024-03-04T17:48:23.325 DEBUG:boto:Token: None 2024-03-04T17:48:23.326 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.326 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5kpotR+YgLxrlVQUf/AWCYjUlxQ= 2024-03-04T17:48:23.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5kpotR+YgLxrlVQUf/AWCYjUlxQ=', 'Content-Length': '0'} 2024-03-04T17:48:23.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000613250bf57dec026-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.329 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.329 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.329 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.330 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.330 DEBUG:boto:Method: GET 2024-03-04T17:48:23.330 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.330 DEBUG:boto:Data: 2024-03-04T17:48:23.330 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.330 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:23.330 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.330 DEBUG:boto:Params: {} 2024-03-04T17:48:23.330 DEBUG:boto:Token: None 2024-03-04T17:48:23.330 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.331 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5kpotR+YgLxrlVQUf/AWCYjUlxQ= 2024-03-04T17:48:23.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5kpotR+YgLxrlVQUf/AWCYjUlxQ=', 'Content-Length': '0'} 2024-03-04T17:48:23.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001eef19354b040b58-0065e60967-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.376 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.376 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.376 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.377 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.377 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.377 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.377 DEBUG:boto:Data: 2024-03-04T17:48:23.377 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.377 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.377 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.377 DEBUG:boto:Params: {} 2024-03-04T17:48:23.377 DEBUG:boto:Token: None 2024-03-04T17:48:23.377 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.378 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8SmNNXVgkET4pg0Q6GuKCOl9OLs= 2024-03-04T17:48:23.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8SmNNXVgkET4pg0Q6GuKCOl9OLs=', 'Content-Length': '0'} 2024-03-04T17:48:23.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bcd7478c3fda6301-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.382 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.382 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.382 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.382 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.382 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.383 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.383 DEBUG:boto:Data: 2024-03-04T17:48:23.383 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.383 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:23.383 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.383 DEBUG:boto:Params: {} 2024-03-04T17:48:23.384 DEBUG:boto:Token: None 2024-03-04T17:48:23.384 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.384 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8SmNNXVgkET4pg0Q6GuKCOl9OLs= 2024-03-04T17:48:23.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8SmNNXVgkET4pg0Q6GuKCOl9OLs=', 'Content-Length': '0'} 2024-03-04T17:48:23.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007860f0bbe7378e10-0065e60967-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:23.387 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.388 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.388 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.388 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.388 DEBUG:boto:Method: GET 2024-03-04T17:48:23.388 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.389 DEBUG:boto:Data: 2024-03-04T17:48:23.389 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.389 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.389 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.390 DEBUG:boto:Params: {} 2024-03-04T17:48:23.390 DEBUG:boto:Token: None 2024-03-04T17:48:23.390 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.390 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5kpotR+YgLxrlVQUf/AWCYjUlxQ= 2024-03-04T17:48:23.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5kpotR+YgLxrlVQUf/AWCYjUlxQ=', 'Content-Length': '0'} 2024-03-04T17:48:23.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000339875b1ee4cd3ac-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.395 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.395 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.395 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.395 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.395 DEBUG:boto:Method: GET 2024-03-04T17:48:23.395 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.395 DEBUG:boto:Data: 2024-03-04T17:48:23.395 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.396 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:23.396 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.396 DEBUG:boto:Params: {} 2024-03-04T17:48:23.396 DEBUG:boto:Token: None 2024-03-04T17:48:23.396 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.396 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5kpotR+YgLxrlVQUf/AWCYjUlxQ= 2024-03-04T17:48:23.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5kpotR+YgLxrlVQUf/AWCYjUlxQ=', 'Content-Length': '0'} 2024-03-04T17:48:23.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000728ceae8b8f2e018-0065e60967-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.440 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.440 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.440 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:23.441 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:23.441 DEBUG:boto:Method: GET 2024-03-04T17:48:23.441 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:23.441 DEBUG:boto:Data: 2024-03-04T17:48:23.441 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.441 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.441 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.441 DEBUG:boto:Params: {} 2024-03-04T17:48:23.441 DEBUG:boto:Token: None 2024-03-04T17:48:23.442 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.442 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM= 2024-03-04T17:48:23.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM=', 'Content-Length': '0'} 2024-03-04T17:48:23.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa6707679a962f0c-0065e60967-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.447 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.447 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.447 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.447 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:23.447 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:23.447 DEBUG:boto:Method: GET 2024-03-04T17:48:23.448 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:23.448 DEBUG:boto:Data: 2024-03-04T17:48:23.448 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.448 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:23.448 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.448 DEBUG:boto:Params: {} 2024-03-04T17:48:23.448 DEBUG:boto:Token: None 2024-03-04T17:48:23.448 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.448 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM= 2024-03-04T17:48:23.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM=', 'Content-Length': '0'} 2024-03-04T17:48:23.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f427abc0605ef3b8-0065e60967-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.496 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.496 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.497 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.497 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.497 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.497 DEBUG:boto:Data: 2024-03-04T17:48:23.497 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.497 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.497 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.497 DEBUG:boto:Params: {} 2024-03-04T17:48:23.497 DEBUG:boto:Token: None 2024-03-04T17:48:23.498 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.498 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:X90BjNDy4nc1iWQtrqrjyJ+uWYU= 2024-03-04T17:48:23.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:X90BjNDy4nc1iWQtrqrjyJ+uWYU=', 'Content-Length': '0'} 2024-03-04T17:48:23.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c802ebb04f768e85-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.502 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.502 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.502 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.502 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.502 DEBUG:boto:Data: 2024-03-04T17:48:23.502 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.503 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:23.503 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.503 DEBUG:boto:Params: {} 2024-03-04T17:48:23.503 DEBUG:boto:Token: None 2024-03-04T17:48:23.503 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.503 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:X90BjNDy4nc1iWQtrqrjyJ+uWYU= 2024-03-04T17:48:23.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:X90BjNDy4nc1iWQtrqrjyJ+uWYU=', 'Content-Length': '0'} 2024-03-04T17:48:23.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000352714e45cfdb15e-0065e60967-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:23.510 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.510 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.510 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.510 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.510 DEBUG:boto:Method: GET 2024-03-04T17:48:23.510 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.510 DEBUG:boto:Data: 2024-03-04T17:48:23.510 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.510 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.511 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.511 DEBUG:boto:Params: {} 2024-03-04T17:48:23.511 DEBUG:boto:Token: None 2024-03-04T17:48:23.511 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.511 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5kpotR+YgLxrlVQUf/AWCYjUlxQ= 2024-03-04T17:48:23.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5kpotR+YgLxrlVQUf/AWCYjUlxQ=', 'Content-Length': '0'} 2024-03-04T17:48:23.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038c8ef543c8e2978-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.514 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.515 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:23.515 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.515 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.515 DEBUG:boto:Method: GET 2024-03-04T17:48:23.515 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.515 DEBUG:boto:Data: 2024-03-04T17:48:23.515 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.515 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:23.515 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.515 DEBUG:boto:Params: {} 2024-03-04T17:48:23.515 DEBUG:boto:Token: None 2024-03-04T17:48:23.516 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:23.516 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5kpotR+YgLxrlVQUf/AWCYjUlxQ= 2024-03-04T17:48:23.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5kpotR+YgLxrlVQUf/AWCYjUlxQ=', 'Content-Length': '0'} 2024-03-04T17:48:23.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ea7f5b28f4968e7-0065e60967-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.560 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.563 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.563 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:23.563 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:23.563 DEBUG:boto:Method: GET 2024-03-04T17:48:23.564 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:23.564 DEBUG:boto:Data: 2024-03-04T17:48:23.564 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.564 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.564 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.564 DEBUG:boto:Params: {} 2024-03-04T17:48:23.564 DEBUG:boto:Token: None 2024-03-04T17:48:23.564 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.565 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM= 2024-03-04T17:48:23.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM=', 'Content-Length': '0'} 2024-03-04T17:48:23.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c507b566e6113db7-0065e60967-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.570 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.570 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.570 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.570 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:23.571 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:23.571 DEBUG:boto:Method: GET 2024-03-04T17:48:23.571 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:23.571 DEBUG:boto:Data: 2024-03-04T17:48:23.571 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.571 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:23.571 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.571 DEBUG:boto:Params: {} 2024-03-04T17:48:23.571 DEBUG:boto:Token: None 2024-03-04T17:48:23.572 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.572 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM= 2024-03-04T17:48:23.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM=', 'Content-Length': '0'} 2024-03-04T17:48:23.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022dde9c2a2c4c0c8-0065e60967-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.624 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.624 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-21 zones={test-zone3, test-zone1} 2024-03-04T17:48:23.625 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-21 zones={test-zone3, test-zone2} 2024-03-04T17:48:23.625 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.625 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.625 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.625 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:48:23.625 DEBUG:boto:Data: 2024-03-04T17:48:23.625 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.625 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.625 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.626 DEBUG:boto:Params: {} 2024-03-04T17:48:23.626 DEBUG:boto:Token: None 2024-03-04T17:48:23.626 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.626 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Q+FMm1xocnNmEjZWjE6RBd9pXvA= 2024-03-04T17:48:23.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Q+FMm1xocnNmEjZWjE6RBd9pXvA=', 'Content-Length': '0'} 2024-03-04T17:48:23.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b91f15786afba968-0065e60967-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.630 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.630 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.631 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.631 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:48:23.631 DEBUG:boto:Data: 2024-03-04T17:48:23.631 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.631 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:23.631 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.631 DEBUG:boto:Params: {} 2024-03-04T17:48:23.631 DEBUG:boto:Token: None 2024-03-04T17:48:23.631 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.632 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Q+FMm1xocnNmEjZWjE6RBd9pXvA= 2024-03-04T17:48:23.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Q+FMm1xocnNmEjZWjE6RBd9pXvA=', 'Content-Length': '0'} 2024-03-04T17:48:23.636 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a039137105752333-0065e60967-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.636 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:48:23.636 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.636 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.637 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:23.637 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:23.637 DEBUG:boto:Method: GET 2024-03-04T17:48:23.637 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:23.637 DEBUG:boto:Data: 2024-03-04T17:48:23.637 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.637 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.637 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.637 DEBUG:boto:Params: {} 2024-03-04T17:48:23.637 DEBUG:boto:Token: None 2024-03-04T17:48:23.638 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/?versions 2024-03-04T17:48:23.638 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:k7x4JUYjOSJgKJ1zJPEIhrT22y0= 2024-03-04T17:48:23.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:k7x4JUYjOSJgKJ1zJPEIhrT22y0=', 'Content-Length': '0'} 2024-03-04T17:48:23.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002436b755f7a2a94d-0065e60967-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.643 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.643 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:23.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:23.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:23.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:23.644 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:48:23.644 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.644 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.644 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:23.644 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:23.644 DEBUG:boto:Method: GET 2024-03-04T17:48:23.644 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:23.645 DEBUG:boto:Data: 2024-03-04T17:48:23.645 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.645 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:23.645 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.645 DEBUG:boto:Params: {} 2024-03-04T17:48:23.645 DEBUG:boto:Token: None 2024-03-04T17:48:23.645 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/?versions 2024-03-04T17:48:23.645 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:k7x4JUYjOSJgKJ1zJPEIhrT22y0= 2024-03-04T17:48:23.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:k7x4JUYjOSJgKJ1zJPEIhrT22y0=', 'Content-Length': '0'} 2024-03-04T17:48:23.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000869c5d016cccfc8d-0065e60967-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.650 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.651 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:48:23.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:48:23.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:48:23.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:48:23.651 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.652 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.652 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:23.652 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:23.652 DEBUG:boto:Method: GET 2024-03-04T17:48:23.652 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:23.652 DEBUG:boto:Data: 2024-03-04T17:48:23.652 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.652 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.652 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.652 DEBUG:boto:Params: {} 2024-03-04T17:48:23.653 DEBUG:boto:Token: None 2024-03-04T17:48:23.653 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/?versions 2024-03-04T17:48:23.653 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:k7x4JUYjOSJgKJ1zJPEIhrT22y0= 2024-03-04T17:48:23.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:k7x4JUYjOSJgKJ1zJPEIhrT22y0=', 'Content-Length': '0'} 2024-03-04T17:48:23.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b97a9f4adcf8381f-0065e60967-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.657 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.658 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.658 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.658 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:23.658 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:23.658 DEBUG:boto:Method: GET 2024-03-04T17:48:23.658 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:48:23.659 DEBUG:boto:Data: 2024-03-04T17:48:23.659 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.659 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:23.659 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.659 DEBUG:boto:Params: {} 2024-03-04T17:48:23.659 DEBUG:boto:Token: None 2024-03-04T17:48:23.659 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/?versions 2024-03-04T17:48:23.659 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:k7x4JUYjOSJgKJ1zJPEIhrT22y0= 2024-03-04T17:48:23.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:k7x4JUYjOSJgKJ1zJPEIhrT22y0=', 'Content-Length': '0'} 2024-03-04T17:48:23.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3abc536b5949a53-0065e60967-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.665 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:23.665 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:23.665 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:23.665 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.666 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.666 DEBUG:boto:Method: GET 2024-03-04T17:48:23.666 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.666 DEBUG:boto:Data: 2024-03-04T17:48:23.666 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.666 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.666 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.666 DEBUG:boto:Params: {} 2024-03-04T17:48:23.666 DEBUG:boto:Token: None 2024-03-04T17:48:23.666 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.667 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zV/G9LUj9nWwzx5DWLEPML5cbAM= 2024-03-04T17:48:23.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zV/G9LUj9nWwzx5DWLEPML5cbAM=', 'Content-Length': '0'} 2024-03-04T17:48:23.669 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c4b4a2f94b9fe13b-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.670 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:23.670 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:23.670 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.670 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.670 DEBUG:boto:Method: GET 2024-03-04T17:48:23.670 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.670 DEBUG:boto:Data: 2024-03-04T17:48:23.670 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.670 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:23.670 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.670 DEBUG:boto:Params: {} 2024-03-04T17:48:23.671 DEBUG:boto:Token: None 2024-03-04T17:48:23.671 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.671 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zV/G9LUj9nWwzx5DWLEPML5cbAM= 2024-03-04T17:48:23.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zV/G9LUj9nWwzx5DWLEPML5cbAM=', 'Content-Length': '0'} 2024-03-04T17:48:23.674 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b9bdf9c4e7bccda8-0065e60967-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.674 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:23.674 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:23.674 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.674 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.674 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.675 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.675 DEBUG:boto:Data: 2024-03-04T17:48:23.675 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.675 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.675 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.675 DEBUG:boto:Params: {} 2024-03-04T17:48:23.675 DEBUG:boto:Token: None 2024-03-04T17:48:23.675 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.675 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KJj84xVXCRjwsBTSK9Ajba79lMg= 2024-03-04T17:48:23.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KJj84xVXCRjwsBTSK9Ajba79lMg=', 'Content-Length': '0'} 2024-03-04T17:48:23.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ffa607e4dc0c4f8f-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.679 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:23.679 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:23.679 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.679 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.679 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.679 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.679 DEBUG:boto:Data: 2024-03-04T17:48:23.679 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.680 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:23.680 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.680 DEBUG:boto:Params: {} 2024-03-04T17:48:23.680 DEBUG:boto:Token: None 2024-03-04T17:48:23.680 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.680 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KJj84xVXCRjwsBTSK9Ajba79lMg= 2024-03-04T17:48:23.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KJj84xVXCRjwsBTSK9Ajba79lMg=', 'Content-Length': '0'} 2024-03-04T17:48:23.683 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007f6abcfff3936cf0-0065e60967-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:23.683 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:23.684 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:23.684 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.684 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.684 DEBUG:boto:Method: GET 2024-03-04T17:48:23.684 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.684 DEBUG:boto:Data: 2024-03-04T17:48:23.684 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.684 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.684 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.684 DEBUG:boto:Params: {} 2024-03-04T17:48:23.685 DEBUG:boto:Token: None 2024-03-04T17:48:23.685 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.685 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zV/G9LUj9nWwzx5DWLEPML5cbAM= 2024-03-04T17:48:23.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zV/G9LUj9nWwzx5DWLEPML5cbAM=', 'Content-Length': '0'} 2024-03-04T17:48:23.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008fa038ef017520f3-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.688 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:23.688 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:23.688 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.688 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.688 DEBUG:boto:Method: GET 2024-03-04T17:48:23.688 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.689 DEBUG:boto:Data: 2024-03-04T17:48:23.689 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.689 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:23.689 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.689 DEBUG:boto:Params: {} 2024-03-04T17:48:23.689 DEBUG:boto:Token: None 2024-03-04T17:48:23.689 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.689 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zV/G9LUj9nWwzx5DWLEPML5cbAM= 2024-03-04T17:48:23.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zV/G9LUj9nWwzx5DWLEPML5cbAM=', 'Content-Length': '0'} 2024-03-04T17:48:23.692 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c24d063f947c76c-0065e60967-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.692 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.692 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.692 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:23.692 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:23.693 DEBUG:boto:Method: GET 2024-03-04T17:48:23.693 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:48:23.693 DEBUG:boto:Data: 2024-03-04T17:48:23.693 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.693 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.693 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.693 DEBUG:boto:Params: {} 2024-03-04T17:48:23.693 DEBUG:boto:Token: None 2024-03-04T17:48:23.693 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.694 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM= 2024-03-04T17:48:23.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM=', 'Content-Length': '0'} 2024-03-04T17:48:23.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004552df9bf262c4cc-0065e60967-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.740 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.741 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.741 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.741 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:23.741 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:23.741 DEBUG:boto:Method: GET 2024-03-04T17:48:23.741 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:48:23.741 DEBUG:boto:Data: 2024-03-04T17:48:23.741 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.741 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:23.742 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.742 DEBUG:boto:Params: {} 2024-03-04T17:48:23.742 DEBUG:boto:Token: None 2024-03-04T17:48:23.742 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.742 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM= 2024-03-04T17:48:23.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM=', 'Content-Length': '0'} 2024-03-04T17:48:23.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1e7660fd620744c-0065e60967-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.747 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.747 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:23.747 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:23.747 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.748 DEBUG:boto:Path: /ppfahn-21/dummy 2024-03-04T17:48:23.748 DEBUG:boto:Data: 2024-03-04T17:48:23.748 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.748 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.748 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.748 DEBUG:boto:Params: {} 2024-03-04T17:48:23.748 DEBUG:boto:Token: None 2024-03-04T17:48:23.748 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/dummy 2024-03-04T17:48:23.748 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yThAAz5NHfHSgBd3SSyoRLICk64= 2024-03-04T17:48:23.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yThAAz5NHfHSgBd3SSyoRLICk64=', 'Content-Length': '0'} 2024-03-04T17:48:23.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000844b77435bda159b-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.752 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:23.752 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:23.752 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.753 DEBUG:boto:Path: /ppfahn-21/dummy 2024-03-04T17:48:23.753 DEBUG:boto:Data: 2024-03-04T17:48:23.753 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.753 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:23.753 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.753 DEBUG:boto:Params: {} 2024-03-04T17:48:23.753 DEBUG:boto:Token: None 2024-03-04T17:48:23.753 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/dummy 2024-03-04T17:48:23.753 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yThAAz5NHfHSgBd3SSyoRLICk64= 2024-03-04T17:48:23.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yThAAz5NHfHSgBd3SSyoRLICk64=', 'Content-Length': '0'} 2024-03-04T17:48:23.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052db41e7a61bdf00-0065e60967-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:48:23.758 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:23.758 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:23.758 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.758 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.758 DEBUG:boto:Method: GET 2024-03-04T17:48:23.758 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.758 DEBUG:boto:Data: 2024-03-04T17:48:23.758 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.758 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.758 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.758 DEBUG:boto:Params: {} 2024-03-04T17:48:23.759 DEBUG:boto:Token: None 2024-03-04T17:48:23.759 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.759 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zV/G9LUj9nWwzx5DWLEPML5cbAM= 2024-03-04T17:48:23.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zV/G9LUj9nWwzx5DWLEPML5cbAM=', 'Content-Length': '0'} 2024-03-04T17:48:23.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f4d31c9a4cad216c-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.762 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:48:23.762 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:48:23.762 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.762 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.762 DEBUG:boto:Method: GET 2024-03-04T17:48:23.762 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.762 DEBUG:boto:Data: 2024-03-04T17:48:23.762 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.762 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:23.763 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.763 DEBUG:boto:Params: {} 2024-03-04T17:48:23.763 DEBUG:boto:Token: None 2024-03-04T17:48:23.763 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:48:23.763 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zV/G9LUj9nWwzx5DWLEPML5cbAM= 2024-03-04T17:48:23.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zV/G9LUj9nWwzx5DWLEPML5cbAM=', 'Content-Length': '0'} 2024-03-04T17:48:23.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000736d5b8e53782155-0065e60967-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.767 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.768 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.768 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:23.768 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:23.768 DEBUG:boto:Method: GET 2024-03-04T17:48:23.768 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:48:23.768 DEBUG:boto:Data: 2024-03-04T17:48:23.768 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.768 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.768 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.768 DEBUG:boto:Params: {} 2024-03-04T17:48:23.769 DEBUG:boto:Token: None 2024-03-04T17:48:23.769 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.769 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM= 2024-03-04T17:48:23.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM=', 'Content-Length': '0'} 2024-03-04T17:48:23.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c471be177d2a5348-0065e60967-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.816 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.817 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:23.817 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:23.817 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:23.817 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:48:23.817 DEBUG:boto:Method: GET 2024-03-04T17:48:23.817 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:48:23.817 DEBUG:boto:Data: 2024-03-04T17:48:23.817 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.817 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:23.817 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.818 DEBUG:boto:Params: {} 2024-03-04T17:48:23.818 DEBUG:boto:Token: None 2024-03-04T17:48:23.818 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/ 2024-03-04T17:48:23.818 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM= 2024-03-04T17:48:23.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1JZ0Du50WUP+HpWLglCJZAbEEM=', 'Content-Length': '0'} 2024-03-04T17:48:23.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051e8184bf1874494-0065e60967-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.864 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:23.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:23.865 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:23.865 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:23.865 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.865 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.865 DEBUG:boto:Method: GET 2024-03-04T17:48:23.865 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.865 DEBUG:boto:Data: 2024-03-04T17:48:23.865 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.865 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.866 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.866 DEBUG:boto:Params: {} 2024-03-04T17:48:23.866 DEBUG:boto:Token: None 2024-03-04T17:48:23.866 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.866 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HRNj+hwBw8XAgVFU4zwfH/gY0VM= 2024-03-04T17:48:23.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HRNj+hwBw8XAgVFU4zwfH/gY0VM=', 'Content-Length': '0'} 2024-03-04T17:48:23.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085291d2907d0bcfb-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.870 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:23.870 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:23.870 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.870 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.870 DEBUG:boto:Method: GET 2024-03-04T17:48:23.870 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.870 DEBUG:boto:Data: 2024-03-04T17:48:23.870 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.870 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:23.871 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.871 DEBUG:boto:Params: {} 2024-03-04T17:48:23.871 DEBUG:boto:Token: None 2024-03-04T17:48:23.871 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.871 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HRNj+hwBw8XAgVFU4zwfH/gY0VM= 2024-03-04T17:48:23.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HRNj+hwBw8XAgVFU4zwfH/gY0VM=', 'Content-Length': '0'} 2024-03-04T17:48:23.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5530f2faacb5f2e-0065e60967-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.916 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:23.916 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:23.916 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.916 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.917 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.917 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.917 DEBUG:boto:Data: 2024-03-04T17:48:23.917 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.917 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.917 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.917 DEBUG:boto:Params: {} 2024-03-04T17:48:23.917 DEBUG:boto:Token: None 2024-03-04T17:48:23.917 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.918 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TW3bRBxe95u54IgaVppmQ6MUOL8= 2024-03-04T17:48:23.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TW3bRBxe95u54IgaVppmQ6MUOL8=', 'Content-Length': '0'} 2024-03-04T17:48:23.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc1455e5e5d8e84e-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.921 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:23.921 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:23.921 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.921 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.921 DEBUG:boto:Method: HEAD 2024-03-04T17:48:23.922 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.922 DEBUG:boto:Data: 2024-03-04T17:48:23.922 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.922 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:23.922 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.922 DEBUG:boto:Params: {} 2024-03-04T17:48:23.922 DEBUG:boto:Token: None 2024-03-04T17:48:23.922 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.922 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TW3bRBxe95u54IgaVppmQ6MUOL8= 2024-03-04T17:48:23.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TW3bRBxe95u54IgaVppmQ6MUOL8=', 'Content-Length': '0'} 2024-03-04T17:48:23.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000058df03a3cf5661c-0065e60967-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:23.926 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:23.926 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:23.926 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.926 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.926 DEBUG:boto:Method: GET 2024-03-04T17:48:23.927 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.927 DEBUG:boto:Data: 2024-03-04T17:48:23.927 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.927 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:23.927 DEBUG:boto:Port: 8001 2024-03-04T17:48:23.927 DEBUG:boto:Params: {} 2024-03-04T17:48:23.927 DEBUG:boto:Token: None 2024-03-04T17:48:23.927 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.927 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HRNj+hwBw8XAgVFU4zwfH/gY0VM= 2024-03-04T17:48:23.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HRNj+hwBw8XAgVFU4zwfH/gY0VM=', 'Content-Length': '0'} 2024-03-04T17:48:23.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002aa1e8fbad06ba39-0065e60967-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:23.976 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:23.976 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:23.976 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.976 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.977 DEBUG:boto:Method: GET 2024-03-04T17:48:23.977 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.977 DEBUG:boto:Data: 2024-03-04T17:48:23.977 DEBUG:boto:Headers: {} 2024-03-04T17:48:23.977 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:23.977 DEBUG:boto:Port: 8000 2024-03-04T17:48:23.977 DEBUG:boto:Params: {} 2024-03-04T17:48:23.977 DEBUG:boto:Token: None 2024-03-04T17:48:23.977 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:23 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:23.978 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HRNj+hwBw8XAgVFU4zwfH/gY0VM= 2024-03-04T17:48:23.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:23 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HRNj+hwBw8XAgVFU4zwfH/gY0VM=', 'Content-Length': '0'} 2024-03-04T17:48:23.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031fb0e1deacc253c-0065e60967-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.024 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:24.024 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:24.024 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:24.024 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:24.025 DEBUG:boto:Method: GET 2024-03-04T17:48:24.025 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:24.025 DEBUG:boto:Data: 2024-03-04T17:48:24.025 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.025 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.025 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.025 DEBUG:boto:Params: {} 2024-03-04T17:48:24.025 DEBUG:boto:Token: None 2024-03-04T17:48:24.025 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/ 2024-03-04T17:48:24.026 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM= 2024-03-04T17:48:24.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM=', 'Content-Length': '0'} 2024-03-04T17:48:24.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b95ac51d071d9cb2-0065e60968-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.030 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.031 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:24.031 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:24.031 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:24.031 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:24.031 DEBUG:boto:Method: GET 2024-03-04T17:48:24.031 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:24.031 DEBUG:boto:Data: 2024-03-04T17:48:24.031 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.032 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.032 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.032 DEBUG:boto:Params: {} 2024-03-04T17:48:24.032 DEBUG:boto:Token: None 2024-03-04T17:48:24.032 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/ 2024-03-04T17:48:24.032 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM= 2024-03-04T17:48:24.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM=', 'Content-Length': '0'} 2024-03-04T17:48:24.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007a93cd8d10ea996-0065e60968-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.037 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.037 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:24.037 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:24.037 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.037 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:48:24.037 DEBUG:boto:Data: 2024-03-04T17:48:24.037 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.037 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.038 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.038 DEBUG:boto:Params: {} 2024-03-04T17:48:24.038 DEBUG:boto:Token: None 2024-03-04T17:48:24.038 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:48:24.038 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:O6lvG1MqBL7LZPJIUgsjpXHCNWQ= 2024-03-04T17:48:24.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:O6lvG1MqBL7LZPJIUgsjpXHCNWQ=', 'Content-Length': '0'} 2024-03-04T17:48:24.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000343c6ae084287f45-0065e60968-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.043 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:24.043 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:24.043 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.043 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:48:24.043 DEBUG:boto:Data: 2024-03-04T17:48:24.043 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.043 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.043 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.043 DEBUG:boto:Params: {} 2024-03-04T17:48:24.043 DEBUG:boto:Token: None 2024-03-04T17:48:24.044 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:48:24.044 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:O6lvG1MqBL7LZPJIUgsjpXHCNWQ= 2024-03-04T17:48:24.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:O6lvG1MqBL7LZPJIUgsjpXHCNWQ=', 'Content-Length': '0'} 2024-03-04T17:48:24.047 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee57b0adc4c5a143-0065e60968-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:48:24.048 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:24.048 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:24.048 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:24.048 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:24.048 DEBUG:boto:Method: GET 2024-03-04T17:48:24.048 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:24.048 DEBUG:boto:Data: 2024-03-04T17:48:24.048 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.048 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.049 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.049 DEBUG:boto:Params: {} 2024-03-04T17:48:24.049 DEBUG:boto:Token: None 2024-03-04T17:48:24.049 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:24.049 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:k/y4gqPq4CwH59IViRfWGpZL1/0= 2024-03-04T17:48:24.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:k/y4gqPq4CwH59IViRfWGpZL1/0=', 'Content-Length': '0'} 2024-03-04T17:48:24.052 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061692922f87437a1-0065e60968-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.096 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:24.096 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:24.096 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:24.096 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:24.097 DEBUG:boto:Method: GET 2024-03-04T17:48:24.097 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:24.097 DEBUG:boto:Data: 2024-03-04T17:48:24.097 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.097 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.097 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.097 DEBUG:boto:Params: {} 2024-03-04T17:48:24.097 DEBUG:boto:Token: None 2024-03-04T17:48:24.097 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:48:24.098 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:k/y4gqPq4CwH59IViRfWGpZL1/0= 2024-03-04T17:48:24.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:k/y4gqPq4CwH59IViRfWGpZL1/0=', 'Content-Length': '0'} 2024-03-04T17:48:24.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbaeed23c3ecc61c-0065e60968-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.148 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:24.148 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:24.148 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:24.148 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:24.148 DEBUG:boto:Method: GET 2024-03-04T17:48:24.149 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:24.149 DEBUG:boto:Data: 2024-03-04T17:48:24.149 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.149 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.149 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.149 DEBUG:boto:Params: {} 2024-03-04T17:48:24.149 DEBUG:boto:Token: None 2024-03-04T17:48:24.149 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/ 2024-03-04T17:48:24.149 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM= 2024-03-04T17:48:24.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM=', 'Content-Length': '0'} 2024-03-04T17:48:24.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000888217228773af09-0065e60968-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.155 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.155 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:24.155 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:24.155 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:24.155 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:24.156 DEBUG:boto:Method: GET 2024-03-04T17:48:24.156 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:48:24.156 DEBUG:boto:Data: 2024-03-04T17:48:24.156 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.156 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.156 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.156 DEBUG:boto:Params: {} 2024-03-04T17:48:24.156 DEBUG:boto:Token: None 2024-03-04T17:48:24.156 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/ 2024-03-04T17:48:24.156 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM= 2024-03-04T17:48:24.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM=', 'Content-Length': '0'} 2024-03-04T17:48:24.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a0e34576ce59cd2-0065e60968-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.208 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:24.209 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.209 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.209 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.209 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.209 DEBUG:boto:Method: GET 2024-03-04T17:48:24.209 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.209 DEBUG:boto:Data: 2024-03-04T17:48:24.209 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.209 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.210 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.210 DEBUG:boto:Params: {} 2024-03-04T17:48:24.210 DEBUG:boto:Token: None 2024-03-04T17:48:24.210 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.210 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uaSmqHXXLmZb9Of7obNNGSFIlxI= 2024-03-04T17:48:24.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uaSmqHXXLmZb9Of7obNNGSFIlxI=', 'Content-Length': '0'} 2024-03-04T17:48:24.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000152c6fa45bf9ca6b-0065e60968-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.213 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.214 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.214 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.214 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.214 DEBUG:boto:Method: GET 2024-03-04T17:48:24.214 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.214 DEBUG:boto:Data: 2024-03-04T17:48:24.214 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.214 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.214 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.214 DEBUG:boto:Params: {} 2024-03-04T17:48:24.215 DEBUG:boto:Token: None 2024-03-04T17:48:24.215 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.215 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uaSmqHXXLmZb9Of7obNNGSFIlxI= 2024-03-04T17:48:24.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uaSmqHXXLmZb9Of7obNNGSFIlxI=', 'Content-Length': '0'} 2024-03-04T17:48:24.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f831b00fab78e558-0065e60968-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.219 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.219 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.219 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.219 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.219 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.219 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.219 DEBUG:boto:Data: 2024-03-04T17:48:24.219 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.219 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.219 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.220 DEBUG:boto:Params: {} 2024-03-04T17:48:24.220 DEBUG:boto:Token: None 2024-03-04T17:48:24.220 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.220 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:z0XODBYwjeoGFIVSHMRUaDz6v68= 2024-03-04T17:48:24.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:z0XODBYwjeoGFIVSHMRUaDz6v68=', 'Content-Length': '0'} 2024-03-04T17:48:24.223 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa0972479c39b36c-0065e60968-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.223 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.224 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.224 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.224 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.224 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.224 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.224 DEBUG:boto:Data: 2024-03-04T17:48:24.224 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.224 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.224 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.224 DEBUG:boto:Params: {} 2024-03-04T17:48:24.225 DEBUG:boto:Token: None 2024-03-04T17:48:24.225 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.225 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:z0XODBYwjeoGFIVSHMRUaDz6v68= 2024-03-04T17:48:24.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:z0XODBYwjeoGFIVSHMRUaDz6v68=', 'Content-Length': '0'} 2024-03-04T17:48:24.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009abde8efe91a8fa4-0065e60968-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:24.228 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.228 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.229 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.229 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.229 DEBUG:boto:Method: GET 2024-03-04T17:48:24.229 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.229 DEBUG:boto:Data: 2024-03-04T17:48:24.229 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.229 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.229 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.229 DEBUG:boto:Params: {} 2024-03-04T17:48:24.229 DEBUG:boto:Token: None 2024-03-04T17:48:24.230 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.230 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uaSmqHXXLmZb9Of7obNNGSFIlxI= 2024-03-04T17:48:24.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uaSmqHXXLmZb9Of7obNNGSFIlxI=', 'Content-Length': '0'} 2024-03-04T17:48:24.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002aabfca057fdd6f6-0065e60968-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.276 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.276 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.276 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.276 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.277 DEBUG:boto:Method: GET 2024-03-04T17:48:24.277 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.277 DEBUG:boto:Data: 2024-03-04T17:48:24.277 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.277 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.277 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.277 DEBUG:boto:Params: {} 2024-03-04T17:48:24.277 DEBUG:boto:Token: None 2024-03-04T17:48:24.277 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.278 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uaSmqHXXLmZb9Of7obNNGSFIlxI= 2024-03-04T17:48:24.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uaSmqHXXLmZb9Of7obNNGSFIlxI=', 'Content-Length': '0'} 2024-03-04T17:48:24.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001049b16feb19ecbb-0065e60968-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.281 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:24.281 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:24.281 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:24.281 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:24.282 DEBUG:boto:Method: GET 2024-03-04T17:48:24.282 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:24.282 DEBUG:boto:Data: 2024-03-04T17:48:24.282 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.282 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.282 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.282 DEBUG:boto:Params: {} 2024-03-04T17:48:24.282 DEBUG:boto:Token: None 2024-03-04T17:48:24.282 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/ 2024-03-04T17:48:24.283 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM= 2024-03-04T17:48:24.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM=', 'Content-Length': '0'} 2024-03-04T17:48:24.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004764960a1bdbccb7-0065e60968-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.287 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.288 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:24.288 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:24.288 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:24.288 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:24.288 DEBUG:boto:Method: GET 2024-03-04T17:48:24.288 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:24.289 DEBUG:boto:Data: 2024-03-04T17:48:24.289 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.289 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.289 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.289 DEBUG:boto:Params: {} 2024-03-04T17:48:24.289 DEBUG:boto:Token: None 2024-03-04T17:48:24.289 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/ 2024-03-04T17:48:24.289 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM= 2024-03-04T17:48:24.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM=', 'Content-Length': '0'} 2024-03-04T17:48:24.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d17172d41891149a-0065e60968-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.336 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.337 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.337 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.337 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.337 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.337 DEBUG:boto:Data: 2024-03-04T17:48:24.337 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.337 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.337 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.337 DEBUG:boto:Params: {} 2024-03-04T17:48:24.338 DEBUG:boto:Token: None 2024-03-04T17:48:24.338 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.338 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hiP9ATtmWfiFyQBz+hDSqKZaC4M= 2024-03-04T17:48:24.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hiP9ATtmWfiFyQBz+hDSqKZaC4M=', 'Content-Length': '0'} 2024-03-04T17:48:24.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d2a04cd4e9ce83e-0065e60968-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.342 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.342 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.342 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.342 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.342 DEBUG:boto:Data: 2024-03-04T17:48:24.342 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.342 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.343 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.343 DEBUG:boto:Params: {} 2024-03-04T17:48:24.343 DEBUG:boto:Token: None 2024-03-04T17:48:24.343 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.343 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hiP9ATtmWfiFyQBz+hDSqKZaC4M= 2024-03-04T17:48:24.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hiP9ATtmWfiFyQBz+hDSqKZaC4M=', 'Content-Length': '0'} 2024-03-04T17:48:24.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096ce302f9a44a275-0065e60968-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:48:24.347 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.347 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.348 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.348 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.348 DEBUG:boto:Method: GET 2024-03-04T17:48:24.348 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.348 DEBUG:boto:Data: 2024-03-04T17:48:24.348 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.348 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.348 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.348 DEBUG:boto:Params: {} 2024-03-04T17:48:24.348 DEBUG:boto:Token: None 2024-03-04T17:48:24.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.349 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uaSmqHXXLmZb9Of7obNNGSFIlxI= 2024-03-04T17:48:24.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uaSmqHXXLmZb9Of7obNNGSFIlxI=', 'Content-Length': '0'} 2024-03-04T17:48:24.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051205bf9026c5ac9-0065e60968-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.396 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.396 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:24.396 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.396 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.397 DEBUG:boto:Method: GET 2024-03-04T17:48:24.397 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.397 DEBUG:boto:Data: 2024-03-04T17:48:24.397 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.397 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.397 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.397 DEBUG:boto:Params: {} 2024-03-04T17:48:24.397 DEBUG:boto:Token: None 2024-03-04T17:48:24.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:48:24.397 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uaSmqHXXLmZb9Of7obNNGSFIlxI= 2024-03-04T17:48:24.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uaSmqHXXLmZb9Of7obNNGSFIlxI=', 'Content-Length': '0'} 2024-03-04T17:48:24.400 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004dce811029bdd507-0065e60968-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.401 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:24.401 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:24.401 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:24.401 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:24.401 DEBUG:boto:Method: GET 2024-03-04T17:48:24.401 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:24.401 DEBUG:boto:Data: 2024-03-04T17:48:24.401 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.401 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.402 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.402 DEBUG:boto:Params: {} 2024-03-04T17:48:24.402 DEBUG:boto:Token: None 2024-03-04T17:48:24.402 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/ 2024-03-04T17:48:24.402 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM= 2024-03-04T17:48:24.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM=', 'Content-Length': '0'} 2024-03-04T17:48:24.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3500f5ccd03cd00-0065e60968-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.407 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.407 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:24.407 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:24.407 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:24.407 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:24.408 DEBUG:boto:Method: GET 2024-03-04T17:48:24.408 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:48:24.408 DEBUG:boto:Data: 2024-03-04T17:48:24.408 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.408 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.408 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.408 DEBUG:boto:Params: {} 2024-03-04T17:48:24.408 DEBUG:boto:Token: None 2024-03-04T17:48:24.408 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/ 2024-03-04T17:48:24.408 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM= 2024-03-04T17:48:24.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM=', 'Content-Length': '0'} 2024-03-04T17:48:24.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000556d20cd0955c07a-0065e60968-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.413 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:24.414 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.414 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.414 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.414 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.414 DEBUG:boto:Method: GET 2024-03-04T17:48:24.414 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.414 DEBUG:boto:Data: 2024-03-04T17:48:24.414 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.414 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.415 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.415 DEBUG:boto:Params: {} 2024-03-04T17:48:24.415 DEBUG:boto:Token: None 2024-03-04T17:48:24.415 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.415 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QZxaxpZXfXUo4HRTOblc084YsCI= 2024-03-04T17:48:24.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QZxaxpZXfXUo4HRTOblc084YsCI=', 'Content-Length': '0'} 2024-03-04T17:48:24.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025c3a47087b3c40d-0065e60968-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.418 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.418 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.418 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.419 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.419 DEBUG:boto:Method: GET 2024-03-04T17:48:24.419 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.419 DEBUG:boto:Data: 2024-03-04T17:48:24.419 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.419 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.419 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.419 DEBUG:boto:Params: {} 2024-03-04T17:48:24.419 DEBUG:boto:Token: None 2024-03-04T17:48:24.419 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.419 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QZxaxpZXfXUo4HRTOblc084YsCI= 2024-03-04T17:48:24.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QZxaxpZXfXUo4HRTOblc084YsCI=', 'Content-Length': '0'} 2024-03-04T17:48:24.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053e8200c898105c6-0065e60968-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.464 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.464 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.464 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.464 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.464 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.465 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.465 DEBUG:boto:Data: 2024-03-04T17:48:24.465 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.465 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.465 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.465 DEBUG:boto:Params: {} 2024-03-04T17:48:24.465 DEBUG:boto:Token: None 2024-03-04T17:48:24.465 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.466 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nJSPPmwKX/mG3R+BRiztLs2AwAE= 2024-03-04T17:48:24.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nJSPPmwKX/mG3R+BRiztLs2AwAE=', 'Content-Length': '0'} 2024-03-04T17:48:24.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a3afcec447a7e6b-0065e60968-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.469 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.469 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.469 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.469 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.469 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.469 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.469 DEBUG:boto:Data: 2024-03-04T17:48:24.469 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.469 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.470 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.470 DEBUG:boto:Params: {} 2024-03-04T17:48:24.470 DEBUG:boto:Token: None 2024-03-04T17:48:24.470 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.470 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nJSPPmwKX/mG3R+BRiztLs2AwAE= 2024-03-04T17:48:24.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nJSPPmwKX/mG3R+BRiztLs2AwAE=', 'Content-Length': '0'} 2024-03-04T17:48:24.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b60a1ec9cccdc4e-0065e60968-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:24.474 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.474 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.474 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.474 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.474 DEBUG:boto:Method: GET 2024-03-04T17:48:24.474 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.474 DEBUG:boto:Data: 2024-03-04T17:48:24.474 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.474 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.474 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.475 DEBUG:boto:Params: {} 2024-03-04T17:48:24.475 DEBUG:boto:Token: None 2024-03-04T17:48:24.475 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.475 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QZxaxpZXfXUo4HRTOblc084YsCI= 2024-03-04T17:48:24.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QZxaxpZXfXUo4HRTOblc084YsCI=', 'Content-Length': '0'} 2024-03-04T17:48:24.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039344410c7f11de3-0065e60968-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.524 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.524 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.524 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.524 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.525 DEBUG:boto:Method: GET 2024-03-04T17:48:24.525 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.525 DEBUG:boto:Data: 2024-03-04T17:48:24.525 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.525 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.525 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.525 DEBUG:boto:Params: {} 2024-03-04T17:48:24.525 DEBUG:boto:Token: None 2024-03-04T17:48:24.525 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.526 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QZxaxpZXfXUo4HRTOblc084YsCI= 2024-03-04T17:48:24.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QZxaxpZXfXUo4HRTOblc084YsCI=', 'Content-Length': '0'} 2024-03-04T17:48:24.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018897cfaf2b2ec5f-0065e60968-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.572 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:24.572 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:24.572 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:24.572 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:24.573 DEBUG:boto:Method: GET 2024-03-04T17:48:24.573 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:24.573 DEBUG:boto:Data: 2024-03-04T17:48:24.573 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.573 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.573 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.573 DEBUG:boto:Params: {} 2024-03-04T17:48:24.573 DEBUG:boto:Token: None 2024-03-04T17:48:24.574 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/ 2024-03-04T17:48:24.574 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM= 2024-03-04T17:48:24.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM=', 'Content-Length': '0'} 2024-03-04T17:48:24.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001445a545ffb4eead-0065e60968-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.579 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.579 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:24.579 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:24.580 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:24.580 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:24.580 DEBUG:boto:Method: GET 2024-03-04T17:48:24.583 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:24.583 DEBUG:boto:Data: 2024-03-04T17:48:24.583 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.583 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.583 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.583 DEBUG:boto:Params: {} 2024-03-04T17:48:24.584 DEBUG:boto:Token: None 2024-03-04T17:48:24.584 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/ 2024-03-04T17:48:24.584 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM= 2024-03-04T17:48:24.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM=', 'Content-Length': '0'} 2024-03-04T17:48:24.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000273c5a01b9817479-0065e60968-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.632 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.632 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.633 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.633 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.633 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.633 DEBUG:boto:Data: 2024-03-04T17:48:24.633 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.633 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.633 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.633 DEBUG:boto:Params: {} 2024-03-04T17:48:24.633 DEBUG:boto:Token: None 2024-03-04T17:48:24.633 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.634 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ua2T5lEyeL/HjREfV+hHMvmcKXg= 2024-03-04T17:48:24.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ua2T5lEyeL/HjREfV+hHMvmcKXg=', 'Content-Length': '0'} 2024-03-04T17:48:24.641 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007983d03093a4c467-0065e60968-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.641 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.641 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.641 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.641 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.641 DEBUG:boto:Data: 2024-03-04T17:48:24.641 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.641 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.641 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.642 DEBUG:boto:Params: {} 2024-03-04T17:48:24.642 DEBUG:boto:Token: None 2024-03-04T17:48:24.642 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.642 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ua2T5lEyeL/HjREfV+hHMvmcKXg= 2024-03-04T17:48:24.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ua2T5lEyeL/HjREfV+hHMvmcKXg=', 'Content-Length': '0'} 2024-03-04T17:48:24.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7784e5498dfb1e6-0065e60968-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:48:24.649 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.649 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.649 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.649 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.649 DEBUG:boto:Method: GET 2024-03-04T17:48:24.649 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.649 DEBUG:boto:Data: 2024-03-04T17:48:24.649 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.649 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.649 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.649 DEBUG:boto:Params: {} 2024-03-04T17:48:24.650 DEBUG:boto:Token: None 2024-03-04T17:48:24.650 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.650 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QZxaxpZXfXUo4HRTOblc084YsCI= 2024-03-04T17:48:24.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QZxaxpZXfXUo4HRTOblc084YsCI=', 'Content-Length': '0'} 2024-03-04T17:48:24.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003061de0590e7b303-0065e60968-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.653 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.653 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:24.653 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.653 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.653 DEBUG:boto:Method: GET 2024-03-04T17:48:24.653 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.653 DEBUG:boto:Data: 2024-03-04T17:48:24.653 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.653 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.654 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.654 DEBUG:boto:Params: {} 2024-03-04T17:48:24.654 DEBUG:boto:Token: None 2024-03-04T17:48:24.654 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:48:24.654 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QZxaxpZXfXUo4HRTOblc084YsCI= 2024-03-04T17:48:24.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QZxaxpZXfXUo4HRTOblc084YsCI=', 'Content-Length': '0'} 2024-03-04T17:48:24.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004abbb281269aef8a-0065e60968-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.700 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:24.700 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:24.700 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:24.700 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:24.701 DEBUG:boto:Method: GET 2024-03-04T17:48:24.701 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:24.701 DEBUG:boto:Data: 2024-03-04T17:48:24.701 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.701 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.701 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.701 DEBUG:boto:Params: {} 2024-03-04T17:48:24.701 DEBUG:boto:Token: None 2024-03-04T17:48:24.701 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/ 2024-03-04T17:48:24.702 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM= 2024-03-04T17:48:24.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM=', 'Content-Length': '0'} 2024-03-04T17:48:24.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007770eaa14eb639f1-0065e60968-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.716 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.716 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:24.716 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:24.716 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:24.717 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:24.717 DEBUG:boto:Method: GET 2024-03-04T17:48:24.717 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:48:24.717 DEBUG:boto:Data: 2024-03-04T17:48:24.717 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.717 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.717 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.717 DEBUG:boto:Params: {} 2024-03-04T17:48:24.717 DEBUG:boto:Token: None 2024-03-04T17:48:24.717 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-21/ 2024-03-04T17:48:24.718 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM= 2024-03-04T17:48:24.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iE5utpHIS7MZmXav8aU9oxRfBFM=', 'Content-Length': '0'} 2024-03-04T17:48:24.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7f2a33d35ae2017-0065e60968-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.764 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.764 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-21 zones={test-zone3, test-zone2} 2024-03-04T17:48:24.765 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:24.765 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:24.765 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.765 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:48:24.765 DEBUG:boto:Data: 2024-03-04T17:48:24.765 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.765 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:24.765 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.766 DEBUG:boto:Params: {} 2024-03-04T17:48:24.766 DEBUG:boto:Token: None 2024-03-04T17:48:24.766 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-19/ 2024-03-04T17:48:24.766 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e0XoQ9XLWOas5pKZfy65oQ7zQHU= 2024-03-04T17:48:24.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e0XoQ9XLWOas5pKZfy65oQ7zQHU=', 'Content-Length': '0'} 2024-03-04T17:48:24.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065aef990ee4b0775-0065e60968-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.775 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:24.775 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:24.775 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T17:48:24.775 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T17:48:24.775 DEBUG:boto:Method: GET 2024-03-04T17:48:24.775 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T17:48:24.775 DEBUG:boto:Data: 2024-03-04T17:48:24.775 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.775 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:24.775 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.776 DEBUG:boto:Params: {} 2024-03-04T17:48:24.776 DEBUG:boto:Token: None 2024-03-04T17:48:24.776 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-19/?versions 2024-03-04T17:48:24.776 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LfnficvXppBayy/KOBe1njieBMM= 2024-03-04T17:48:24.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LfnficvXppBayy/KOBe1njieBMM=', 'Content-Length': '0'} 2024-03-04T17:48:24.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1859a4c4e96d4f5-0065e60968-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.783 DEBUG:boto:b'ppfahn-19obj-test-zone1999falseobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.783 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:24.784 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:48:24.784 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?acl 2024-03-04T17:48:24.784 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?acl 2024-03-04T17:48:24.784 DEBUG:boto:Method: PUT 2024-03-04T17:48:24.784 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?acl 2024-03-04T17:48:24.784 DEBUG:boto:Data: 2024-03-04T17:48:24.787 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T17:48:24.787 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:24.787 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.787 DEBUG:boto:Params: {} 2024-03-04T17:48:24.787 DEBUG:boto:Token: None 2024-03-04T17:48:24.787 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:48:24 GMT x-amz-acl:public-read /ppfahn-19/obj-test-zone1?acl 2024-03-04T17:48:24.788 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68wnDqB8nqNr26vV/dLI6QwSS1w= 2024-03-04T17:48:24.788 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68wnDqB8nqNr26vV/dLI6QwSS1w=', 'Content-Length': '0'} 2024-03-04T17:48:24.804 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003fe33cb53fc131cb-0065e60968-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.804 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:24.804 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:24.804 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.804 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:48:24.804 DEBUG:boto:Data: 2024-03-04T17:48:24.804 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.805 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.805 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.805 DEBUG:boto:Params: {} 2024-03-04T17:48:24.805 DEBUG:boto:Token: None 2024-03-04T17:48:24.805 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-19/ 2024-03-04T17:48:24.805 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e0XoQ9XLWOas5pKZfy65oQ7zQHU= 2024-03-04T17:48:24.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e0XoQ9XLWOas5pKZfy65oQ7zQHU=', 'Content-Length': '0'} 2024-03-04T17:48:24.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8ac11554e9f425d-0065e60968-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.821 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:24.821 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:24.821 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T17:48:24.821 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T17:48:24.821 DEBUG:boto:Method: GET 2024-03-04T17:48:24.821 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T17:48:24.821 DEBUG:boto:Data: 2024-03-04T17:48:24.822 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.822 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.822 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.822 DEBUG:boto:Params: {} 2024-03-04T17:48:24.822 DEBUG:boto:Token: None 2024-03-04T17:48:24.822 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-19/?versions 2024-03-04T17:48:24.822 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LfnficvXppBayy/KOBe1njieBMM= 2024-03-04T17:48:24.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LfnficvXppBayy/KOBe1njieBMM=', 'Content-Length': '0'} 2024-03-04T17:48:24.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b3f18cc4a276e3-0065e60968-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.847 DEBUG:boto:b'ppfahn-19obj-test-zone2999falseobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.847 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:24.847 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:48:24.847 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?acl 2024-03-04T17:48:24.848 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?acl 2024-03-04T17:48:24.848 DEBUG:boto:Method: PUT 2024-03-04T17:48:24.848 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?acl 2024-03-04T17:48:24.848 DEBUG:boto:Data: 2024-03-04T17:48:24.848 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T17:48:24.848 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.848 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.848 DEBUG:boto:Params: {} 2024-03-04T17:48:24.848 DEBUG:boto:Token: None 2024-03-04T17:48:24.848 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:48:24 GMT x-amz-acl:public-read /ppfahn-19/obj-test-zone2?acl 2024-03-04T17:48:24.849 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dpylItjhHa4qoa/G7P9vZMNrAYE= 2024-03-04T17:48:24.849 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dpylItjhHa4qoa/G7P9vZMNrAYE=', 'Content-Length': '0'} 2024-03-04T17:48:24.860 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f185e38a54276f69-0065e60968-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.860 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:24.860 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:24.860 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.860 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:48:24.860 DEBUG:boto:Data: 2024-03-04T17:48:24.860 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.860 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.860 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.861 DEBUG:boto:Params: {} 2024-03-04T17:48:24.861 DEBUG:boto:Token: None 2024-03-04T17:48:24.861 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-19/ 2024-03-04T17:48:24.861 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e0XoQ9XLWOas5pKZfy65oQ7zQHU= 2024-03-04T17:48:24.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e0XoQ9XLWOas5pKZfy65oQ7zQHU=', 'Content-Length': '0'} 2024-03-04T17:48:24.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057a8832411d2b4ea-0065e60968-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.864 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:48:24.864 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:48:24.864 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T17:48:24.864 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T17:48:24.864 DEBUG:boto:Method: GET 2024-03-04T17:48:24.864 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T17:48:24.864 DEBUG:boto:Data: 2024-03-04T17:48:24.865 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.865 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.865 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.865 DEBUG:boto:Params: {} 2024-03-04T17:48:24.865 DEBUG:boto:Token: None 2024-03-04T17:48:24.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-19/?versions 2024-03-04T17:48:24.865 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LfnficvXppBayy/KOBe1njieBMM= 2024-03-04T17:48:24.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LfnficvXppBayy/KOBe1njieBMM=', 'Content-Length': '0'} 2024-03-04T17:48:24.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cca034b1a1497230-0065e60968-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.868 DEBUG:boto:b'ppfahn-19obj-test-zone3999falseobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.868 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:24.868 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:48:24.869 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?acl 2024-03-04T17:48:24.869 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?acl 2024-03-04T17:48:24.869 DEBUG:boto:Method: PUT 2024-03-04T17:48:24.869 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?acl 2024-03-04T17:48:24.869 DEBUG:boto:Data: 2024-03-04T17:48:24.869 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T17:48:24.869 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:24.869 DEBUG:boto:Port: 8001 2024-03-04T17:48:24.869 DEBUG:boto:Params: {} 2024-03-04T17:48:24.869 DEBUG:boto:Token: None 2024-03-04T17:48:24.870 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:48:24 GMT x-amz-acl:public-read /ppfahn-19/obj-test-zone3?acl 2024-03-04T17:48:24.870 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bkl7cDM1+Hu3YHCx259hVgT26sM= 2024-03-04T17:48:24.870 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bkl7cDM1+Hu3YHCx259hVgT26sM=', 'Content-Length': '0'} 2024-03-04T17:48:24.876 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fcda7d75655f39f6-0065e60968-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.876 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:24.876 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:24.876 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.877 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:48:24.877 DEBUG:boto:Data: 2024-03-04T17:48:24.877 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.877 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:24.877 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.877 DEBUG:boto:Params: {} 2024-03-04T17:48:24.877 DEBUG:boto:Token: None 2024-03-04T17:48:24.877 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-20/ 2024-03-04T17:48:24.877 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QsV2Rfp9oNLSCC6Qvxv72b7w4aE= 2024-03-04T17:48:24.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QsV2Rfp9oNLSCC6Qvxv72b7w4aE=', 'Content-Length': '0'} 2024-03-04T17:48:24.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd3dd3d65749d566-0065e60968-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.880 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:24.881 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:24.881 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T17:48:24.881 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T17:48:24.881 DEBUG:boto:Method: GET 2024-03-04T17:48:24.881 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T17:48:24.881 DEBUG:boto:Data: 2024-03-04T17:48:24.881 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.881 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:24.881 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.881 DEBUG:boto:Params: {} 2024-03-04T17:48:24.881 DEBUG:boto:Token: None 2024-03-04T17:48:24.882 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-20/?versions 2024-03-04T17:48:24.882 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:84yTmRU8m3sl/vHZAYtwIA6ABAI= 2024-03-04T17:48:24.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:84yTmRU8m3sl/vHZAYtwIA6ABAI=', 'Content-Length': '0'} 2024-03-04T17:48:24.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002c7e8261994d17c-0065e60968-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.928 DEBUG:boto:b'ppfahn-20obj-test-zone1999falseobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.928 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:24.928 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:48:24.929 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?acl 2024-03-04T17:48:24.929 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?acl 2024-03-04T17:48:24.929 DEBUG:boto:Method: PUT 2024-03-04T17:48:24.929 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?acl 2024-03-04T17:48:24.929 DEBUG:boto:Data: 2024-03-04T17:48:24.929 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T17:48:24.929 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:24.929 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.929 DEBUG:boto:Params: {} 2024-03-04T17:48:24.929 DEBUG:boto:Token: None 2024-03-04T17:48:24.929 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:48:24 GMT x-amz-acl:public-read /ppfahn-20/obj-test-zone1?acl 2024-03-04T17:48:24.930 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AeWfdIReipW0cDyvK19QBwU/ew4= 2024-03-04T17:48:24.930 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AeWfdIReipW0cDyvK19QBwU/ew4=', 'Content-Length': '0'} 2024-03-04T17:48:24.943 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000156b031d738756a5-0065e60968-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.944 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:24.944 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:24.944 DEBUG:boto:Method: HEAD 2024-03-04T17:48:24.944 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:48:24.944 DEBUG:boto:Data: 2024-03-04T17:48:24.944 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.944 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.944 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.944 DEBUG:boto:Params: {} 2024-03-04T17:48:24.945 DEBUG:boto:Token: None 2024-03-04T17:48:24.945 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-20/ 2024-03-04T17:48:24.945 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QsV2Rfp9oNLSCC6Qvxv72b7w4aE= 2024-03-04T17:48:24.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QsV2Rfp9oNLSCC6Qvxv72b7w4aE=', 'Content-Length': '0'} 2024-03-04T17:48:24.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1d1befe3b613b3f-0065e60968-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.949 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:24.949 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:24.949 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T17:48:24.949 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T17:48:24.949 DEBUG:boto:Method: GET 2024-03-04T17:48:24.949 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T17:48:24.949 DEBUG:boto:Data: 2024-03-04T17:48:24.950 DEBUG:boto:Headers: {} 2024-03-04T17:48:24.950 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.950 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.950 DEBUG:boto:Params: {} 2024-03-04T17:48:24.950 DEBUG:boto:Token: None 2024-03-04T17:48:24.950 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:24 GMT /ppfahn-20/?versions 2024-03-04T17:48:24.950 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:84yTmRU8m3sl/vHZAYtwIA6ABAI= 2024-03-04T17:48:24.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:84yTmRU8m3sl/vHZAYtwIA6ABAI=', 'Content-Length': '0'} 2024-03-04T17:48:24.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000180f9082eb732932-0065e60968-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:24.996 DEBUG:boto:b'ppfahn-20obj-test-zone2999falseobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:24.996 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:24.997 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:48:24.997 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?acl 2024-03-04T17:48:24.997 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?acl 2024-03-04T17:48:24.997 DEBUG:boto:Method: PUT 2024-03-04T17:48:24.997 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?acl 2024-03-04T17:48:24.997 DEBUG:boto:Data: 2024-03-04T17:48:24.997 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T17:48:24.997 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:24.997 DEBUG:boto:Port: 8000 2024-03-04T17:48:24.997 DEBUG:boto:Params: {} 2024-03-04T17:48:24.998 DEBUG:boto:Token: None 2024-03-04T17:48:24.998 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:48:24 GMT x-amz-acl:public-read /ppfahn-20/obj-test-zone2?acl 2024-03-04T17:48:24.998 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8zzao9WjcAJxoWtY2Hyuv8TAJRE= 2024-03-04T17:48:24.998 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8zzao9WjcAJxoWtY2Hyuv8TAJRE=', 'Content-Length': '0'} 2024-03-04T17:48:25.012 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000017086db6f894b52c-0065e60969-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:25 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:25.012 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:25.012 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:25.012 DEBUG:boto:Method: HEAD 2024-03-04T17:48:25.012 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:48:25.012 DEBUG:boto:Data: 2024-03-04T17:48:25.012 DEBUG:boto:Headers: {} 2024-03-04T17:48:25.012 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:25.012 DEBUG:boto:Port: 8001 2024-03-04T17:48:25.013 DEBUG:boto:Params: {} 2024-03-04T17:48:25.013 DEBUG:boto:Token: None 2024-03-04T17:48:25.013 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:25 GMT /ppfahn-20/ 2024-03-04T17:48:25.013 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oSyrNcaaIHuvhjTZwpM2vGCQSro= 2024-03-04T17:48:25.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:25 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oSyrNcaaIHuvhjTZwpM2vGCQSro=', 'Content-Length': '0'} 2024-03-04T17:48:25.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9ef34a4a718729b-0065e60969-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:25 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:25.017 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:48:25.017 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:48:25.018 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T17:48:25.018 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T17:48:25.018 DEBUG:boto:Method: GET 2024-03-04T17:48:25.018 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T17:48:25.018 DEBUG:boto:Data: 2024-03-04T17:48:25.018 DEBUG:boto:Headers: {} 2024-03-04T17:48:25.018 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:25.018 DEBUG:boto:Port: 8001 2024-03-04T17:48:25.018 DEBUG:boto:Params: {} 2024-03-04T17:48:25.018 DEBUG:boto:Token: None 2024-03-04T17:48:25.018 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:25 GMT /ppfahn-20/?versions 2024-03-04T17:48:25.019 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZiFP9B78BfzDuafYcpY9MgjvYh4= 2024-03-04T17:48:25.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:25 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZiFP9B78BfzDuafYcpY9MgjvYh4=', 'Content-Length': '0'} 2024-03-04T17:48:25.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9b730844cca4d6d-0065e60969-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:25 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:25.068 DEBUG:boto:b'ppfahn-20obj-test-zone3999falseobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:25.068 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:25.069 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:48:25.069 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?acl 2024-03-04T17:48:25.069 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?acl 2024-03-04T17:48:25.069 DEBUG:boto:Method: PUT 2024-03-04T17:48:25.069 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?acl 2024-03-04T17:48:25.069 DEBUG:boto:Data: 2024-03-04T17:48:25.069 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T17:48:25.069 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:25.069 DEBUG:boto:Port: 8001 2024-03-04T17:48:25.069 DEBUG:boto:Params: {} 2024-03-04T17:48:25.070 DEBUG:boto:Token: None 2024-03-04T17:48:25.070 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:48:25 GMT x-amz-acl:public-read /ppfahn-20/obj-test-zone3?acl 2024-03-04T17:48:25.070 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vnWNRkCGPmLRW5QhzSiO9kfM0ik= 2024-03-04T17:48:25.070 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:25 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vnWNRkCGPmLRW5QhzSiO9kfM0ik=', 'Content-Length': '0'} 2024-03-04T17:48:25.084 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000042c377acd2bf2870-0065e60969-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:25 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:25.084 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:25.084 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:25.084 DEBUG:boto:Method: HEAD 2024-03-04T17:48:25.084 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:48:25.085 DEBUG:boto:Data: 2024-03-04T17:48:25.085 DEBUG:boto:Headers: {} 2024-03-04T17:48:25.085 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:25.085 DEBUG:boto:Port: 8000 2024-03-04T17:48:25.085 DEBUG:boto:Params: {} 2024-03-04T17:48:25.085 DEBUG:boto:Token: None 2024-03-04T17:48:25.085 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:25 GMT /ppfahn-21/ 2024-03-04T17:48:25.085 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rtfPbVb4IlMfmlQH+pR+0coAJU0= 2024-03-04T17:48:25.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:25 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rtfPbVb4IlMfmlQH+pR+0coAJU0=', 'Content-Length': '0'} 2024-03-04T17:48:25.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054d6cd74713b5ecb-0065e60969-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:25 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:25.090 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:25.090 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:25.090 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T17:48:25.090 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T17:48:25.090 DEBUG:boto:Method: GET 2024-03-04T17:48:25.090 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T17:48:25.091 DEBUG:boto:Data: 2024-03-04T17:48:25.091 DEBUG:boto:Headers: {} 2024-03-04T17:48:25.091 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:25.091 DEBUG:boto:Port: 8000 2024-03-04T17:48:25.091 DEBUG:boto:Params: {} 2024-03-04T17:48:25.091 DEBUG:boto:Token: None 2024-03-04T17:48:25.091 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:25 GMT /ppfahn-21/?versions 2024-03-04T17:48:25.091 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MM10HiUrUEHuYnnFvBe8Q03Sawk= 2024-03-04T17:48:25.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:25 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MM10HiUrUEHuYnnFvBe8Q03Sawk=', 'Content-Length': '0'} 2024-03-04T17:48:25.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052bc05aea1434b66-0065e60969-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:25 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:25.096 DEBUG:boto:b'ppfahn-21obj-test-zone1999falseobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:25.096 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:25.096 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:48:25.097 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?acl 2024-03-04T17:48:25.097 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?acl 2024-03-04T17:48:25.097 DEBUG:boto:Method: PUT 2024-03-04T17:48:25.097 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?acl 2024-03-04T17:48:25.097 DEBUG:boto:Data: 2024-03-04T17:48:25.097 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T17:48:25.097 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:48:25.097 DEBUG:boto:Port: 8000 2024-03-04T17:48:25.097 DEBUG:boto:Params: {} 2024-03-04T17:48:25.097 DEBUG:boto:Token: None 2024-03-04T17:48:25.097 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:48:25 GMT x-amz-acl:public-read /ppfahn-21/obj-test-zone1?acl 2024-03-04T17:48:25.098 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GsJ9LUxFs116Z7zbgyhuaZ1jf/0= 2024-03-04T17:48:25.098 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:25 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GsJ9LUxFs116Z7zbgyhuaZ1jf/0=', 'Content-Length': '0'} 2024-03-04T17:48:25.110 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000026ec907dce52a7ff-0065e60969-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:25 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:25.110 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:25.110 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:25.110 DEBUG:boto:Method: HEAD 2024-03-04T17:48:25.111 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:48:25.111 DEBUG:boto:Data: 2024-03-04T17:48:25.111 DEBUG:boto:Headers: {} 2024-03-04T17:48:25.111 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:25.111 DEBUG:boto:Port: 8000 2024-03-04T17:48:25.111 DEBUG:boto:Params: {} 2024-03-04T17:48:25.111 DEBUG:boto:Token: None 2024-03-04T17:48:25.111 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:25 GMT /ppfahn-21/ 2024-03-04T17:48:25.112 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rtfPbVb4IlMfmlQH+pR+0coAJU0= 2024-03-04T17:48:25.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:25 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rtfPbVb4IlMfmlQH+pR+0coAJU0=', 'Content-Length': '0'} 2024-03-04T17:48:25.116 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000551863ec14300419-0065e60969-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:25 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:25.116 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:25.116 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:25.116 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T17:48:25.116 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T17:48:25.117 DEBUG:boto:Method: GET 2024-03-04T17:48:25.117 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T17:48:25.117 DEBUG:boto:Data: 2024-03-04T17:48:25.117 DEBUG:boto:Headers: {} 2024-03-04T17:48:25.117 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:25.117 DEBUG:boto:Port: 8000 2024-03-04T17:48:25.117 DEBUG:boto:Params: {} 2024-03-04T17:48:25.117 DEBUG:boto:Token: None 2024-03-04T17:48:25.117 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:25 GMT /ppfahn-21/?versions 2024-03-04T17:48:25.118 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MM10HiUrUEHuYnnFvBe8Q03Sawk= 2024-03-04T17:48:25.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:25 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MM10HiUrUEHuYnnFvBe8Q03Sawk=', 'Content-Length': '0'} 2024-03-04T17:48:25.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047ecbf4e7055c890-0065e60969-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:25 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:25.122 DEBUG:boto:b'ppfahn-21obj-test-zone2999falseobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:25.123 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:25.123 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:48:25.123 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?acl 2024-03-04T17:48:25.123 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?acl 2024-03-04T17:48:25.123 DEBUG:boto:Method: PUT 2024-03-04T17:48:25.123 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?acl 2024-03-04T17:48:25.123 DEBUG:boto:Data: 2024-03-04T17:48:25.123 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T17:48:25.124 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:48:25.124 DEBUG:boto:Port: 8000 2024-03-04T17:48:25.124 DEBUG:boto:Params: {} 2024-03-04T17:48:25.124 DEBUG:boto:Token: None 2024-03-04T17:48:25.124 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:48:25 GMT x-amz-acl:public-read /ppfahn-21/obj-test-zone2?acl 2024-03-04T17:48:25.124 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WuDFRN5Bp1a2a/TBvgOVikcAMT0= 2024-03-04T17:48:25.124 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:25 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WuDFRN5Bp1a2a/TBvgOVikcAMT0=', 'Content-Length': '0'} 2024-03-04T17:48:25.137 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000babfd85b272e29b2-0065e60969-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:25 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:25.137 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:25.138 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:25.138 DEBUG:boto:Method: HEAD 2024-03-04T17:48:25.138 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:48:25.138 DEBUG:boto:Data: 2024-03-04T17:48:25.138 DEBUG:boto:Headers: {} 2024-03-04T17:48:25.138 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:25.138 DEBUG:boto:Port: 8001 2024-03-04T17:48:25.138 DEBUG:boto:Params: {} 2024-03-04T17:48:25.139 DEBUG:boto:Token: None 2024-03-04T17:48:25.139 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:48:25 GMT /ppfahn-21/ 2024-03-04T17:48:25.139 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rtfPbVb4IlMfmlQH+pR+0coAJU0= 2024-03-04T17:48:25.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:25 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rtfPbVb4IlMfmlQH+pR+0coAJU0=', 'Content-Length': '0'} 2024-03-04T17:48:25.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041b3f3bd93bfb744-0065e60969-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:25 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:25.144 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:48:25.144 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:48:25.144 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T17:48:25.144 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T17:48:25.144 DEBUG:boto:Method: GET 2024-03-04T17:48:25.144 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T17:48:25.145 DEBUG:boto:Data: 2024-03-04T17:48:25.145 DEBUG:boto:Headers: {} 2024-03-04T17:48:25.145 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:25.145 DEBUG:boto:Port: 8001 2024-03-04T17:48:25.145 DEBUG:boto:Params: {} 2024-03-04T17:48:25.145 DEBUG:boto:Token: None 2024-03-04T17:48:25.145 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:48:25 GMT /ppfahn-21/?versions 2024-03-04T17:48:25.145 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MM10HiUrUEHuYnnFvBe8Q03Sawk= 2024-03-04T17:48:25.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:25 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MM10HiUrUEHuYnnFvBe8Q03Sawk=', 'Content-Length': '0'} 2024-03-04T17:48:25.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037658765f7a3119a-0065e60969-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:48:25 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:25.150 DEBUG:boto:b'ppfahn-21obj-test-zone3999falseobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:48:25.151 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:25.151 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:48:25.151 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?acl 2024-03-04T17:48:25.151 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?acl 2024-03-04T17:48:25.151 DEBUG:boto:Method: PUT 2024-03-04T17:48:25.152 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?acl 2024-03-04T17:48:25.152 DEBUG:boto:Data: 2024-03-04T17:48:25.152 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T17:48:25.152 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:48:25.152 DEBUG:boto:Port: 8001 2024-03-04T17:48:25.152 DEBUG:boto:Params: {} 2024-03-04T17:48:25.152 DEBUG:boto:Token: None 2024-03-04T17:48:25.153 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:48:25 GMT x-amz-acl:public-read /ppfahn-21/obj-test-zone3?acl 2024-03-04T17:48:25.153 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Q0MdiYBfjbW0ns9EYcbm5MnjTiw= 2024-03-04T17:48:25.153 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:48:25 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Q0MdiYBfjbW0ns9EYcbm5MnjTiw=', 'Content-Length': '0'} 2024-03-04T17:48:25.166 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fbbebe80e0a2e8a6-0065e60969-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:48:25 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:48:25.166 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-19 2024-03-04T17:48:25.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:25.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:25.167 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:25.217 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:25.217 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:25.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:25.280+0000 7f2330af0980 1 waiting for incremental sync to catch up: 2024-03-04T17:48:25.280 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000010.30.6, , 00000000003.38.10, , 00000000007.25.10, , , , , ] 2024-03-04T17:48:25.280 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000011.33.6, , 00000000003.38.10, , 00000000007.25.10, , , , , ] 2024-03-04T17:48:30.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:30.284+0000 7f2330af0980 1 waiting for incremental sync to catch up: 2024-03-04T17:48:30.283 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000010.30.6, , 00000000003.38.10, , 00000000007.25.10, , , , , ] 2024-03-04T17:48:30.283 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000011.33.6, , 00000000003.38.10, , 00000000007.25.10, , , , , ] 2024-03-04T17:48:35.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:35.284+0000 7f2330af0980 1 bucket sync caught up with source: 2024-03-04T17:48:35.285 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000011.33.6, , 00000000003.38.10, , 00000000007.25.10, , , , , ] 2024-03-04T17:48:35.285 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000011.33.6, , 00000000003.38.10, , 00000000007.25.10, , , , , ] 2024-03-04T17:48:35.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:48:35.284+0000 7f2330af0980 0 bucket checkpoint complete 2024-03-04T17:48:35.327 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-19 2024-03-04T17:48:35.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:35.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:35.327 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:35.378 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:35.378 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:35.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:35.444+0000 7f5008439980 1 bucket sync caught up with source: 2024-03-04T17:48:35.439 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000011.33.6, , 00000000003.38.10, , 00000000007.25.10, , , , , ] 2024-03-04T17:48:35.439 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000011.33.6, , 00000000003.38.10, , 00000000007.25.10, , , , , ] 2024-03-04T17:48:35.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:35.444+0000 7f5008439980 0 bucket checkpoint complete 2024-03-04T17:48:35.461 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-19 2024-03-04T17:48:35.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:35.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:48:35.461 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:48:35.502 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:35.502 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:35.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:35.568+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:48:35.561 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:48:35.561 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:48:40.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:40.568+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:48:40.564 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:48:40.564 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:48:45.566 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:45.572+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:48:45.567 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:48:45.567 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:48:50.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:50.572+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:48:50.569 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:48:50.569 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:48:55.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:48:55.576+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:48:55.572 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:48:55.572 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:00.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:49:00.579+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:49:00.574 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:00.574 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:05.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:49:05.579+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:49:05.577 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:05.577 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:10.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:49:10.583+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:49:10.579 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:10.579 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:15.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:49:15.587+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:49:15.582 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:15.582 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:20.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:49:20.587+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:49:20.584 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:20.585 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:25.586 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:49:25.591+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:49:25.587 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:25.587 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:30.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:49:30.598+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:49:30.594 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:30.594 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:49:35.598+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:49:35.596 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:35.597 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:40.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:49:40.602+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:49:40.599 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:40.599 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:45.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:49:45.602+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:49:45.601 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:45.601 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:50.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:49:50.606+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:49:50.603 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:50.604 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:55.606 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:49:55.610+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:49:55.607 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:49:55.607 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:00.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:00.609+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:50:00.609 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:00.609 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:05.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:05.613+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:50:05.611 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:05.611 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:10.613 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:10.613+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:50:10.614 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:10.614 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:15.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:15.617+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:50:15.617 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:15.638 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:20.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:20.621+0000 7f3f3eed5980 1 waiting for incremental sync to catch up: 2024-03-04T17:50:20.620 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000010.54.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:20.620 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:25.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:25.625+0000 7f3f3eed5980 1 bucket sync caught up with source: 2024-03-04T17:50:25.623 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:25.623 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:25.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:25.625+0000 7f3f3eed5980 0 bucket checkpoint complete 2024-03-04T17:50:25.661 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-19 2024-03-04T17:50:25.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:25.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:25.662 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:25.755 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:25.755 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:25.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:25.817+0000 7fbaedaeb980 1 bucket sync caught up with source: 2024-03-04T17:50:25.816 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:25.816 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.17.10, , 00000000011.57.6, , 00000000007.51.10, , , , , ] 2024-03-04T17:50:25.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:25.817+0000 7fbaedaeb980 0 bucket checkpoint complete 2024-03-04T17:50:25.839 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-19 2024-03-04T17:50:25.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:25.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:25.839 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:25.881 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:25.881 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:25.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:25.937+0000 7fa1cc3b9980 1 bucket sync caught up with source: 2024-03-04T17:50:25.938 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.45.10, , 00000000003.96.10, , 00000000015.183.6, , , , , ] 2024-03-04T17:50:25.938 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.45.10, , 00000000003.96.10, , 00000000015.183.6, , , , , ] 2024-03-04T17:50:25.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:25.937+0000 7fa1cc3b9980 0 bucket checkpoint complete 2024-03-04T17:50:25.959 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-19 2024-03-04T17:50:25.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:25.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:25.960 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:26.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:26.075+0000 7f55b8c83980 1 bucket sync caught up with source: 2024-03-04T17:50:26.073 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000003.45.10, , 00000000003.96.10, , 00000000015.183.6, , , , , ] 2024-03-04T17:50:26.073 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000003.45.10, , 00000000003.96.10, , 00000000015.183.6, , , , , ] 2024-03-04T17:50:26.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:26.075+0000 7f55b8c83980 0 bucket checkpoint complete 2024-03-04T17:50:26.095 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-19 zones={test-zone2, test-zone1} 2024-03-04T17:50:26.096 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.096 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.096 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.096 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:50:26.096 DEBUG:boto:Data: 2024-03-04T17:50:26.096 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.096 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.096 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.096 DEBUG:boto:Params: {} 2024-03-04T17:50:26.097 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:50:26.097 DEBUG:boto:Token: None 2024-03-04T17:50:26.097 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.097 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GvGHAaP5PqJTpgmkjtqh4vlypdM= 2024-03-04T17:50:26.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GvGHAaP5PqJTpgmkjtqh4vlypdM=', 'Content-Length': '0'} 2024-03-04T17:50:26.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dda91fae9e1d2971-0065e609e2-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.102 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.102 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.102 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.102 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:50:26.102 DEBUG:boto:Data: 2024-03-04T17:50:26.102 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.102 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.102 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.103 DEBUG:boto:Params: {} 2024-03-04T17:50:26.103 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:50:26.103 DEBUG:boto:Token: None 2024-03-04T17:50:26.103 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.103 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GvGHAaP5PqJTpgmkjtqh4vlypdM= 2024-03-04T17:50:26.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GvGHAaP5PqJTpgmkjtqh4vlypdM=', 'Content-Length': '0'} 2024-03-04T17:50:26.111 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb2a16b983cf4179-0065e609e2-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.111 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:50:26.111 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.111 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.111 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:26.112 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:26.112 DEBUG:boto:Method: GET 2024-03-04T17:50:26.112 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:26.112 DEBUG:boto:Data: 2024-03-04T17:50:26.112 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.112 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.112 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.112 DEBUG:boto:Params: {} 2024-03-04T17:50:26.113 DEBUG:boto:Token: None 2024-03-04T17:50:26.113 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/?versions 2024-03-04T17:50:26.113 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QIjT51DhS45/zAHs9G0p7EftO9g= 2024-03-04T17:50:26.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QIjT51DhS45/zAHs9G0p7EftO9g=', 'Content-Length': '0'} 2024-03-04T17:50:26.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059e5cf471580a254-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.116 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.117 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:26.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:26.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:26.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:26.118 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:50:26.118 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.118 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.118 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:26.118 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:26.118 DEBUG:boto:Method: GET 2024-03-04T17:50:26.118 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:26.118 DEBUG:boto:Data: 2024-03-04T17:50:26.118 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.118 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.118 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.118 DEBUG:boto:Params: {} 2024-03-04T17:50:26.119 DEBUG:boto:Token: None 2024-03-04T17:50:26.119 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/?versions 2024-03-04T17:50:26.119 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QIjT51DhS45/zAHs9G0p7EftO9g= 2024-03-04T17:50:26.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QIjT51DhS45/zAHs9G0p7EftO9g=', 'Content-Length': '0'} 2024-03-04T17:50:26.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000729dc6a7de47f48-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.125 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.126 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:26.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:26.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:26.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:26.126 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.126 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.126 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:26.126 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:26.126 DEBUG:boto:Method: GET 2024-03-04T17:50:26.126 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:26.127 DEBUG:boto:Data: 2024-03-04T17:50:26.127 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.127 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.127 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.127 DEBUG:boto:Params: {} 2024-03-04T17:50:26.127 DEBUG:boto:Token: None 2024-03-04T17:50:26.127 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/?versions 2024-03-04T17:50:26.127 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QIjT51DhS45/zAHs9G0p7EftO9g= 2024-03-04T17:50:26.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QIjT51DhS45/zAHs9G0p7EftO9g=', 'Content-Length': '0'} 2024-03-04T17:50:26.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0f1e48ecde3aef7-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.130 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.131 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.131 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.131 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:26.131 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:26.131 DEBUG:boto:Method: GET 2024-03-04T17:50:26.131 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:26.131 DEBUG:boto:Data: 2024-03-04T17:50:26.131 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.131 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.132 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.132 DEBUG:boto:Params: {} 2024-03-04T17:50:26.132 DEBUG:boto:Token: None 2024-03-04T17:50:26.132 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/?versions 2024-03-04T17:50:26.132 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QIjT51DhS45/zAHs9G0p7EftO9g= 2024-03-04T17:50:26.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QIjT51DhS45/zAHs9G0p7EftO9g=', 'Content-Length': '0'} 2024-03-04T17:50:26.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b42d71a5564d5058-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.137 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:26.138 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:26.138 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:26.138 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.138 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.138 DEBUG:boto:Method: GET 2024-03-04T17:50:26.138 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.138 DEBUG:boto:Data: 2024-03-04T17:50:26.138 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.138 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.138 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.138 DEBUG:boto:Params: {} 2024-03-04T17:50:26.139 DEBUG:boto:Token: None 2024-03-04T17:50:26.139 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.139 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PUvq3CIN3GyeqobI2vKKOqagaDE= 2024-03-04T17:50:26.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PUvq3CIN3GyeqobI2vKKOqagaDE=', 'Content-Length': '0'} 2024-03-04T17:50:26.141 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050b8d9da6ceb220e-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.141 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:26.141 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:26.141 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.141 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.142 DEBUG:boto:Method: GET 2024-03-04T17:50:26.142 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.142 DEBUG:boto:Data: 2024-03-04T17:50:26.142 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.142 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.142 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.142 DEBUG:boto:Params: {} 2024-03-04T17:50:26.142 DEBUG:boto:Token: None 2024-03-04T17:50:26.142 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.143 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PUvq3CIN3GyeqobI2vKKOqagaDE= 2024-03-04T17:50:26.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PUvq3CIN3GyeqobI2vKKOqagaDE=', 'Content-Length': '0'} 2024-03-04T17:50:26.145 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d443e93e1c23fa3-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.145 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:26.146 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:26.146 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.146 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.146 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.146 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.146 DEBUG:boto:Data: 2024-03-04T17:50:26.146 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.146 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.146 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.146 DEBUG:boto:Params: {} 2024-03-04T17:50:26.147 DEBUG:boto:Token: None 2024-03-04T17:50:26.147 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.147 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5KlVlC8zhhav0nCQzfJfQIaZ4d4= 2024-03-04T17:50:26.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5KlVlC8zhhav0nCQzfJfQIaZ4d4=', 'Content-Length': '0'} 2024-03-04T17:50:26.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a73de3f2838bb455-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.149 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:26.149 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:26.149 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.149 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.149 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.149 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.149 DEBUG:boto:Data: 2024-03-04T17:50:26.149 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.149 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.150 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.150 DEBUG:boto:Params: {} 2024-03-04T17:50:26.150 DEBUG:boto:Token: None 2024-03-04T17:50:26.150 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.150 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5KlVlC8zhhav0nCQzfJfQIaZ4d4= 2024-03-04T17:50:26.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5KlVlC8zhhav0nCQzfJfQIaZ4d4=', 'Content-Length': '0'} 2024-03-04T17:50:26.153 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e9bec363af242728-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:26.153 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:26.153 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:26.153 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.153 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.154 DEBUG:boto:Method: GET 2024-03-04T17:50:26.154 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.154 DEBUG:boto:Data: 2024-03-04T17:50:26.154 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.154 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.154 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.154 DEBUG:boto:Params: {} 2024-03-04T17:50:26.154 DEBUG:boto:Token: None 2024-03-04T17:50:26.154 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.155 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PUvq3CIN3GyeqobI2vKKOqagaDE= 2024-03-04T17:50:26.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PUvq3CIN3GyeqobI2vKKOqagaDE=', 'Content-Length': '0'} 2024-03-04T17:50:26.157 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d603f3d1e2fd498-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.157 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:26.157 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:26.157 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.157 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.157 DEBUG:boto:Method: GET 2024-03-04T17:50:26.157 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.157 DEBUG:boto:Data: 2024-03-04T17:50:26.157 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.158 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.158 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.158 DEBUG:boto:Params: {} 2024-03-04T17:50:26.158 DEBUG:boto:Token: None 2024-03-04T17:50:26.158 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.158 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PUvq3CIN3GyeqobI2vKKOqagaDE= 2024-03-04T17:50:26.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PUvq3CIN3GyeqobI2vKKOqagaDE=', 'Content-Length': '0'} 2024-03-04T17:50:26.161 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d8b0405253e03e1a-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.161 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.162 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.162 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:26.162 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:26.162 DEBUG:boto:Method: GET 2024-03-04T17:50:26.162 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:50:26.162 DEBUG:boto:Data: 2024-03-04T17:50:26.162 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.162 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.162 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.162 DEBUG:boto:Params: {} 2024-03-04T17:50:26.162 DEBUG:boto:Token: None 2024-03-04T17:50:26.163 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.163 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko= 2024-03-04T17:50:26.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko=', 'Content-Length': '0'} 2024-03-04T17:50:26.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c26d6ba3b1b91fae-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.208 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.208 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.209 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.209 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:26.209 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:26.209 DEBUG:boto:Method: GET 2024-03-04T17:50:26.209 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:50:26.209 DEBUG:boto:Data: 2024-03-04T17:50:26.209 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.209 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.210 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.210 DEBUG:boto:Params: {} 2024-03-04T17:50:26.210 DEBUG:boto:Token: None 2024-03-04T17:50:26.210 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.210 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko= 2024-03-04T17:50:26.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko=', 'Content-Length': '0'} 2024-03-04T17:50:26.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8dc31209ccce159-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.215 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.216 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:26.216 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:26.216 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.216 DEBUG:boto:Path: /ppfahn-19/dummy 2024-03-04T17:50:26.216 DEBUG:boto:Data: 2024-03-04T17:50:26.216 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.216 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.216 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.216 DEBUG:boto:Params: {} 2024-03-04T17:50:26.216 DEBUG:boto:Token: None 2024-03-04T17:50:26.217 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/dummy 2024-03-04T17:50:26.217 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ufC66btP4n/gfpw/qx+l0powf5o= 2024-03-04T17:50:26.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ufC66btP4n/gfpw/qx+l0powf5o=', 'Content-Length': '0'} 2024-03-04T17:50:26.220 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000641585ddf1bafb13-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.221 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:26.221 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:26.221 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.221 DEBUG:boto:Path: /ppfahn-19/dummy 2024-03-04T17:50:26.221 DEBUG:boto:Data: 2024-03-04T17:50:26.221 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.221 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.221 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.221 DEBUG:boto:Params: {} 2024-03-04T17:50:26.221 DEBUG:boto:Token: None 2024-03-04T17:50:26.222 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/dummy 2024-03-04T17:50:26.222 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ufC66btP4n/gfpw/qx+l0powf5o= 2024-03-04T17:50:26.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ufC66btP4n/gfpw/qx+l0powf5o=', 'Content-Length': '0'} 2024-03-04T17:50:26.225 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a832d247a04e18e2-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:26.226 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:26.226 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:26.226 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.226 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.226 DEBUG:boto:Method: GET 2024-03-04T17:50:26.226 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.226 DEBUG:boto:Data: 2024-03-04T17:50:26.226 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.226 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.227 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.227 DEBUG:boto:Params: {} 2024-03-04T17:50:26.227 DEBUG:boto:Token: None 2024-03-04T17:50:26.227 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.227 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PUvq3CIN3GyeqobI2vKKOqagaDE= 2024-03-04T17:50:26.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PUvq3CIN3GyeqobI2vKKOqagaDE=', 'Content-Length': '0'} 2024-03-04T17:50:26.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c94ead177367a9e-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.230 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:26.230 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:26.231 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.231 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.231 DEBUG:boto:Method: GET 2024-03-04T17:50:26.231 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.231 DEBUG:boto:Data: 2024-03-04T17:50:26.231 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.231 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.231 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.231 DEBUG:boto:Params: {} 2024-03-04T17:50:26.231 DEBUG:boto:Token: None 2024-03-04T17:50:26.231 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:26.232 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PUvq3CIN3GyeqobI2vKKOqagaDE= 2024-03-04T17:50:26.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PUvq3CIN3GyeqobI2vKKOqagaDE=', 'Content-Length': '0'} 2024-03-04T17:50:26.234 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001e61c4cecff931a6-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.235 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.235 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.235 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:26.235 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:26.235 DEBUG:boto:Method: GET 2024-03-04T17:50:26.235 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:50:26.235 DEBUG:boto:Data: 2024-03-04T17:50:26.235 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.235 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.235 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.235 DEBUG:boto:Params: {} 2024-03-04T17:50:26.235 DEBUG:boto:Token: None 2024-03-04T17:50:26.236 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.236 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko= 2024-03-04T17:50:26.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko=', 'Content-Length': '0'} 2024-03-04T17:50:26.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c30d88b4cee13a3c-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.280 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.281 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.281 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.281 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:26.281 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:26.281 DEBUG:boto:Method: GET 2024-03-04T17:50:26.281 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:50:26.281 DEBUG:boto:Data: 2024-03-04T17:50:26.281 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.281 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.281 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.281 DEBUG:boto:Params: {} 2024-03-04T17:50:26.282 DEBUG:boto:Token: None 2024-03-04T17:50:26.282 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.282 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko= 2024-03-04T17:50:26.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko=', 'Content-Length': '0'} 2024-03-04T17:50:26.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3f4dbe3d1510847-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.287 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:26.288 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.288 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.288 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.288 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.288 DEBUG:boto:Method: GET 2024-03-04T17:50:26.288 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.288 DEBUG:boto:Data: 2024-03-04T17:50:26.288 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.289 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.289 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.289 DEBUG:boto:Params: {} 2024-03-04T17:50:26.289 DEBUG:boto:Token: None 2024-03-04T17:50:26.289 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.289 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YBcvacQR7+Z13ry+WrOz9kmqnOo= 2024-03-04T17:50:26.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YBcvacQR7+Z13ry+WrOz9kmqnOo=', 'Content-Length': '0'} 2024-03-04T17:50:26.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffbbe9b714ee4dff-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.336 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.336 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.336 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.336 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.336 DEBUG:boto:Method: GET 2024-03-04T17:50:26.337 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.337 DEBUG:boto:Data: 2024-03-04T17:50:26.337 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.337 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.337 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.337 DEBUG:boto:Params: {} 2024-03-04T17:50:26.337 DEBUG:boto:Token: None 2024-03-04T17:50:26.337 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.338 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YBcvacQR7+Z13ry+WrOz9kmqnOo= 2024-03-04T17:50:26.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YBcvacQR7+Z13ry+WrOz9kmqnOo=', 'Content-Length': '0'} 2024-03-04T17:50:26.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3af288352606a22-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.341 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.341 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.341 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.341 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.342 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.342 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.342 DEBUG:boto:Data: 2024-03-04T17:50:26.342 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.342 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.342 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.342 DEBUG:boto:Params: {} 2024-03-04T17:50:26.342 DEBUG:boto:Token: None 2024-03-04T17:50:26.342 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.342 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dAxBfgPjUurmzh1kjqZ9gYmSH5k= 2024-03-04T17:50:26.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dAxBfgPjUurmzh1kjqZ9gYmSH5k=', 'Content-Length': '0'} 2024-03-04T17:50:26.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a939bc0d75f77c2-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.345 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.346 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.346 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.346 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.346 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.346 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.346 DEBUG:boto:Data: 2024-03-04T17:50:26.346 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.346 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.346 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.346 DEBUG:boto:Params: {} 2024-03-04T17:50:26.346 DEBUG:boto:Token: None 2024-03-04T17:50:26.347 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.347 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dAxBfgPjUurmzh1kjqZ9gYmSH5k= 2024-03-04T17:50:26.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dAxBfgPjUurmzh1kjqZ9gYmSH5k=', 'Content-Length': '0'} 2024-03-04T17:50:26.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025f122655c4e4a68-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:26.350 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.350 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.350 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.350 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.350 DEBUG:boto:Method: GET 2024-03-04T17:50:26.350 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.350 DEBUG:boto:Data: 2024-03-04T17:50:26.350 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.350 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.350 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.351 DEBUG:boto:Params: {} 2024-03-04T17:50:26.351 DEBUG:boto:Token: None 2024-03-04T17:50:26.351 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.351 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YBcvacQR7+Z13ry+WrOz9kmqnOo= 2024-03-04T17:50:26.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YBcvacQR7+Z13ry+WrOz9kmqnOo=', 'Content-Length': '0'} 2024-03-04T17:50:26.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007900fa9bfa880974-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.400 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.400 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.400 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.400 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.401 DEBUG:boto:Method: GET 2024-03-04T17:50:26.401 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.401 DEBUG:boto:Data: 2024-03-04T17:50:26.401 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.401 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.401 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.401 DEBUG:boto:Params: {} 2024-03-04T17:50:26.401 DEBUG:boto:Token: None 2024-03-04T17:50:26.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.402 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YBcvacQR7+Z13ry+WrOz9kmqnOo= 2024-03-04T17:50:26.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YBcvacQR7+Z13ry+WrOz9kmqnOo=', 'Content-Length': '0'} 2024-03-04T17:50:26.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf5dc47aeca921bf-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.405 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.405 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.405 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:26.405 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:26.405 DEBUG:boto:Method: GET 2024-03-04T17:50:26.405 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:26.405 DEBUG:boto:Data: 2024-03-04T17:50:26.405 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.405 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.406 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.406 DEBUG:boto:Params: {} 2024-03-04T17:50:26.406 DEBUG:boto:Token: None 2024-03-04T17:50:26.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.406 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko= 2024-03-04T17:50:26.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko=', 'Content-Length': '0'} 2024-03-04T17:50:26.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d17e47eb2d51724d-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.410 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.411 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.411 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.411 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:26.411 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:26.411 DEBUG:boto:Method: GET 2024-03-04T17:50:26.411 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:26.411 DEBUG:boto:Data: 2024-03-04T17:50:26.411 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.412 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.412 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.412 DEBUG:boto:Params: {} 2024-03-04T17:50:26.412 DEBUG:boto:Token: None 2024-03-04T17:50:26.412 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.412 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko= 2024-03-04T17:50:26.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko=', 'Content-Length': '0'} 2024-03-04T17:50:26.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d23ebcd960c94b84-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.418 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.418 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.418 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.418 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.418 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.418 DEBUG:boto:Data: 2024-03-04T17:50:26.418 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.419 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.419 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.419 DEBUG:boto:Params: {} 2024-03-04T17:50:26.419 DEBUG:boto:Token: None 2024-03-04T17:50:26.419 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.419 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:66NXVC5o/jQWoXJS9RAQN6KCcUs= 2024-03-04T17:50:26.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:66NXVC5o/jQWoXJS9RAQN6KCcUs=', 'Content-Length': '0'} 2024-03-04T17:50:26.423 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa00f58a41f0ac1c-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.423 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.423 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.423 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.423 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.423 DEBUG:boto:Data: 2024-03-04T17:50:26.424 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.424 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.424 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.424 DEBUG:boto:Params: {} 2024-03-04T17:50:26.424 DEBUG:boto:Token: None 2024-03-04T17:50:26.424 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.424 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:66NXVC5o/jQWoXJS9RAQN6KCcUs= 2024-03-04T17:50:26.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:66NXVC5o/jQWoXJS9RAQN6KCcUs=', 'Content-Length': '0'} 2024-03-04T17:50:26.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a24022c83859196d-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:26.428 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.429 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.429 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.429 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.429 DEBUG:boto:Method: GET 2024-03-04T17:50:26.429 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.429 DEBUG:boto:Data: 2024-03-04T17:50:26.429 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.429 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.429 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.429 DEBUG:boto:Params: {} 2024-03-04T17:50:26.429 DEBUG:boto:Token: None 2024-03-04T17:50:26.430 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.430 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YBcvacQR7+Z13ry+WrOz9kmqnOo= 2024-03-04T17:50:26.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YBcvacQR7+Z13ry+WrOz9kmqnOo=', 'Content-Length': '0'} 2024-03-04T17:50:26.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b538e3077ea13dbe-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.476 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.476 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:26.476 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.476 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.476 DEBUG:boto:Method: GET 2024-03-04T17:50:26.477 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.477 DEBUG:boto:Data: 2024-03-04T17:50:26.477 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.477 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.477 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.477 DEBUG:boto:Params: {} 2024-03-04T17:50:26.477 DEBUG:boto:Token: None 2024-03-04T17:50:26.477 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:26.477 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YBcvacQR7+Z13ry+WrOz9kmqnOo= 2024-03-04T17:50:26.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YBcvacQR7+Z13ry+WrOz9kmqnOo=', 'Content-Length': '0'} 2024-03-04T17:50:26.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bda1bc069414e6d-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.481 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.481 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.481 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:26.481 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:26.481 DEBUG:boto:Method: GET 2024-03-04T17:50:26.481 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:26.481 DEBUG:boto:Data: 2024-03-04T17:50:26.481 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.481 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.482 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.482 DEBUG:boto:Params: {} 2024-03-04T17:50:26.482 DEBUG:boto:Token: None 2024-03-04T17:50:26.482 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.482 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko= 2024-03-04T17:50:26.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko=', 'Content-Length': '0'} 2024-03-04T17:50:26.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a114900e95abc21b-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.487 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.487 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.487 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.487 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:26.487 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:26.487 DEBUG:boto:Method: GET 2024-03-04T17:50:26.488 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:26.488 DEBUG:boto:Data: 2024-03-04T17:50:26.488 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.488 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.488 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.488 DEBUG:boto:Params: {} 2024-03-04T17:50:26.488 DEBUG:boto:Token: None 2024-03-04T17:50:26.488 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.488 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko= 2024-03-04T17:50:26.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko=', 'Content-Length': '0'} 2024-03-04T17:50:26.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000475be28b257decf8-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.536 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:26.537 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.537 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.537 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.537 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.538 DEBUG:boto:Method: GET 2024-03-04T17:50:26.538 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.538 DEBUG:boto:Data: 2024-03-04T17:50:26.538 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.538 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.538 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.539 DEBUG:boto:Params: {} 2024-03-04T17:50:26.539 DEBUG:boto:Token: None 2024-03-04T17:50:26.539 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.539 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UJ1aSuJtNla5eLJVr+A6No5OnbY= 2024-03-04T17:50:26.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UJ1aSuJtNla5eLJVr+A6No5OnbY=', 'Content-Length': '0'} 2024-03-04T17:50:26.543 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000676fd11e9e56c0a8-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.543 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.543 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.543 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.543 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.544 DEBUG:boto:Method: GET 2024-03-04T17:50:26.544 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.544 DEBUG:boto:Data: 2024-03-04T17:50:26.544 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.544 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.544 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.544 DEBUG:boto:Params: {} 2024-03-04T17:50:26.544 DEBUG:boto:Token: None 2024-03-04T17:50:26.544 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.545 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UJ1aSuJtNla5eLJVr+A6No5OnbY= 2024-03-04T17:50:26.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UJ1aSuJtNla5eLJVr+A6No5OnbY=', 'Content-Length': '0'} 2024-03-04T17:50:26.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9340a7b8a6bfb8a-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.548 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.548 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.548 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.548 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.549 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.549 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.549 DEBUG:boto:Data: 2024-03-04T17:50:26.549 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.549 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.550 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.550 DEBUG:boto:Params: {} 2024-03-04T17:50:26.550 DEBUG:boto:Token: None 2024-03-04T17:50:26.550 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.550 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vPrKnImFq9zeLMil0t4Dh5cNLXE= 2024-03-04T17:50:26.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vPrKnImFq9zeLMil0t4Dh5cNLXE=', 'Content-Length': '0'} 2024-03-04T17:50:26.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b50c9a85c8d3a87-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.553 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.553 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.553 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.553 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.554 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.554 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.554 DEBUG:boto:Data: 2024-03-04T17:50:26.554 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.554 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.554 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.554 DEBUG:boto:Params: {} 2024-03-04T17:50:26.554 DEBUG:boto:Token: None 2024-03-04T17:50:26.554 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.554 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vPrKnImFq9zeLMil0t4Dh5cNLXE= 2024-03-04T17:50:26.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vPrKnImFq9zeLMil0t4Dh5cNLXE=', 'Content-Length': '0'} 2024-03-04T17:50:26.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f12586e5e8e7cf6-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:26.558 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.558 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.558 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.559 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.559 DEBUG:boto:Method: GET 2024-03-04T17:50:26.559 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.559 DEBUG:boto:Data: 2024-03-04T17:50:26.559 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.559 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.559 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.559 DEBUG:boto:Params: {} 2024-03-04T17:50:26.559 DEBUG:boto:Token: None 2024-03-04T17:50:26.559 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.560 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UJ1aSuJtNla5eLJVr+A6No5OnbY= 2024-03-04T17:50:26.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UJ1aSuJtNla5eLJVr+A6No5OnbY=', 'Content-Length': '0'} 2024-03-04T17:50:26.563 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce74a8b760120fcd-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.563 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.563 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.563 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.563 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.564 DEBUG:boto:Method: GET 2024-03-04T17:50:26.564 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.564 DEBUG:boto:Data: 2024-03-04T17:50:26.564 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.564 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.564 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.564 DEBUG:boto:Params: {} 2024-03-04T17:50:26.564 DEBUG:boto:Token: None 2024-03-04T17:50:26.564 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.565 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UJ1aSuJtNla5eLJVr+A6No5OnbY= 2024-03-04T17:50:26.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UJ1aSuJtNla5eLJVr+A6No5OnbY=', 'Content-Length': '0'} 2024-03-04T17:50:26.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009aa25017c1e9d918-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.608 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.608 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.608 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:26.609 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:26.609 DEBUG:boto:Method: GET 2024-03-04T17:50:26.609 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:26.609 DEBUG:boto:Data: 2024-03-04T17:50:26.609 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.609 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.610 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.610 DEBUG:boto:Params: {} 2024-03-04T17:50:26.610 DEBUG:boto:Token: None 2024-03-04T17:50:26.610 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.610 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko= 2024-03-04T17:50:26.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko=', 'Content-Length': '0'} 2024-03-04T17:50:26.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049c86b85e2f5f903-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.616 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.616 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.616 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.617 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:26.617 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:26.617 DEBUG:boto:Method: GET 2024-03-04T17:50:26.617 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:26.617 DEBUG:boto:Data: 2024-03-04T17:50:26.617 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.617 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.617 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.618 DEBUG:boto:Params: {} 2024-03-04T17:50:26.618 DEBUG:boto:Token: None 2024-03-04T17:50:26.618 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.618 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko= 2024-03-04T17:50:26.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko=', 'Content-Length': '0'} 2024-03-04T17:50:26.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000744b59722c8ef07b-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.668 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.668 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.669 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.669 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.669 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.669 DEBUG:boto:Data: 2024-03-04T17:50:26.669 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.669 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.669 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.669 DEBUG:boto:Params: {} 2024-03-04T17:50:26.669 DEBUG:boto:Token: None 2024-03-04T17:50:26.669 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.670 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RsDYztWRN2xVY+gOxrHM2BHo+HY= 2024-03-04T17:50:26.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RsDYztWRN2xVY+gOxrHM2BHo+HY=', 'Content-Length': '0'} 2024-03-04T17:50:26.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed02da6d789cf166-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.674 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.674 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.674 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.674 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.674 DEBUG:boto:Data: 2024-03-04T17:50:26.674 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.674 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.674 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.675 DEBUG:boto:Params: {} 2024-03-04T17:50:26.675 DEBUG:boto:Token: None 2024-03-04T17:50:26.675 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.675 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RsDYztWRN2xVY+gOxrHM2BHo+HY= 2024-03-04T17:50:26.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RsDYztWRN2xVY+gOxrHM2BHo+HY=', 'Content-Length': '0'} 2024-03-04T17:50:26.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cb48c8d701ed18f-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:26.679 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.679 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.679 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.679 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.679 DEBUG:boto:Method: GET 2024-03-04T17:50:26.680 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.680 DEBUG:boto:Data: 2024-03-04T17:50:26.680 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.680 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.680 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.680 DEBUG:boto:Params: {} 2024-03-04T17:50:26.680 DEBUG:boto:Token: None 2024-03-04T17:50:26.680 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.680 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UJ1aSuJtNla5eLJVr+A6No5OnbY= 2024-03-04T17:50:26.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UJ1aSuJtNla5eLJVr+A6No5OnbY=', 'Content-Length': '0'} 2024-03-04T17:50:26.683 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c0b1e420bb345f6-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.728 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.728 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:26.728 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.728 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.728 DEBUG:boto:Method: GET 2024-03-04T17:50:26.729 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.729 DEBUG:boto:Data: 2024-03-04T17:50:26.729 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.729 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.729 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.729 DEBUG:boto:Params: {} 2024-03-04T17:50:26.729 DEBUG:boto:Token: None 2024-03-04T17:50:26.729 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:26.729 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UJ1aSuJtNla5eLJVr+A6No5OnbY= 2024-03-04T17:50:26.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UJ1aSuJtNla5eLJVr+A6No5OnbY=', 'Content-Length': '0'} 2024-03-04T17:50:26.733 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ae439f6efa27338-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.780 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.780 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.780 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:26.781 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:26.781 DEBUG:boto:Method: GET 2024-03-04T17:50:26.781 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:26.781 DEBUG:boto:Data: 2024-03-04T17:50:26.781 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.781 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.781 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.781 DEBUG:boto:Params: {} 2024-03-04T17:50:26.781 DEBUG:boto:Token: None 2024-03-04T17:50:26.781 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.782 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko= 2024-03-04T17:50:26.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko=', 'Content-Length': '0'} 2024-03-04T17:50:26.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c1be90aa853c5ba-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.786 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.787 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:26.787 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:26.787 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:26.787 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:26.787 DEBUG:boto:Method: GET 2024-03-04T17:50:26.787 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:26.788 DEBUG:boto:Data: 2024-03-04T17:50:26.788 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.788 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.788 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.788 DEBUG:boto:Params: {} 2024-03-04T17:50:26.788 DEBUG:boto:Token: None 2024-03-04T17:50:26.788 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/ 2024-03-04T17:50:26.788 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko= 2024-03-04T17:50:26.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yhk+Y4+dK9aBd4v9zuzkG3QZ+Ko=', 'Content-Length': '0'} 2024-03-04T17:50:26.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c87b6e5a9d2993ca-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.836 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:26.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:26.837 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:26.837 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:26.837 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.837 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.837 DEBUG:boto:Method: GET 2024-03-04T17:50:26.837 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.837 DEBUG:boto:Data: 2024-03-04T17:50:26.837 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.837 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.837 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.837 DEBUG:boto:Params: {} 2024-03-04T17:50:26.838 DEBUG:boto:Token: None 2024-03-04T17:50:26.838 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.838 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pl5kc/HiSYdP5jtxynMadqW8n9c= 2024-03-04T17:50:26.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pl5kc/HiSYdP5jtxynMadqW8n9c=', 'Content-Length': '0'} 2024-03-04T17:50:26.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d71adf60ee16a22b-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.884 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:26.884 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:26.884 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.884 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.884 DEBUG:boto:Method: GET 2024-03-04T17:50:26.885 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.885 DEBUG:boto:Data: 2024-03-04T17:50:26.885 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.885 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.885 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.885 DEBUG:boto:Params: {} 2024-03-04T17:50:26.885 DEBUG:boto:Token: None 2024-03-04T17:50:26.885 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.885 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pl5kc/HiSYdP5jtxynMadqW8n9c= 2024-03-04T17:50:26.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pl5kc/HiSYdP5jtxynMadqW8n9c=', 'Content-Length': '0'} 2024-03-04T17:50:26.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000804452b78a161ea7-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.932 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:26.932 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:26.932 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.932 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.932 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.933 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.933 DEBUG:boto:Data: 2024-03-04T17:50:26.933 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.933 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.933 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.933 DEBUG:boto:Params: {} 2024-03-04T17:50:26.933 DEBUG:boto:Token: None 2024-03-04T17:50:26.933 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.933 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OHOs7fuDF0sC24xZTgG0I6As9N4= 2024-03-04T17:50:26.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OHOs7fuDF0sC24xZTgG0I6As9N4=', 'Content-Length': '0'} 2024-03-04T17:50:26.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091473228ed41db69-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.937 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:26.937 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:26.937 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.937 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.937 DEBUG:boto:Method: HEAD 2024-03-04T17:50:26.937 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.937 DEBUG:boto:Data: 2024-03-04T17:50:26.937 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.937 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.937 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.937 DEBUG:boto:Params: {} 2024-03-04T17:50:26.938 DEBUG:boto:Token: None 2024-03-04T17:50:26.938 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.938 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OHOs7fuDF0sC24xZTgG0I6As9N4= 2024-03-04T17:50:26.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OHOs7fuDF0sC24xZTgG0I6As9N4=', 'Content-Length': '0'} 2024-03-04T17:50:26.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e54fc8911cb5ea43-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:26.941 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:26.941 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:26.941 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.941 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.941 DEBUG:boto:Method: GET 2024-03-04T17:50:26.941 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.941 DEBUG:boto:Data: 2024-03-04T17:50:26.942 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.942 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:26.942 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.942 DEBUG:boto:Params: {} 2024-03-04T17:50:26.942 DEBUG:boto:Token: None 2024-03-04T17:50:26.942 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.942 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pl5kc/HiSYdP5jtxynMadqW8n9c= 2024-03-04T17:50:26.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pl5kc/HiSYdP5jtxynMadqW8n9c=', 'Content-Length': '0'} 2024-03-04T17:50:26.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fea4a1585a2ac46a-0065e609e2-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:26.988 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:26.988 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:26.988 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.988 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.989 DEBUG:boto:Method: GET 2024-03-04T17:50:26.989 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.989 DEBUG:boto:Data: 2024-03-04T17:50:26.989 DEBUG:boto:Headers: {} 2024-03-04T17:50:26.989 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:26.989 DEBUG:boto:Port: 8000 2024-03-04T17:50:26.989 DEBUG:boto:Params: {} 2024-03-04T17:50:26.990 DEBUG:boto:Token: None 2024-03-04T17:50:26.990 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:26 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:26.990 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pl5kc/HiSYdP5jtxynMadqW8n9c= 2024-03-04T17:50:26.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pl5kc/HiSYdP5jtxynMadqW8n9c=', 'Content-Length': '0'} 2024-03-04T17:50:26.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079b4382d66d94abb-0065e609e2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.036 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.036 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.036 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:27.036 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:27.036 DEBUG:boto:Method: GET 2024-03-04T17:50:27.037 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:27.037 DEBUG:boto:Data: 2024-03-04T17:50:27.037 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.037 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:27.037 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.037 DEBUG:boto:Params: {} 2024-03-04T17:50:27.037 DEBUG:boto:Token: None 2024-03-04T17:50:27.037 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.037 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009616cbf08cf56a83-0065e609e3-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.084 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.085 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.085 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.085 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:27.085 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:27.085 DEBUG:boto:Method: GET 2024-03-04T17:50:27.085 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:27.085 DEBUG:boto:Data: 2024-03-04T17:50:27.085 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.085 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.085 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.085 DEBUG:boto:Params: {} 2024-03-04T17:50:27.086 DEBUG:boto:Token: None 2024-03-04T17:50:27.086 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.086 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000983cd7d12408fec2-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.132 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.132 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:27.132 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:27.133 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.133 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:50:27.133 DEBUG:boto:Data: 2024-03-04T17:50:27.133 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.133 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:27.133 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.133 DEBUG:boto:Params: {} 2024-03-04T17:50:27.133 DEBUG:boto:Token: None 2024-03-04T17:50:27.133 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:50:27.133 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cFVcGby1l6mYseozkPqgawHq5/U= 2024-03-04T17:50:27.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cFVcGby1l6mYseozkPqgawHq5/U=', 'Content-Length': '0'} 2024-03-04T17:50:27.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b6ca78fe7287396-0065e609e3-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.137 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:27.138 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:27.138 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.138 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:50:27.138 DEBUG:boto:Data: 2024-03-04T17:50:27.138 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.138 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.138 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.138 DEBUG:boto:Params: {} 2024-03-04T17:50:27.139 DEBUG:boto:Token: None 2024-03-04T17:50:27.139 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:50:27.139 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cFVcGby1l6mYseozkPqgawHq5/U= 2024-03-04T17:50:27.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cFVcGby1l6mYseozkPqgawHq5/U=', 'Content-Length': '0'} 2024-03-04T17:50:27.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041a1e35dd025db71-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:27.143 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:27.143 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:27.143 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:27.143 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:27.143 DEBUG:boto:Method: GET 2024-03-04T17:50:27.143 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:27.143 DEBUG:boto:Data: 2024-03-04T17:50:27.144 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.144 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:27.144 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.144 DEBUG:boto:Params: {} 2024-03-04T17:50:27.144 DEBUG:boto:Token: None 2024-03-04T17:50:27.144 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:27.144 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wUNBWFJgM7EM9rAmghvQPM6uq2U= 2024-03-04T17:50:27.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wUNBWFJgM7EM9rAmghvQPM6uq2U=', 'Content-Length': '0'} 2024-03-04T17:50:27.147 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ce080729aa214a4-0065e609e3-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.188 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:27.188 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:27.189 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:27.189 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:27.189 DEBUG:boto:Method: GET 2024-03-04T17:50:27.189 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:27.189 DEBUG:boto:Data: 2024-03-04T17:50:27.189 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.190 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.190 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.190 DEBUG:boto:Params: {} 2024-03-04T17:50:27.190 DEBUG:boto:Token: None 2024-03-04T17:50:27.190 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:27.191 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wUNBWFJgM7EM9rAmghvQPM6uq2U= 2024-03-04T17:50:27.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wUNBWFJgM7EM9rAmghvQPM6uq2U=', 'Content-Length': '0'} 2024-03-04T17:50:27.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd14995c72721d7f-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.236 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.237 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.237 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:27.237 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:27.237 DEBUG:boto:Method: GET 2024-03-04T17:50:27.237 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:27.237 DEBUG:boto:Data: 2024-03-04T17:50:27.237 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.237 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:27.238 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.238 DEBUG:boto:Params: {} 2024-03-04T17:50:27.238 DEBUG:boto:Token: None 2024-03-04T17:50:27.238 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.238 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2c7abf4fab96c1a-0065e609e3-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.243 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.243 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.243 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.243 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:27.244 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:27.244 DEBUG:boto:Method: GET 2024-03-04T17:50:27.244 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:27.244 DEBUG:boto:Data: 2024-03-04T17:50:27.244 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.244 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.244 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.245 DEBUG:boto:Params: {} 2024-03-04T17:50:27.245 DEBUG:boto:Token: None 2024-03-04T17:50:27.245 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.245 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000169ba600aef0d821-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.292 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.292 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-19 zones={test-zone2, test-zone1} 2024-03-04T17:50:27.293 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-19 zones={test-zone3, test-zone1} 2024-03-04T17:50:27.293 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.293 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.293 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.293 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:50:27.293 DEBUG:boto:Data: 2024-03-04T17:50:27.293 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.293 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.293 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.293 DEBUG:boto:Params: {} 2024-03-04T17:50:27.294 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T17:50:27.294 DEBUG:boto:Token: None 2024-03-04T17:50:27.294 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.294 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:C1IYeL8lxP4XP0ib6ithrlqGjjU= 2024-03-04T17:50:27.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:C1IYeL8lxP4XP0ib6ithrlqGjjU=', 'Content-Length': '0'} 2024-03-04T17:50:27.302 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c508805c4e47d8e-0065e609e3-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.302 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.302 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.302 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.303 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:50:27.303 DEBUG:boto:Data: 2024-03-04T17:50:27.303 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.303 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.303 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.303 DEBUG:boto:Params: {} 2024-03-04T17:50:27.303 DEBUG:boto:Token: None 2024-03-04T17:50:27.303 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.303 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:C1IYeL8lxP4XP0ib6ithrlqGjjU= 2024-03-04T17:50:27.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:C1IYeL8lxP4XP0ib6ithrlqGjjU=', 'Content-Length': '0'} 2024-03-04T17:50:27.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b79594b81c59eeaf-0065e609e3-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:50:27.308 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.308 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.308 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:27.308 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:27.308 DEBUG:boto:Method: GET 2024-03-04T17:50:27.308 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:27.308 DEBUG:boto:Data: 2024-03-04T17:50:27.308 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.309 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.309 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.309 DEBUG:boto:Params: {} 2024-03-04T17:50:27.309 DEBUG:boto:Token: None 2024-03-04T17:50:27.309 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/?versions 2024-03-04T17:50:27.309 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cAd5DSkx47TT1XCRxtnnMfaKiE0= 2024-03-04T17:50:27.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cAd5DSkx47TT1XCRxtnnMfaKiE0=', 'Content-Length': '0'} 2024-03-04T17:50:27.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2c3571b220cc35b-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.315 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.316 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:27.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:27.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:27.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:27.316 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:50:27.316 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.316 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.317 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:27.317 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:27.317 DEBUG:boto:Method: GET 2024-03-04T17:50:27.317 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:27.317 DEBUG:boto:Data: 2024-03-04T17:50:27.317 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.317 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.317 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.317 DEBUG:boto:Params: {} 2024-03-04T17:50:27.317 DEBUG:boto:Token: None 2024-03-04T17:50:27.317 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/?versions 2024-03-04T17:50:27.318 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cAd5DSkx47TT1XCRxtnnMfaKiE0= 2024-03-04T17:50:27.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cAd5DSkx47TT1XCRxtnnMfaKiE0=', 'Content-Length': '0'} 2024-03-04T17:50:27.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056c0b980b2a401df-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.322 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.323 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:27.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:27.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:27.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:27.323 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.323 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.324 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:27.324 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:27.324 DEBUG:boto:Method: GET 2024-03-04T17:50:27.324 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:27.324 DEBUG:boto:Data: 2024-03-04T17:50:27.324 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.324 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.324 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.324 DEBUG:boto:Params: {} 2024-03-04T17:50:27.324 DEBUG:boto:Token: None 2024-03-04T17:50:27.325 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/?versions 2024-03-04T17:50:27.325 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cAd5DSkx47TT1XCRxtnnMfaKiE0= 2024-03-04T17:50:27.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cAd5DSkx47TT1XCRxtnnMfaKiE0=', 'Content-Length': '0'} 2024-03-04T17:50:27.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007287d8ca9f3607c1-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.329 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.330 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.330 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.330 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:27.330 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:27.330 DEBUG:boto:Method: GET 2024-03-04T17:50:27.330 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:27.331 DEBUG:boto:Data: 2024-03-04T17:50:27.331 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.331 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.331 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.331 DEBUG:boto:Params: {} 2024-03-04T17:50:27.331 DEBUG:boto:Token: None 2024-03-04T17:50:27.331 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/?versions 2024-03-04T17:50:27.331 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cAd5DSkx47TT1XCRxtnnMfaKiE0= 2024-03-04T17:50:27.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cAd5DSkx47TT1XCRxtnnMfaKiE0=', 'Content-Length': '0'} 2024-03-04T17:50:27.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec75b834174d9fb6-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.337 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:27.337 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:27.337 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:27.337 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.337 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.337 DEBUG:boto:Method: GET 2024-03-04T17:50:27.338 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.338 DEBUG:boto:Data: 2024-03-04T17:50:27.338 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.338 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.338 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.338 DEBUG:boto:Params: {} 2024-03-04T17:50:27.338 DEBUG:boto:Token: None 2024-03-04T17:50:27.338 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.338 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2exavrzapRnUq8fhByVI6cNL6Og= 2024-03-04T17:50:27.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2exavrzapRnUq8fhByVI6cNL6Og=', 'Content-Length': '0'} 2024-03-04T17:50:27.341 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dcedd2d80fa72f30-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.341 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:27.342 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:27.342 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.342 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.342 DEBUG:boto:Method: GET 2024-03-04T17:50:27.342 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.342 DEBUG:boto:Data: 2024-03-04T17:50:27.342 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.342 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.342 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.342 DEBUG:boto:Params: {} 2024-03-04T17:50:27.342 DEBUG:boto:Token: None 2024-03-04T17:50:27.343 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.343 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2exavrzapRnUq8fhByVI6cNL6Og= 2024-03-04T17:50:27.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2exavrzapRnUq8fhByVI6cNL6Og=', 'Content-Length': '0'} 2024-03-04T17:50:27.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a5a0705c79e28751-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.346 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:27.346 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:27.346 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.346 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.346 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.346 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.347 DEBUG:boto:Data: 2024-03-04T17:50:27.347 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.347 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.347 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.347 DEBUG:boto:Params: {} 2024-03-04T17:50:27.347 DEBUG:boto:Token: None 2024-03-04T17:50:27.347 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.347 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qHpXyuxnQ/U5p1vyphJBflVnNXo= 2024-03-04T17:50:27.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qHpXyuxnQ/U5p1vyphJBflVnNXo=', 'Content-Length': '0'} 2024-03-04T17:50:27.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d26f31ed3ab45f7e-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.350 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:27.350 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:27.350 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.351 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.351 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.351 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.351 DEBUG:boto:Data: 2024-03-04T17:50:27.351 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.351 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.351 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.351 DEBUG:boto:Params: {} 2024-03-04T17:50:27.351 DEBUG:boto:Token: None 2024-03-04T17:50:27.351 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.351 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qHpXyuxnQ/U5p1vyphJBflVnNXo= 2024-03-04T17:50:27.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qHpXyuxnQ/U5p1vyphJBflVnNXo=', 'Content-Length': '0'} 2024-03-04T17:50:27.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db74e4144083dd05-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:27.355 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:27.355 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:27.355 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.355 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.355 DEBUG:boto:Method: GET 2024-03-04T17:50:27.355 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.355 DEBUG:boto:Data: 2024-03-04T17:50:27.355 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.355 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.355 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.356 DEBUG:boto:Params: {} 2024-03-04T17:50:27.356 DEBUG:boto:Token: None 2024-03-04T17:50:27.356 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.356 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2exavrzapRnUq8fhByVI6cNL6Og= 2024-03-04T17:50:27.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2exavrzapRnUq8fhByVI6cNL6Og=', 'Content-Length': '0'} 2024-03-04T17:50:27.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c863da8585b2d920-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.359 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:27.359 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:27.359 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.359 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.359 DEBUG:boto:Method: GET 2024-03-04T17:50:27.359 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.360 DEBUG:boto:Data: 2024-03-04T17:50:27.360 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.360 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.360 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.360 DEBUG:boto:Params: {} 2024-03-04T17:50:27.360 DEBUG:boto:Token: None 2024-03-04T17:50:27.360 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.360 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2exavrzapRnUq8fhByVI6cNL6Og= 2024-03-04T17:50:27.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2exavrzapRnUq8fhByVI6cNL6Og=', 'Content-Length': '0'} 2024-03-04T17:50:27.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c784f288348d1074-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.363 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.363 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.364 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:27.364 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:27.364 DEBUG:boto:Method: GET 2024-03-04T17:50:27.364 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:50:27.364 DEBUG:boto:Data: 2024-03-04T17:50:27.364 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.364 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.364 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.364 DEBUG:boto:Params: {} 2024-03-04T17:50:27.364 DEBUG:boto:Token: None 2024-03-04T17:50:27.365 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.365 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eed276ac3b9418d2-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.416 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.416 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.417 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.417 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:27.417 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:27.417 DEBUG:boto:Method: GET 2024-03-04T17:50:27.417 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:50:27.417 DEBUG:boto:Data: 2024-03-04T17:50:27.417 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.417 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.417 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.417 DEBUG:boto:Params: {} 2024-03-04T17:50:27.418 DEBUG:boto:Token: None 2024-03-04T17:50:27.418 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.418 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bdfdcf49cc9a940-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.423 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.423 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:27.424 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:27.424 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.424 DEBUG:boto:Path: /ppfahn-19/dummy 2024-03-04T17:50:27.424 DEBUG:boto:Data: 2024-03-04T17:50:27.424 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.424 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.424 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.424 DEBUG:boto:Params: {} 2024-03-04T17:50:27.424 DEBUG:boto:Token: None 2024-03-04T17:50:27.425 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/dummy 2024-03-04T17:50:27.425 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E0tl2nmScXz+jRw7C6cn/nrLYJc= 2024-03-04T17:50:27.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E0tl2nmScXz+jRw7C6cn/nrLYJc=', 'Content-Length': '0'} 2024-03-04T17:50:27.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000089b945521e095883-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.429 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:27.429 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:27.429 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.429 DEBUG:boto:Path: /ppfahn-19/dummy 2024-03-04T17:50:27.430 DEBUG:boto:Data: 2024-03-04T17:50:27.430 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.430 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.430 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.430 DEBUG:boto:Params: {} 2024-03-04T17:50:27.430 DEBUG:boto:Token: None 2024-03-04T17:50:27.430 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/dummy 2024-03-04T17:50:27.430 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E0tl2nmScXz+jRw7C6cn/nrLYJc= 2024-03-04T17:50:27.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E0tl2nmScXz+jRw7C6cn/nrLYJc=', 'Content-Length': '0'} 2024-03-04T17:50:27.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092141e940578f429-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:27.434 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:27.434 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:27.434 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.434 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.435 DEBUG:boto:Method: GET 2024-03-04T17:50:27.435 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.435 DEBUG:boto:Data: 2024-03-04T17:50:27.435 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.435 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.435 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.435 DEBUG:boto:Params: {} 2024-03-04T17:50:27.435 DEBUG:boto:Token: None 2024-03-04T17:50:27.435 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.435 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2exavrzapRnUq8fhByVI6cNL6Og= 2024-03-04T17:50:27.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2exavrzapRnUq8fhByVI6cNL6Og=', 'Content-Length': '0'} 2024-03-04T17:50:27.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ce85e5c63765431-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.439 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:27.439 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:27.439 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.439 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.439 DEBUG:boto:Method: GET 2024-03-04T17:50:27.439 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.439 DEBUG:boto:Data: 2024-03-04T17:50:27.439 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.439 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.439 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.439 DEBUG:boto:Params: {} 2024-03-04T17:50:27.440 DEBUG:boto:Token: None 2024-03-04T17:50:27.440 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:27.440 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2exavrzapRnUq8fhByVI6cNL6Og= 2024-03-04T17:50:27.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2exavrzapRnUq8fhByVI6cNL6Og=', 'Content-Length': '0'} 2024-03-04T17:50:27.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f61fa876585f2459-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.443 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.443 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.443 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:27.443 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:27.443 DEBUG:boto:Method: GET 2024-03-04T17:50:27.443 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:50:27.443 DEBUG:boto:Data: 2024-03-04T17:50:27.443 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.443 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.443 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.444 DEBUG:boto:Params: {} 2024-03-04T17:50:27.444 DEBUG:boto:Token: None 2024-03-04T17:50:27.444 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.444 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f65bc9762ebd536-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.449 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.450 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.450 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.450 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:27.450 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:27.450 DEBUG:boto:Method: GET 2024-03-04T17:50:27.450 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:50:27.450 DEBUG:boto:Data: 2024-03-04T17:50:27.450 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.450 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.450 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.450 DEBUG:boto:Params: {} 2024-03-04T17:50:27.451 DEBUG:boto:Token: None 2024-03-04T17:50:27.451 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.451 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8115689144eef30-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.456 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:27.457 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.457 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.457 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.457 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.457 DEBUG:boto:Method: GET 2024-03-04T17:50:27.458 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.458 DEBUG:boto:Data: 2024-03-04T17:50:27.458 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.458 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.458 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.458 DEBUG:boto:Params: {} 2024-03-04T17:50:27.458 DEBUG:boto:Token: None 2024-03-04T17:50:27.458 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.458 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FqixrXzlGlkFUB9YpTyG750T/8M= 2024-03-04T17:50:27.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FqixrXzlGlkFUB9YpTyG750T/8M=', 'Content-Length': '0'} 2024-03-04T17:50:27.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025e9e6d2a6bb1a88-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.504 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.504 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.504 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.504 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.504 DEBUG:boto:Method: GET 2024-03-04T17:50:27.504 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.505 DEBUG:boto:Data: 2024-03-04T17:50:27.505 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.505 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.505 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.505 DEBUG:boto:Params: {} 2024-03-04T17:50:27.505 DEBUG:boto:Token: None 2024-03-04T17:50:27.505 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.505 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FqixrXzlGlkFUB9YpTyG750T/8M= 2024-03-04T17:50:27.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FqixrXzlGlkFUB9YpTyG750T/8M=', 'Content-Length': '0'} 2024-03-04T17:50:27.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae083348809fac28-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.508 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.509 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.509 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.509 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.509 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.509 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.509 DEBUG:boto:Data: 2024-03-04T17:50:27.509 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.509 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.509 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.509 DEBUG:boto:Params: {} 2024-03-04T17:50:27.509 DEBUG:boto:Token: None 2024-03-04T17:50:27.510 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.510 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Ta1dHBRGugwBiMAdbo2Y3SIkvmg= 2024-03-04T17:50:27.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Ta1dHBRGugwBiMAdbo2Y3SIkvmg=', 'Content-Length': '0'} 2024-03-04T17:50:27.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000886a30fd45c5f799-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.513 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.513 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.513 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.513 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.513 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.513 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.513 DEBUG:boto:Data: 2024-03-04T17:50:27.514 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.514 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.514 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.514 DEBUG:boto:Params: {} 2024-03-04T17:50:27.514 DEBUG:boto:Token: None 2024-03-04T17:50:27.514 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.514 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Ta1dHBRGugwBiMAdbo2Y3SIkvmg= 2024-03-04T17:50:27.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Ta1dHBRGugwBiMAdbo2Y3SIkvmg=', 'Content-Length': '0'} 2024-03-04T17:50:27.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004eabacee971f5fbb-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:27.517 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.518 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.518 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.518 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.518 DEBUG:boto:Method: GET 2024-03-04T17:50:27.518 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.518 DEBUG:boto:Data: 2024-03-04T17:50:27.518 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.518 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.518 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.518 DEBUG:boto:Params: {} 2024-03-04T17:50:27.518 DEBUG:boto:Token: None 2024-03-04T17:50:27.519 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.519 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FqixrXzlGlkFUB9YpTyG750T/8M= 2024-03-04T17:50:27.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FqixrXzlGlkFUB9YpTyG750T/8M=', 'Content-Length': '0'} 2024-03-04T17:50:27.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce751ea89c8e7a77-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.564 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.564 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.564 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.564 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.564 DEBUG:boto:Method: GET 2024-03-04T17:50:27.564 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.565 DEBUG:boto:Data: 2024-03-04T17:50:27.565 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.565 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.565 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.565 DEBUG:boto:Params: {} 2024-03-04T17:50:27.565 DEBUG:boto:Token: None 2024-03-04T17:50:27.565 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.565 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FqixrXzlGlkFUB9YpTyG750T/8M= 2024-03-04T17:50:27.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FqixrXzlGlkFUB9YpTyG750T/8M=', 'Content-Length': '0'} 2024-03-04T17:50:27.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096b353d853c06c34-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.569 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.569 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.569 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:27.569 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:27.569 DEBUG:boto:Method: GET 2024-03-04T17:50:27.569 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:27.569 DEBUG:boto:Data: 2024-03-04T17:50:27.569 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.569 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.570 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.570 DEBUG:boto:Params: {} 2024-03-04T17:50:27.570 DEBUG:boto:Token: None 2024-03-04T17:50:27.570 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.570 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3da4d3db2a3d398-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.620 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.620 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.621 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.621 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:27.621 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:27.621 DEBUG:boto:Method: GET 2024-03-04T17:50:27.621 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:27.621 DEBUG:boto:Data: 2024-03-04T17:50:27.621 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.621 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.621 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.621 DEBUG:boto:Params: {} 2024-03-04T17:50:27.621 DEBUG:boto:Token: None 2024-03-04T17:50:27.622 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.622 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff43fd8494662f6b-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.627 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.627 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.628 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.628 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.628 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.628 DEBUG:boto:Data: 2024-03-04T17:50:27.628 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.628 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.628 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.628 DEBUG:boto:Params: {} 2024-03-04T17:50:27.628 DEBUG:boto:Token: None 2024-03-04T17:50:27.629 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.629 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3wRsQPs6bQlbtt5xdsBm9AOl6zA= 2024-03-04T17:50:27.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3wRsQPs6bQlbtt5xdsBm9AOl6zA=', 'Content-Length': '0'} 2024-03-04T17:50:27.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a9c05e7681f77f3-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.633 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.633 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.633 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.633 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.633 DEBUG:boto:Data: 2024-03-04T17:50:27.634 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.634 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.634 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.634 DEBUG:boto:Params: {} 2024-03-04T17:50:27.634 DEBUG:boto:Token: None 2024-03-04T17:50:27.634 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.634 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3wRsQPs6bQlbtt5xdsBm9AOl6zA= 2024-03-04T17:50:27.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3wRsQPs6bQlbtt5xdsBm9AOl6zA=', 'Content-Length': '0'} 2024-03-04T17:50:27.638 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d634b97ee645a62-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:27.638 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.638 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.638 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.638 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.638 DEBUG:boto:Method: GET 2024-03-04T17:50:27.639 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.639 DEBUG:boto:Data: 2024-03-04T17:50:27.639 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.639 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.639 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.639 DEBUG:boto:Params: {} 2024-03-04T17:50:27.639 DEBUG:boto:Token: None 2024-03-04T17:50:27.639 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.639 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FqixrXzlGlkFUB9YpTyG750T/8M= 2024-03-04T17:50:27.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FqixrXzlGlkFUB9YpTyG750T/8M=', 'Content-Length': '0'} 2024-03-04T17:50:27.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000522a6668f2aab03a-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.684 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.684 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:27.684 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.685 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.685 DEBUG:boto:Method: GET 2024-03-04T17:50:27.685 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.685 DEBUG:boto:Data: 2024-03-04T17:50:27.685 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.685 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.685 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.685 DEBUG:boto:Params: {} 2024-03-04T17:50:27.686 DEBUG:boto:Token: None 2024-03-04T17:50:27.686 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:27.686 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FqixrXzlGlkFUB9YpTyG750T/8M= 2024-03-04T17:50:27.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FqixrXzlGlkFUB9YpTyG750T/8M=', 'Content-Length': '0'} 2024-03-04T17:50:27.689 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e739b6a6df29bef0-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.689 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.689 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.690 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:27.690 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:27.690 DEBUG:boto:Method: GET 2024-03-04T17:50:27.690 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:27.690 DEBUG:boto:Data: 2024-03-04T17:50:27.690 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.690 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.691 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.691 DEBUG:boto:Params: {} 2024-03-04T17:50:27.691 DEBUG:boto:Token: None 2024-03-04T17:50:27.691 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.691 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d02dfbaa3516232a-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.736 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.736 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.737 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.737 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:27.737 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:27.737 DEBUG:boto:Method: GET 2024-03-04T17:50:27.737 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:27.737 DEBUG:boto:Data: 2024-03-04T17:50:27.737 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.737 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.737 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.737 DEBUG:boto:Params: {} 2024-03-04T17:50:27.738 DEBUG:boto:Token: None 2024-03-04T17:50:27.738 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.738 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038b2ff63fc6ebc5c-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.743 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:27.743 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.743 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.743 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.744 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.744 DEBUG:boto:Method: GET 2024-03-04T17:50:27.744 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.744 DEBUG:boto:Data: 2024-03-04T17:50:27.744 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.744 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.744 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.744 DEBUG:boto:Params: {} 2024-03-04T17:50:27.745 DEBUG:boto:Token: None 2024-03-04T17:50:27.745 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.745 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1xQ0Y29/LGlxkSf0t0BK+uTGErE= 2024-03-04T17:50:27.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1xQ0Y29/LGlxkSf0t0BK+uTGErE=', 'Content-Length': '0'} 2024-03-04T17:50:27.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9a17f8e2cf84ad3-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.748 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.748 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.749 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.749 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.749 DEBUG:boto:Method: GET 2024-03-04T17:50:27.749 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.749 DEBUG:boto:Data: 2024-03-04T17:50:27.749 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.749 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.749 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.749 DEBUG:boto:Params: {} 2024-03-04T17:50:27.749 DEBUG:boto:Token: None 2024-03-04T17:50:27.750 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.750 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1xQ0Y29/LGlxkSf0t0BK+uTGErE= 2024-03-04T17:50:27.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1xQ0Y29/LGlxkSf0t0BK+uTGErE=', 'Content-Length': '0'} 2024-03-04T17:50:27.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c81510509d7ab64-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.753 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.753 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.753 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.753 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.753 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.753 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.753 DEBUG:boto:Data: 2024-03-04T17:50:27.753 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.754 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.754 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.754 DEBUG:boto:Params: {} 2024-03-04T17:50:27.754 DEBUG:boto:Token: None 2024-03-04T17:50:27.754 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.754 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K8miGM1FO5pOEePkYT1zPSFEos8= 2024-03-04T17:50:27.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K8miGM1FO5pOEePkYT1zPSFEos8=', 'Content-Length': '0'} 2024-03-04T17:50:27.757 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042219e93d1de9b90-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.757 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.757 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.758 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.758 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.758 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.758 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.758 DEBUG:boto:Data: 2024-03-04T17:50:27.758 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.758 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.758 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.758 DEBUG:boto:Params: {} 2024-03-04T17:50:27.758 DEBUG:boto:Token: None 2024-03-04T17:50:27.758 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.759 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K8miGM1FO5pOEePkYT1zPSFEos8= 2024-03-04T17:50:27.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K8miGM1FO5pOEePkYT1zPSFEos8=', 'Content-Length': '0'} 2024-03-04T17:50:27.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8c5a2ed8b847276-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:27.762 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.762 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.762 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.762 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.762 DEBUG:boto:Method: GET 2024-03-04T17:50:27.763 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.763 DEBUG:boto:Data: 2024-03-04T17:50:27.763 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.763 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.763 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.763 DEBUG:boto:Params: {} 2024-03-04T17:50:27.763 DEBUG:boto:Token: None 2024-03-04T17:50:27.763 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.763 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1xQ0Y29/LGlxkSf0t0BK+uTGErE= 2024-03-04T17:50:27.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1xQ0Y29/LGlxkSf0t0BK+uTGErE=', 'Content-Length': '0'} 2024-03-04T17:50:27.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcb0a7c670eee835-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.808 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.808 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.808 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.808 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.809 DEBUG:boto:Method: GET 2024-03-04T17:50:27.809 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.809 DEBUG:boto:Data: 2024-03-04T17:50:27.809 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.809 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.809 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.809 DEBUG:boto:Params: {} 2024-03-04T17:50:27.809 DEBUG:boto:Token: None 2024-03-04T17:50:27.809 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.810 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1xQ0Y29/LGlxkSf0t0BK+uTGErE= 2024-03-04T17:50:27.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1xQ0Y29/LGlxkSf0t0BK+uTGErE=', 'Content-Length': '0'} 2024-03-04T17:50:27.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c3f038749066ed3-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.856 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.856 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.856 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:27.856 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:27.856 DEBUG:boto:Method: GET 2024-03-04T17:50:27.857 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:27.857 DEBUG:boto:Data: 2024-03-04T17:50:27.857 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.857 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.857 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.857 DEBUG:boto:Params: {} 2024-03-04T17:50:27.857 DEBUG:boto:Token: None 2024-03-04T17:50:27.857 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.857 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e88689668f36ebc6-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.862 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.863 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.863 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.863 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:27.863 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:27.863 DEBUG:boto:Method: GET 2024-03-04T17:50:27.863 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:27.863 DEBUG:boto:Data: 2024-03-04T17:50:27.863 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.864 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.864 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.864 DEBUG:boto:Params: {} 2024-03-04T17:50:27.864 DEBUG:boto:Token: None 2024-03-04T17:50:27.864 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.864 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071db6beeb1a9fb78-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.869 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:27.869 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.870 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.870 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.870 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.870 DEBUG:boto:Data: 2024-03-04T17:50:27.870 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.870 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.870 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.870 DEBUG:boto:Params: {} 2024-03-04T17:50:27.870 DEBUG:boto:Token: None 2024-03-04T17:50:27.871 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.871 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZMrcdHz3j4JLT7jKq6Q4FiEw1gw= 2024-03-04T17:50:27.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZMrcdHz3j4JLT7jKq6Q4FiEw1gw=', 'Content-Length': '0'} 2024-03-04T17:50:27.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e246d21525fd334e-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.875 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.875 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.875 DEBUG:boto:Method: HEAD 2024-03-04T17:50:27.875 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.875 DEBUG:boto:Data: 2024-03-04T17:50:27.875 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.875 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.875 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.876 DEBUG:boto:Params: {} 2024-03-04T17:50:27.876 DEBUG:boto:Token: None 2024-03-04T17:50:27.876 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.876 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZMrcdHz3j4JLT7jKq6Q4FiEw1gw= 2024-03-04T17:50:27.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZMrcdHz3j4JLT7jKq6Q4FiEw1gw=', 'Content-Length': '0'} 2024-03-04T17:50:27.880 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e50109543a166b50-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:27.880 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.881 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.881 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.881 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.881 DEBUG:boto:Method: GET 2024-03-04T17:50:27.881 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.881 DEBUG:boto:Data: 2024-03-04T17:50:27.881 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.881 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.881 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.882 DEBUG:boto:Params: {} 2024-03-04T17:50:27.882 DEBUG:boto:Token: None 2024-03-04T17:50:27.882 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.882 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1xQ0Y29/LGlxkSf0t0BK+uTGErE= 2024-03-04T17:50:27.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1xQ0Y29/LGlxkSf0t0BK+uTGErE=', 'Content-Length': '0'} 2024-03-04T17:50:27.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fe6497ed02e008d-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.928 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.928 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:27.928 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.928 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.929 DEBUG:boto:Method: GET 2024-03-04T17:50:27.929 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.929 DEBUG:boto:Data: 2024-03-04T17:50:27.929 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.929 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:27.929 DEBUG:boto:Port: 8000 2024-03-04T17:50:27.929 DEBUG:boto:Params: {} 2024-03-04T17:50:27.929 DEBUG:boto:Token: None 2024-03-04T17:50:27.930 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:27.930 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1xQ0Y29/LGlxkSf0t0BK+uTGErE= 2024-03-04T17:50:27.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1xQ0Y29/LGlxkSf0t0BK+uTGErE=', 'Content-Length': '0'} 2024-03-04T17:50:27.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d265110c890ea87-0065e609e3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:27.976 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:27.976 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:27.976 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:27.976 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:27.977 DEBUG:boto:Method: GET 2024-03-04T17:50:27.977 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:27.977 DEBUG:boto:Data: 2024-03-04T17:50:27.977 DEBUG:boto:Headers: {} 2024-03-04T17:50:27.977 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:27.977 DEBUG:boto:Port: 8001 2024-03-04T17:50:27.977 DEBUG:boto:Params: {} 2024-03-04T17:50:27.977 DEBUG:boto:Token: None 2024-03-04T17:50:27.977 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:27 GMT /ppfahn-19/ 2024-03-04T17:50:27.978 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM= 2024-03-04T17:50:27.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3We8NzZYtb2nJuHqhNxK0qhKGPM=', 'Content-Length': '0'} 2024-03-04T17:50:27.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000436ed767b645862e-0065e609e3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.024 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.025 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.025 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.025 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:28.025 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:28.025 DEBUG:boto:Method: GET 2024-03-04T17:50:28.025 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:28.025 DEBUG:boto:Data: 2024-03-04T17:50:28.025 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.026 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:28.026 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.026 DEBUG:boto:Params: {} 2024-03-04T17:50:28.026 DEBUG:boto:Token: None 2024-03-04T17:50:28.026 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.026 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8= 2024-03-04T17:50:28.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8=', 'Content-Length': '0'} 2024-03-04T17:50:28.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000188e2b155924cf8d-0065e609e4-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.072 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:28.073 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.073 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.073 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.073 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.073 DEBUG:boto:Method: GET 2024-03-04T17:50:28.073 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.073 DEBUG:boto:Data: 2024-03-04T17:50:28.073 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.074 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.074 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.074 DEBUG:boto:Params: {} 2024-03-04T17:50:28.074 DEBUG:boto:Token: None 2024-03-04T17:50:28.074 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.074 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LQmOGh8dl5WWWLX4LyyvB8y58hM= 2024-03-04T17:50:28.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LQmOGh8dl5WWWLX4LyyvB8y58hM=', 'Content-Length': '0'} 2024-03-04T17:50:28.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000245db685f77dfd55-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.078 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.078 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.078 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.078 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.078 DEBUG:boto:Method: GET 2024-03-04T17:50:28.079 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.079 DEBUG:boto:Data: 2024-03-04T17:50:28.079 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.079 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:28.079 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.079 DEBUG:boto:Params: {} 2024-03-04T17:50:28.079 DEBUG:boto:Token: None 2024-03-04T17:50:28.079 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.079 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LQmOGh8dl5WWWLX4LyyvB8y58hM= 2024-03-04T17:50:28.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LQmOGh8dl5WWWLX4LyyvB8y58hM=', 'Content-Length': '0'} 2024-03-04T17:50:28.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c9b9f112762b106-0065e609e4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.124 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.124 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.124 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.124 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.124 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.124 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.125 DEBUG:boto:Data: 2024-03-04T17:50:28.125 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.125 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.125 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.125 DEBUG:boto:Params: {} 2024-03-04T17:50:28.125 DEBUG:boto:Token: None 2024-03-04T17:50:28.125 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.125 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:18WrMYjxGhSatVI2QLlSyacD9d0= 2024-03-04T17:50:28.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:18WrMYjxGhSatVI2QLlSyacD9d0=', 'Content-Length': '0'} 2024-03-04T17:50:28.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000308116b013f0eb0-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.129 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.129 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.129 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.129 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.130 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.130 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.130 DEBUG:boto:Data: 2024-03-04T17:50:28.130 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.130 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:28.130 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.130 DEBUG:boto:Params: {} 2024-03-04T17:50:28.130 DEBUG:boto:Token: None 2024-03-04T17:50:28.130 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.130 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:18WrMYjxGhSatVI2QLlSyacD9d0= 2024-03-04T17:50:28.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:18WrMYjxGhSatVI2QLlSyacD9d0=', 'Content-Length': '0'} 2024-03-04T17:50:28.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7527a7332844fb6-0065e609e4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:28.134 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.134 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.134 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.134 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.134 DEBUG:boto:Method: GET 2024-03-04T17:50:28.134 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.134 DEBUG:boto:Data: 2024-03-04T17:50:28.134 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.134 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.134 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.135 DEBUG:boto:Params: {} 2024-03-04T17:50:28.135 DEBUG:boto:Token: None 2024-03-04T17:50:28.135 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.135 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LQmOGh8dl5WWWLX4LyyvB8y58hM= 2024-03-04T17:50:28.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LQmOGh8dl5WWWLX4LyyvB8y58hM=', 'Content-Length': '0'} 2024-03-04T17:50:28.138 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011c58c6539f2d9b8-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.138 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.139 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.139 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.139 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.139 DEBUG:boto:Method: GET 2024-03-04T17:50:28.139 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.139 DEBUG:boto:Data: 2024-03-04T17:50:28.139 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.139 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:28.139 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.139 DEBUG:boto:Params: {} 2024-03-04T17:50:28.139 DEBUG:boto:Token: None 2024-03-04T17:50:28.140 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.140 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LQmOGh8dl5WWWLX4LyyvB8y58hM= 2024-03-04T17:50:28.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LQmOGh8dl5WWWLX4LyyvB8y58hM=', 'Content-Length': '0'} 2024-03-04T17:50:28.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8933884107da2da-0065e609e4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.188 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.188 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.188 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:28.188 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:28.188 DEBUG:boto:Method: GET 2024-03-04T17:50:28.189 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:28.189 DEBUG:boto:Data: 2024-03-04T17:50:28.189 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.189 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.189 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.189 DEBUG:boto:Params: {} 2024-03-04T17:50:28.189 DEBUG:boto:Token: None 2024-03-04T17:50:28.189 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.189 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8= 2024-03-04T17:50:28.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8=', 'Content-Length': '0'} 2024-03-04T17:50:28.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009722b165f0d0e00d-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.197 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.198 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.198 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.198 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:28.198 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:28.198 DEBUG:boto:Method: GET 2024-03-04T17:50:28.198 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:28.198 DEBUG:boto:Data: 2024-03-04T17:50:28.199 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.199 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:28.199 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.199 DEBUG:boto:Params: {} 2024-03-04T17:50:28.199 DEBUG:boto:Token: None 2024-03-04T17:50:28.199 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.199 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8= 2024-03-04T17:50:28.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8=', 'Content-Length': '0'} 2024-03-04T17:50:28.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aefdf7578afe067a-0065e609e4-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.204 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.205 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.205 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.205 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.205 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.205 DEBUG:boto:Data: 2024-03-04T17:50:28.205 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.205 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.205 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.205 DEBUG:boto:Params: {} 2024-03-04T17:50:28.206 DEBUG:boto:Token: None 2024-03-04T17:50:28.206 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.206 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glUVF7P4/Wm/f1Zs/frpE6hanXc= 2024-03-04T17:50:28.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glUVF7P4/Wm/f1Zs/frpE6hanXc=', 'Content-Length': '0'} 2024-03-04T17:50:28.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f241b542a6c33759-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.210 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.211 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.211 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.211 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.211 DEBUG:boto:Data: 2024-03-04T17:50:28.211 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.211 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:28.211 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.211 DEBUG:boto:Params: {} 2024-03-04T17:50:28.211 DEBUG:boto:Token: None 2024-03-04T17:50:28.211 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.212 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:glUVF7P4/Wm/f1Zs/frpE6hanXc= 2024-03-04T17:50:28.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:glUVF7P4/Wm/f1Zs/frpE6hanXc=', 'Content-Length': '0'} 2024-03-04T17:50:28.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ddb38027707ab5d0-0065e609e4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:28.215 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.215 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.216 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.216 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.216 DEBUG:boto:Method: GET 2024-03-04T17:50:28.216 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.216 DEBUG:boto:Data: 2024-03-04T17:50:28.216 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.216 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.216 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.216 DEBUG:boto:Params: {} 2024-03-04T17:50:28.216 DEBUG:boto:Token: None 2024-03-04T17:50:28.217 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.217 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LQmOGh8dl5WWWLX4LyyvB8y58hM= 2024-03-04T17:50:28.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LQmOGh8dl5WWWLX4LyyvB8y58hM=', 'Content-Length': '0'} 2024-03-04T17:50:28.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054d61b9d9606f1a4-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.260 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.263 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:28.263 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.263 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.263 DEBUG:boto:Method: GET 2024-03-04T17:50:28.263 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.263 DEBUG:boto:Data: 2024-03-04T17:50:28.263 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.263 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:28.263 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.263 DEBUG:boto:Params: {} 2024-03-04T17:50:28.264 DEBUG:boto:Token: None 2024-03-04T17:50:28.264 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:28.264 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LQmOGh8dl5WWWLX4LyyvB8y58hM= 2024-03-04T17:50:28.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LQmOGh8dl5WWWLX4LyyvB8y58hM=', 'Content-Length': '0'} 2024-03-04T17:50:28.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbd1431b1fd7ea6a-0065e609e4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.324 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.324 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.324 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:28.324 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:28.324 DEBUG:boto:Method: GET 2024-03-04T17:50:28.325 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:28.325 DEBUG:boto:Data: 2024-03-04T17:50:28.325 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.325 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.325 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.325 DEBUG:boto:Params: {} 2024-03-04T17:50:28.325 DEBUG:boto:Token: None 2024-03-04T17:50:28.325 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.325 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8= 2024-03-04T17:50:28.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8=', 'Content-Length': '0'} 2024-03-04T17:50:28.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042b8e63867427f38-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.330 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.331 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.331 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.331 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:28.331 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:28.331 DEBUG:boto:Method: GET 2024-03-04T17:50:28.331 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:28.331 DEBUG:boto:Data: 2024-03-04T17:50:28.331 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.331 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:28.331 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.331 DEBUG:boto:Params: {} 2024-03-04T17:50:28.332 DEBUG:boto:Token: None 2024-03-04T17:50:28.332 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.332 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8= 2024-03-04T17:50:28.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8=', 'Content-Length': '0'} 2024-03-04T17:50:28.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005de8c4014196d3db-0065e609e4-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.384 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.384 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-19 zones={test-zone3, test-zone1} 2024-03-04T17:50:28.385 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-19 zones={test-zone3, test-zone2} 2024-03-04T17:50:28.385 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.385 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.385 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.385 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:50:28.385 DEBUG:boto:Data: 2024-03-04T17:50:28.385 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.385 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.385 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.386 DEBUG:boto:Params: {} 2024-03-04T17:50:28.386 DEBUG:boto:Token: None 2024-03-04T17:50:28.386 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.386 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cMBMJpvhDrhXqhxknVB9irnBG1g= 2024-03-04T17:50:28.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cMBMJpvhDrhXqhxknVB9irnBG1g=', 'Content-Length': '0'} 2024-03-04T17:50:28.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000884ca58c9ee11776-0065e609e4-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.390 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.391 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.391 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.391 DEBUG:boto:Path: /ppfahn-19/ 2024-03-04T17:50:28.391 DEBUG:boto:Data: 2024-03-04T17:50:28.391 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.391 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.391 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.391 DEBUG:boto:Params: {} 2024-03-04T17:50:28.391 DEBUG:boto:Token: None 2024-03-04T17:50:28.391 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.392 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cMBMJpvhDrhXqhxknVB9irnBG1g= 2024-03-04T17:50:28.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cMBMJpvhDrhXqhxknVB9irnBG1g=', 'Content-Length': '0'} 2024-03-04T17:50:28.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009798f47a9753c1f3-0065e609e4-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.396 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:50:28.396 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.396 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.396 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:28.396 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:28.396 DEBUG:boto:Method: GET 2024-03-04T17:50:28.396 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:28.396 DEBUG:boto:Data: 2024-03-04T17:50:28.396 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.397 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.397 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.397 DEBUG:boto:Params: {} 2024-03-04T17:50:28.397 DEBUG:boto:Token: None 2024-03-04T17:50:28.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/?versions 2024-03-04T17:50:28.397 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vr+HBUO9IYEyZRnvSrk558bT/I0= 2024-03-04T17:50:28.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vr+HBUO9IYEyZRnvSrk558bT/I0=', 'Content-Length': '0'} 2024-03-04T17:50:28.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000047cd14f2b303416-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.403 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.403 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:28.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:28.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:28.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:28.404 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:50:28.404 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.404 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.404 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:28.404 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:28.404 DEBUG:boto:Method: GET 2024-03-04T17:50:28.404 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:28.404 DEBUG:boto:Data: 2024-03-04T17:50:28.404 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.404 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.404 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.405 DEBUG:boto:Params: {} 2024-03-04T17:50:28.405 DEBUG:boto:Token: None 2024-03-04T17:50:28.405 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/?versions 2024-03-04T17:50:28.405 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vr+HBUO9IYEyZRnvSrk558bT/I0= 2024-03-04T17:50:28.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vr+HBUO9IYEyZRnvSrk558bT/I0=', 'Content-Length': '0'} 2024-03-04T17:50:28.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021ef8932e2732633-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.411 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.411 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:28.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:28.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:28.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:28.412 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.412 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.412 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:28.412 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:28.412 DEBUG:boto:Method: GET 2024-03-04T17:50:28.412 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:28.412 DEBUG:boto:Data: 2024-03-04T17:50:28.413 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.413 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.413 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.413 DEBUG:boto:Params: {} 2024-03-04T17:50:28.413 DEBUG:boto:Token: None 2024-03-04T17:50:28.413 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/?versions 2024-03-04T17:50:28.413 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vr+HBUO9IYEyZRnvSrk558bT/I0= 2024-03-04T17:50:28.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vr+HBUO9IYEyZRnvSrk558bT/I0=', 'Content-Length': '0'} 2024-03-04T17:50:28.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7e495790092d961-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.423 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.423 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.423 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.423 DEBUG:boto:path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:28.423 DEBUG:boto:auth_path=/ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:28.423 DEBUG:boto:Method: GET 2024-03-04T17:50:28.424 DEBUG:boto:Path: /ppfahn-19/?versions&max-keys=999 2024-03-04T17:50:28.424 DEBUG:boto:Data: 2024-03-04T17:50:28.424 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.424 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.424 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.424 DEBUG:boto:Params: {} 2024-03-04T17:50:28.424 DEBUG:boto:Token: None 2024-03-04T17:50:28.424 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/?versions 2024-03-04T17:50:28.424 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vr+HBUO9IYEyZRnvSrk558bT/I0= 2024-03-04T17:50:28.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vr+HBUO9IYEyZRnvSrk558bT/I0=', 'Content-Length': '0'} 2024-03-04T17:50:28.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa0cf2f7e816f64f-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.430 DEBUG:boto:b'ppfahn-19999falsedummysAVAcTpRY1OySF6H.UWROayAd-qRmSStrue2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1eCc5l1SXhS057sNbAGLPH2xxwInfUVctrue2024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.QqVqYCqCLuwRiumOHrFuCduXVOdtiNtrue2024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFvtrue2024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:28.430 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:28.431 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:28.431 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.431 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.431 DEBUG:boto:Method: GET 2024-03-04T17:50:28.431 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.431 DEBUG:boto:Data: 2024-03-04T17:50:28.431 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.431 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.431 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.431 DEBUG:boto:Params: {} 2024-03-04T17:50:28.431 DEBUG:boto:Token: None 2024-03-04T17:50:28.432 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.432 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JXHy6Np9r810++PEfLUnQIkvmG4= 2024-03-04T17:50:28.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JXHy6Np9r810++PEfLUnQIkvmG4=', 'Content-Length': '0'} 2024-03-04T17:50:28.435 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f46ebc5252647435-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.435 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:28.435 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:28.435 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.435 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.435 DEBUG:boto:Method: GET 2024-03-04T17:50:28.436 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.436 DEBUG:boto:Data: 2024-03-04T17:50:28.436 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.436 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.436 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.436 DEBUG:boto:Params: {} 2024-03-04T17:50:28.436 DEBUG:boto:Token: None 2024-03-04T17:50:28.436 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.436 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JXHy6Np9r810++PEfLUnQIkvmG4= 2024-03-04T17:50:28.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JXHy6Np9r810++PEfLUnQIkvmG4=', 'Content-Length': '0'} 2024-03-04T17:50:28.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c7e956f3b0f51a6e-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.439 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:28.439 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:28.440 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.440 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.440 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.440 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.440 DEBUG:boto:Data: 2024-03-04T17:50:28.440 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.440 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.440 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.440 DEBUG:boto:Params: {} 2024-03-04T17:50:28.440 DEBUG:boto:Token: None 2024-03-04T17:50:28.440 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.441 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nVpS8JyR0TOoq4pDa2OQj1fmHwA= 2024-03-04T17:50:28.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nVpS8JyR0TOoq4pDa2OQj1fmHwA=', 'Content-Length': '0'} 2024-03-04T17:50:28.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e3a9a93620a54744-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.444 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:28.444 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:28.444 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.444 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.444 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.444 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.444 DEBUG:boto:Data: 2024-03-04T17:50:28.444 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.444 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.444 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.444 DEBUG:boto:Params: {} 2024-03-04T17:50:28.445 DEBUG:boto:Token: None 2024-03-04T17:50:28.445 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.445 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nVpS8JyR0TOoq4pDa2OQj1fmHwA= 2024-03-04T17:50:28.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nVpS8JyR0TOoq4pDa2OQj1fmHwA=', 'Content-Length': '0'} 2024-03-04T17:50:28.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001fc8bbc8681e43b5-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:28.448 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:28.448 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:28.448 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.448 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.449 DEBUG:boto:Method: GET 2024-03-04T17:50:28.449 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.449 DEBUG:boto:Data: 2024-03-04T17:50:28.449 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.449 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.449 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.449 DEBUG:boto:Params: {} 2024-03-04T17:50:28.449 DEBUG:boto:Token: None 2024-03-04T17:50:28.449 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.450 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JXHy6Np9r810++PEfLUnQIkvmG4= 2024-03-04T17:50:28.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JXHy6Np9r810++PEfLUnQIkvmG4=', 'Content-Length': '0'} 2024-03-04T17:50:28.460 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e99fc38d49436f87-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.460 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:28.460 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:28.460 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.460 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.460 DEBUG:boto:Method: GET 2024-03-04T17:50:28.460 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.460 DEBUG:boto:Data: 2024-03-04T17:50:28.461 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.461 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.461 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.461 DEBUG:boto:Params: {} 2024-03-04T17:50:28.461 DEBUG:boto:Token: None 2024-03-04T17:50:28.461 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.461 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JXHy6Np9r810++PEfLUnQIkvmG4= 2024-03-04T17:50:28.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JXHy6Np9r810++PEfLUnQIkvmG4=', 'Content-Length': '0'} 2024-03-04T17:50:28.464 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000095409c2c349a4f00-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.464 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.464 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.464 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:28.465 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:28.467 DEBUG:boto:Method: GET 2024-03-04T17:50:28.467 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:50:28.467 DEBUG:boto:Data: 2024-03-04T17:50:28.467 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.467 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.467 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.467 DEBUG:boto:Params: {} 2024-03-04T17:50:28.467 DEBUG:boto:Token: None 2024-03-04T17:50:28.468 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.468 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8= 2024-03-04T17:50:28.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8=', 'Content-Length': '0'} 2024-03-04T17:50:28.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000825e52e5f3b1b81a-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.516 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.516 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.517 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.517 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:28.517 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:28.517 DEBUG:boto:Method: GET 2024-03-04T17:50:28.517 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:50:28.517 DEBUG:boto:Data: 2024-03-04T17:50:28.517 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.517 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.517 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.517 DEBUG:boto:Params: {} 2024-03-04T17:50:28.518 DEBUG:boto:Token: None 2024-03-04T17:50:28.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.518 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8= 2024-03-04T17:50:28.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8=', 'Content-Length': '0'} 2024-03-04T17:50:28.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046e6cf031aa7af30-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.527 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.527 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:28.527 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:28.527 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.527 DEBUG:boto:Path: /ppfahn-19/dummy 2024-03-04T17:50:28.527 DEBUG:boto:Data: 2024-03-04T17:50:28.528 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.528 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.528 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.528 DEBUG:boto:Params: {} 2024-03-04T17:50:28.528 DEBUG:boto:Token: None 2024-03-04T17:50:28.528 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/dummy 2024-03-04T17:50:28.528 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cVfKfBxpLUtxXHYaZqMwhKVan78= 2024-03-04T17:50:28.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cVfKfBxpLUtxXHYaZqMwhKVan78=', 'Content-Length': '0'} 2024-03-04T17:50:28.531 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007287516bd9c98af0-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.531 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:28.532 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:28.532 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.532 DEBUG:boto:Path: /ppfahn-19/dummy 2024-03-04T17:50:28.532 DEBUG:boto:Data: 2024-03-04T17:50:28.532 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.532 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.532 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.532 DEBUG:boto:Params: {} 2024-03-04T17:50:28.532 DEBUG:boto:Token: None 2024-03-04T17:50:28.533 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/dummy 2024-03-04T17:50:28.533 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cVfKfBxpLUtxXHYaZqMwhKVan78= 2024-03-04T17:50:28.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cVfKfBxpLUtxXHYaZqMwhKVan78=', 'Content-Length': '0'} 2024-03-04T17:50:28.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c0e98b0845aa9ead-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:28.537 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:28.537 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:28.537 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.537 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.537 DEBUG:boto:Method: GET 2024-03-04T17:50:28.537 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.537 DEBUG:boto:Data: 2024-03-04T17:50:28.537 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.537 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.537 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.537 DEBUG:boto:Params: {} 2024-03-04T17:50:28.538 DEBUG:boto:Token: None 2024-03-04T17:50:28.538 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.538 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JXHy6Np9r810++PEfLUnQIkvmG4= 2024-03-04T17:50:28.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JXHy6Np9r810++PEfLUnQIkvmG4=', 'Content-Length': '0'} 2024-03-04T17:50:28.540 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d3b41e775dab5eaf-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.540 DEBUG:boto:path=/ppfahn-19/dummy 2024-03-04T17:50:28.540 DEBUG:boto:auth_path=/ppfahn-19/dummy 2024-03-04T17:50:28.540 DEBUG:boto:path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.541 DEBUG:boto:auth_path=/ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.541 DEBUG:boto:Method: GET 2024-03-04T17:50:28.541 DEBUG:boto:Path: /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.541 DEBUG:boto:Data: 2024-03-04T17:50:28.541 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.541 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.541 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.541 DEBUG:boto:Params: {} 2024-03-04T17:50:28.541 DEBUG:boto:Token: None 2024-03-04T17:50:28.541 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/dummy?versionId=sAVAcTpRY1OySF6H.UWROayAd-qRmSS 2024-03-04T17:50:28.542 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JXHy6Np9r810++PEfLUnQIkvmG4= 2024-03-04T17:50:28.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JXHy6Np9r810++PEfLUnQIkvmG4=', 'Content-Length': '0'} 2024-03-04T17:50:28.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:45:54 GMT'), ('x-amz-version-id', 'sAVAcTpRY1OySF6H.UWROayAd-qRmSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006e40288e334e1c60-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.544 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.544 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.544 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:28.545 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:28.545 DEBUG:boto:Method: GET 2024-03-04T17:50:28.545 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:50:28.545 DEBUG:boto:Data: 2024-03-04T17:50:28.545 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.545 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.545 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.545 DEBUG:boto:Params: {} 2024-03-04T17:50:28.545 DEBUG:boto:Token: None 2024-03-04T17:50:28.545 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.546 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8= 2024-03-04T17:50:28.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8=', 'Content-Length': '0'} 2024-03-04T17:50:28.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb2da39a15ec7969-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.549 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.549 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.550 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.550 DEBUG:boto:path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:28.550 DEBUG:boto:auth_path=/ppfahn-19/?prefix=dummy 2024-03-04T17:50:28.550 DEBUG:boto:Method: GET 2024-03-04T17:50:28.550 DEBUG:boto:Path: /ppfahn-19/?prefix=dummy 2024-03-04T17:50:28.550 DEBUG:boto:Data: 2024-03-04T17:50:28.550 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.550 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.550 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.550 DEBUG:boto:Params: {} 2024-03-04T17:50:28.550 DEBUG:boto:Token: None 2024-03-04T17:50:28.551 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.551 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8= 2024-03-04T17:50:28.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8=', 'Content-Length': '0'} 2024-03-04T17:50:28.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce6dd75260f2d994-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.600 DEBUG:boto:b'ppfahn-19dummy1000falsedummy2024-03-04T17:45:54.839Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:28.601 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.601 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.601 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.601 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.601 DEBUG:boto:Method: GET 2024-03-04T17:50:28.601 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.601 DEBUG:boto:Data: 2024-03-04T17:50:28.601 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.602 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.602 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.602 DEBUG:boto:Params: {} 2024-03-04T17:50:28.602 DEBUG:boto:Token: None 2024-03-04T17:50:28.602 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.602 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FQjtNXyxSm4fwRg05T52s1ifXTE= 2024-03-04T17:50:28.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FQjtNXyxSm4fwRg05T52s1ifXTE=', 'Content-Length': '0'} 2024-03-04T17:50:28.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000718272d6907e494-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.648 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.648 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.648 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.648 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.649 DEBUG:boto:Method: GET 2024-03-04T17:50:28.649 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.649 DEBUG:boto:Data: 2024-03-04T17:50:28.649 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.649 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.649 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.649 DEBUG:boto:Params: {} 2024-03-04T17:50:28.649 DEBUG:boto:Token: None 2024-03-04T17:50:28.649 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.650 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FQjtNXyxSm4fwRg05T52s1ifXTE= 2024-03-04T17:50:28.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FQjtNXyxSm4fwRg05T52s1ifXTE=', 'Content-Length': '0'} 2024-03-04T17:50:28.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017817d3e8e858953-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.700 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.700 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.700 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.701 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.701 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.701 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.701 DEBUG:boto:Data: 2024-03-04T17:50:28.701 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.701 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.701 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.701 DEBUG:boto:Params: {} 2024-03-04T17:50:28.701 DEBUG:boto:Token: None 2024-03-04T17:50:28.701 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.702 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hY25x46JIwUvEDheDWYmDQV2Dtw= 2024-03-04T17:50:28.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hY25x46JIwUvEDheDWYmDQV2Dtw=', 'Content-Length': '0'} 2024-03-04T17:50:28.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050f6486f6e43585f-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.705 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.705 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.705 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.705 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.705 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.705 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.706 DEBUG:boto:Data: 2024-03-04T17:50:28.706 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.706 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.706 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.706 DEBUG:boto:Params: {} 2024-03-04T17:50:28.706 DEBUG:boto:Token: None 2024-03-04T17:50:28.706 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.706 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hY25x46JIwUvEDheDWYmDQV2Dtw= 2024-03-04T17:50:28.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hY25x46JIwUvEDheDWYmDQV2Dtw=', 'Content-Length': '0'} 2024-03-04T17:50:28.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f68b38fe828628e-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:28.709 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.710 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.710 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.710 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.710 DEBUG:boto:Method: GET 2024-03-04T17:50:28.710 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.710 DEBUG:boto:Data: 2024-03-04T17:50:28.710 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.710 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.710 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.710 DEBUG:boto:Params: {} 2024-03-04T17:50:28.710 DEBUG:boto:Token: None 2024-03-04T17:50:28.711 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.711 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FQjtNXyxSm4fwRg05T52s1ifXTE= 2024-03-04T17:50:28.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FQjtNXyxSm4fwRg05T52s1ifXTE=', 'Content-Length': '0'} 2024-03-04T17:50:28.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012f43ecc76bd21b4-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.756 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.756 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.756 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.756 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.757 DEBUG:boto:Method: GET 2024-03-04T17:50:28.757 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.757 DEBUG:boto:Data: 2024-03-04T17:50:28.757 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.757 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.757 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.757 DEBUG:boto:Params: {} 2024-03-04T17:50:28.757 DEBUG:boto:Token: None 2024-03-04T17:50:28.757 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.758 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FQjtNXyxSm4fwRg05T52s1ifXTE= 2024-03-04T17:50:28.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FQjtNXyxSm4fwRg05T52s1ifXTE=', 'Content-Length': '0'} 2024-03-04T17:50:28.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b9ebe56f74b5131-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.808 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.808 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.808 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:28.808 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:28.808 DEBUG:boto:Method: GET 2024-03-04T17:50:28.809 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:28.809 DEBUG:boto:Data: 2024-03-04T17:50:28.809 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.809 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.809 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.809 DEBUG:boto:Params: {} 2024-03-04T17:50:28.809 DEBUG:boto:Token: None 2024-03-04T17:50:28.809 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.809 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8= 2024-03-04T17:50:28.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8=', 'Content-Length': '0'} 2024-03-04T17:50:28.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a36303dae5abcfe-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.815 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.816 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.816 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.816 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:28.816 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:28.816 DEBUG:boto:Method: GET 2024-03-04T17:50:28.816 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:28.817 DEBUG:boto:Data: 2024-03-04T17:50:28.817 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.817 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.817 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.817 DEBUG:boto:Params: {} 2024-03-04T17:50:28.817 DEBUG:boto:Token: None 2024-03-04T17:50:28.817 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.817 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8= 2024-03-04T17:50:28.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8=', 'Content-Length': '0'} 2024-03-04T17:50:28.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f4357cc7f677584-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.864 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.864 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.865 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.865 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.865 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.865 DEBUG:boto:Data: 2024-03-04T17:50:28.865 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.865 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.865 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.865 DEBUG:boto:Params: {} 2024-03-04T17:50:28.865 DEBUG:boto:Token: None 2024-03-04T17:50:28.866 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.866 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pS4+i2o/Sq8+uY9FUrWd5aNCqsA= 2024-03-04T17:50:28.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pS4+i2o/Sq8+uY9FUrWd5aNCqsA=', 'Content-Length': '0'} 2024-03-04T17:50:28.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039bf78bf2c23a463-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.870 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.870 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.870 DEBUG:boto:Method: HEAD 2024-03-04T17:50:28.870 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.871 DEBUG:boto:Data: 2024-03-04T17:50:28.871 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.871 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.871 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.871 DEBUG:boto:Params: {} 2024-03-04T17:50:28.871 DEBUG:boto:Token: None 2024-03-04T17:50:28.871 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.871 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pS4+i2o/Sq8+uY9FUrWd5aNCqsA= 2024-03-04T17:50:28.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pS4+i2o/Sq8+uY9FUrWd5aNCqsA=', 'Content-Length': '0'} 2024-03-04T17:50:28.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000266391bd8c5cc3f3-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:28.875 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.876 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.876 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.876 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.879 DEBUG:boto:Method: GET 2024-03-04T17:50:28.879 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.879 DEBUG:boto:Data: 2024-03-04T17:50:28.879 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.879 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.879 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.879 DEBUG:boto:Params: {} 2024-03-04T17:50:28.879 DEBUG:boto:Token: None 2024-03-04T17:50:28.880 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.880 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FQjtNXyxSm4fwRg05T52s1ifXTE= 2024-03-04T17:50:28.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FQjtNXyxSm4fwRg05T52s1ifXTE=', 'Content-Length': '0'} 2024-03-04T17:50:28.883 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d53a8246939f9d7-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.924 DEBUG:boto:path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.924 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1 2024-03-04T17:50:28.924 DEBUG:boto:path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.924 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.925 DEBUG:boto:Method: GET 2024-03-04T17:50:28.925 DEBUG:boto:Path: /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.925 DEBUG:boto:Data: 2024-03-04T17:50:28.925 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.925 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.925 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.925 DEBUG:boto:Params: {} 2024-03-04T17:50:28.925 DEBUG:boto:Token: None 2024-03-04T17:50:28.925 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone1?versionId=eCc5l1SXhS057sNbAGLPH2xxwInfUVc 2024-03-04T17:50:28.926 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FQjtNXyxSm4fwRg05T52s1ifXTE= 2024-03-04T17:50:28.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FQjtNXyxSm4fwRg05T52s1ifXTE=', 'Content-Length': '0'} 2024-03-04T17:50:28.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', 'eCc5l1SXhS057sNbAGLPH2xxwInfUVc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4efedefd4200248-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.972 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.972 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.972 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:28.972 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:28.973 DEBUG:boto:Method: GET 2024-03-04T17:50:28.973 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:28.973 DEBUG:boto:Data: 2024-03-04T17:50:28.973 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.973 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.973 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.973 DEBUG:boto:Params: {} 2024-03-04T17:50:28.973 DEBUG:boto:Token: None 2024-03-04T17:50:28.973 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.974 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8= 2024-03-04T17:50:28.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8=', 'Content-Length': '0'} 2024-03-04T17:50:28.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2588279385d5c48-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.979 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.980 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:28.980 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:28.980 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:28.980 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:28.980 DEBUG:boto:Method: GET 2024-03-04T17:50:28.980 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone1 2024-03-04T17:50:28.980 DEBUG:boto:Data: 2024-03-04T17:50:28.980 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.980 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:28.981 DEBUG:boto:Port: 8000 2024-03-04T17:50:28.981 DEBUG:boto:Params: {} 2024-03-04T17:50:28.981 DEBUG:boto:Token: None 2024-03-04T17:50:28.981 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/ 2024-03-04T17:50:28.981 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8= 2024-03-04T17:50:28.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8egU+aJ+/WmBWcE+2vJH3B+IcH8=', 'Content-Length': '0'} 2024-03-04T17:50:28.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071251dabb2eb125f-0065e609e4-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:28.986 DEBUG:boto:b'ppfahn-19obj-test-zone11000falseobj-test-zone12024-03-04T17:46:45.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:28.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:28.987 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:28.987 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:28.987 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:28.987 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:28.987 DEBUG:boto:Method: GET 2024-03-04T17:50:28.987 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:28.987 DEBUG:boto:Data: 2024-03-04T17:50:28.987 DEBUG:boto:Headers: {} 2024-03-04T17:50:28.988 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:28.988 DEBUG:boto:Port: 8001 2024-03-04T17:50:28.988 DEBUG:boto:Params: {} 2024-03-04T17:50:28.988 DEBUG:boto:Token: None 2024-03-04T17:50:28.988 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:28 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:28.988 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v5efyptFw/k+KAsKoh5Juw0lYtk= 2024-03-04T17:50:28.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v5efyptFw/k+KAsKoh5Juw0lYtk=', 'Content-Length': '0'} 2024-03-04T17:50:28.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016655690d4979189-0065e609e4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.032 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.032 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.032 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.032 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.032 DEBUG:boto:Method: GET 2024-03-04T17:50:29.032 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.033 DEBUG:boto:Data: 2024-03-04T17:50:29.033 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.033 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.033 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.033 DEBUG:boto:Params: {} 2024-03-04T17:50:29.033 DEBUG:boto:Token: None 2024-03-04T17:50:29.033 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.033 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1uqtSaly5mnriypFAyUGU5o0cw= 2024-03-04T17:50:29.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1uqtSaly5mnriypFAyUGU5o0cw=', 'Content-Length': '0'} 2024-03-04T17:50:29.036 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2f9ac3eff52cbfb-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.084 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.084 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.084 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.085 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.085 DEBUG:boto:Method: HEAD 2024-03-04T17:50:29.085 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.085 DEBUG:boto:Data: 2024-03-04T17:50:29.085 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.085 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:29.085 DEBUG:boto:Port: 8001 2024-03-04T17:50:29.085 DEBUG:boto:Params: {} 2024-03-04T17:50:29.085 DEBUG:boto:Token: None 2024-03-04T17:50:29.085 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.086 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lTMznu4eN46HI9i8WHikCZifNDA= 2024-03-04T17:50:29.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lTMznu4eN46HI9i8WHikCZifNDA=', 'Content-Length': '0'} 2024-03-04T17:50:29.089 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d4fe844bcf68dca-0065e609e5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.089 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.089 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.089 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.089 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.089 DEBUG:boto:Method: HEAD 2024-03-04T17:50:29.090 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.090 DEBUG:boto:Data: 2024-03-04T17:50:29.090 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.090 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.090 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.090 DEBUG:boto:Params: {} 2024-03-04T17:50:29.090 DEBUG:boto:Token: None 2024-03-04T17:50:29.090 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.090 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lTMznu4eN46HI9i8WHikCZifNDA= 2024-03-04T17:50:29.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lTMznu4eN46HI9i8WHikCZifNDA=', 'Content-Length': '0'} 2024-03-04T17:50:29.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce648d131d44e4af-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:29.094 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.094 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.094 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.094 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.095 DEBUG:boto:Method: GET 2024-03-04T17:50:29.095 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.095 DEBUG:boto:Data: 2024-03-04T17:50:29.095 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.095 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:29.095 DEBUG:boto:Port: 8001 2024-03-04T17:50:29.095 DEBUG:boto:Params: {} 2024-03-04T17:50:29.095 DEBUG:boto:Token: None 2024-03-04T17:50:29.095 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.095 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1uqtSaly5mnriypFAyUGU5o0cw= 2024-03-04T17:50:29.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1uqtSaly5mnriypFAyUGU5o0cw=', 'Content-Length': '0'} 2024-03-04T17:50:29.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069276f4793597545-0065e609e5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.140 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.140 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.141 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.141 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.141 DEBUG:boto:Method: GET 2024-03-04T17:50:29.141 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.141 DEBUG:boto:Data: 2024-03-04T17:50:29.141 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.141 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.141 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.141 DEBUG:boto:Params: {} 2024-03-04T17:50:29.142 DEBUG:boto:Token: None 2024-03-04T17:50:29.142 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.142 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1uqtSaly5mnriypFAyUGU5o0cw= 2024-03-04T17:50:29.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1uqtSaly5mnriypFAyUGU5o0cw=', 'Content-Length': '0'} 2024-03-04T17:50:29.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e75cacc2cd35372c-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.145 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:29.145 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:29.145 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:29.145 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:29.146 DEBUG:boto:Method: GET 2024-03-04T17:50:29.146 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:29.146 DEBUG:boto:Data: 2024-03-04T17:50:29.146 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.146 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:29.146 DEBUG:boto:Port: 8001 2024-03-04T17:50:29.146 DEBUG:boto:Params: {} 2024-03-04T17:50:29.146 DEBUG:boto:Token: None 2024-03-04T17:50:29.146 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/ 2024-03-04T17:50:29.146 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA= 2024-03-04T17:50:29.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA=', 'Content-Length': '0'} 2024-03-04T17:50:29.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000d04c55d34613b2-0065e609e5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.153 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:29.153 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:29.154 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:29.154 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:29.154 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:29.154 DEBUG:boto:Method: GET 2024-03-04T17:50:29.155 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:29.155 DEBUG:boto:Data: 2024-03-04T17:50:29.155 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.155 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.155 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.155 DEBUG:boto:Params: {} 2024-03-04T17:50:29.156 DEBUG:boto:Token: None 2024-03-04T17:50:29.156 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/ 2024-03-04T17:50:29.156 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA= 2024-03-04T17:50:29.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA=', 'Content-Length': '0'} 2024-03-04T17:50:29.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d07d566e6778edc9-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.161 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:29.161 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.162 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.162 DEBUG:boto:Method: HEAD 2024-03-04T17:50:29.162 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.162 DEBUG:boto:Data: 2024-03-04T17:50:29.162 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.162 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:29.162 DEBUG:boto:Port: 8001 2024-03-04T17:50:29.162 DEBUG:boto:Params: {} 2024-03-04T17:50:29.162 DEBUG:boto:Token: None 2024-03-04T17:50:29.162 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.163 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wMKYVhEY8c4VJJu2HD3eE0ODGdQ= 2024-03-04T17:50:29.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wMKYVhEY8c4VJJu2HD3eE0ODGdQ=', 'Content-Length': '0'} 2024-03-04T17:50:29.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d18f2723546be234-0065e609e5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.166 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.167 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.167 DEBUG:boto:Method: HEAD 2024-03-04T17:50:29.167 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.167 DEBUG:boto:Data: 2024-03-04T17:50:29.167 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.167 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.167 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.167 DEBUG:boto:Params: {} 2024-03-04T17:50:29.167 DEBUG:boto:Token: None 2024-03-04T17:50:29.167 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.167 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wMKYVhEY8c4VJJu2HD3eE0ODGdQ= 2024-03-04T17:50:29.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wMKYVhEY8c4VJJu2HD3eE0ODGdQ=', 'Content-Length': '0'} 2024-03-04T17:50:29.171 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a026ca6ca4ceba53-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:29.172 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.172 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.172 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.172 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.172 DEBUG:boto:Method: GET 2024-03-04T17:50:29.172 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.172 DEBUG:boto:Data: 2024-03-04T17:50:29.172 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.172 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:29.173 DEBUG:boto:Port: 8001 2024-03-04T17:50:29.173 DEBUG:boto:Params: {} 2024-03-04T17:50:29.173 DEBUG:boto:Token: None 2024-03-04T17:50:29.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.173 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1uqtSaly5mnriypFAyUGU5o0cw= 2024-03-04T17:50:29.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1uqtSaly5mnriypFAyUGU5o0cw=', 'Content-Length': '0'} 2024-03-04T17:50:29.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002dc87bcad34f692-0065e609e5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.216 DEBUG:boto:path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.216 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2 2024-03-04T17:50:29.216 DEBUG:boto:path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.216 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.216 DEBUG:boto:Method: GET 2024-03-04T17:50:29.217 DEBUG:boto:Path: /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.217 DEBUG:boto:Data: 2024-03-04T17:50:29.217 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.217 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.217 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.217 DEBUG:boto:Params: {} 2024-03-04T17:50:29.217 DEBUG:boto:Token: None 2024-03-04T17:50:29.217 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone2?versionId=.QqVqYCqCLuwRiumOHrFuCduXVOdtiN 2024-03-04T17:50:29.217 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G1uqtSaly5mnriypFAyUGU5o0cw= 2024-03-04T17:50:29.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G1uqtSaly5mnriypFAyUGU5o0cw=', 'Content-Length': '0'} 2024-03-04T17:50:29.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:45 GMT'), ('x-amz-version-id', '.QqVqYCqCLuwRiumOHrFuCduXVOdtiN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000adeb0d698dd19c94-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.221 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:29.221 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:29.222 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:29.222 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:29.222 DEBUG:boto:Method: GET 2024-03-04T17:50:29.222 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:29.222 DEBUG:boto:Data: 2024-03-04T17:50:29.222 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.222 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:29.222 DEBUG:boto:Port: 8001 2024-03-04T17:50:29.222 DEBUG:boto:Params: {} 2024-03-04T17:50:29.222 DEBUG:boto:Token: None 2024-03-04T17:50:29.223 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/ 2024-03-04T17:50:29.223 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA= 2024-03-04T17:50:29.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA=', 'Content-Length': '0'} 2024-03-04T17:50:29.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a99267526f33d079-0065e609e5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.227 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:29.228 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:29.228 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:29.228 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:29.228 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:29.228 DEBUG:boto:Method: GET 2024-03-04T17:50:29.228 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone2 2024-03-04T17:50:29.229 DEBUG:boto:Data: 2024-03-04T17:50:29.229 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.229 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.229 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.229 DEBUG:boto:Params: {} 2024-03-04T17:50:29.229 DEBUG:boto:Token: None 2024-03-04T17:50:29.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/ 2024-03-04T17:50:29.229 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA= 2024-03-04T17:50:29.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA=', 'Content-Length': '0'} 2024-03-04T17:50:29.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ef6bbffd5881210-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.234 DEBUG:boto:b'ppfahn-19obj-test-zone21000falseobj-test-zone22024-03-04T17:46:45.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:29.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:29.234 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.234 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.234 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.234 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.234 DEBUG:boto:Method: GET 2024-03-04T17:50:29.234 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.235 DEBUG:boto:Data: 2024-03-04T17:50:29.235 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.235 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:29.235 DEBUG:boto:Port: 8001 2024-03-04T17:50:29.235 DEBUG:boto:Params: {} 2024-03-04T17:50:29.235 DEBUG:boto:Token: None 2024-03-04T17:50:29.235 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.235 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GkEicoydvIMRX9uGuq9eDbN0Nao= 2024-03-04T17:50:29.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GkEicoydvIMRX9uGuq9eDbN0Nao=', 'Content-Length': '0'} 2024-03-04T17:50:29.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062c04ad271e0c43c-0065e609e5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.238 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.239 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.239 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.239 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.239 DEBUG:boto:Method: GET 2024-03-04T17:50:29.239 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.239 DEBUG:boto:Data: 2024-03-04T17:50:29.239 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.239 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.239 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.239 DEBUG:boto:Params: {} 2024-03-04T17:50:29.239 DEBUG:boto:Token: None 2024-03-04T17:50:29.240 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.240 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GkEicoydvIMRX9uGuq9eDbN0Nao= 2024-03-04T17:50:29.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GkEicoydvIMRX9uGuq9eDbN0Nao=', 'Content-Length': '0'} 2024-03-04T17:50:29.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe43778105ab653b-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.288 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.288 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.289 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.289 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.289 DEBUG:boto:Method: HEAD 2024-03-04T17:50:29.289 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.289 DEBUG:boto:Data: 2024-03-04T17:50:29.289 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.289 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:29.289 DEBUG:boto:Port: 8001 2024-03-04T17:50:29.289 DEBUG:boto:Params: {} 2024-03-04T17:50:29.289 DEBUG:boto:Token: None 2024-03-04T17:50:29.290 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.290 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fLtDmsukA6bwzNDJJu2cjf9Rnv4= 2024-03-04T17:50:29.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fLtDmsukA6bwzNDJJu2cjf9Rnv4=', 'Content-Length': '0'} 2024-03-04T17:50:29.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099a0373be4961477-0065e609e5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.293 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.293 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.293 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.293 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.294 DEBUG:boto:Method: HEAD 2024-03-04T17:50:29.294 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.294 DEBUG:boto:Data: 2024-03-04T17:50:29.294 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.294 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.294 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.294 DEBUG:boto:Params: {} 2024-03-04T17:50:29.294 DEBUG:boto:Token: None 2024-03-04T17:50:29.294 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.294 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fLtDmsukA6bwzNDJJu2cjf9Rnv4= 2024-03-04T17:50:29.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fLtDmsukA6bwzNDJJu2cjf9Rnv4=', 'Content-Length': '0'} 2024-03-04T17:50:29.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011534e24a3119778-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:29.298 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.298 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.298 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.298 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.298 DEBUG:boto:Method: GET 2024-03-04T17:50:29.298 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.298 DEBUG:boto:Data: 2024-03-04T17:50:29.299 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.299 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:29.299 DEBUG:boto:Port: 8001 2024-03-04T17:50:29.299 DEBUG:boto:Params: {} 2024-03-04T17:50:29.299 DEBUG:boto:Token: None 2024-03-04T17:50:29.299 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.299 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GkEicoydvIMRX9uGuq9eDbN0Nao= 2024-03-04T17:50:29.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GkEicoydvIMRX9uGuq9eDbN0Nao=', 'Content-Length': '0'} 2024-03-04T17:50:29.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e643d799c6f39d32-0065e609e5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.344 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.344 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.344 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.344 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.344 DEBUG:boto:Method: GET 2024-03-04T17:50:29.345 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.345 DEBUG:boto:Data: 2024-03-04T17:50:29.345 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.345 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.345 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.345 DEBUG:boto:Params: {} 2024-03-04T17:50:29.345 DEBUG:boto:Token: None 2024-03-04T17:50:29.345 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.345 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GkEicoydvIMRX9uGuq9eDbN0Nao= 2024-03-04T17:50:29.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GkEicoydvIMRX9uGuq9eDbN0Nao=', 'Content-Length': '0'} 2024-03-04T17:50:29.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef2d2bf0c08e8c7c-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.392 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:29.392 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:29.392 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:29.392 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:29.392 DEBUG:boto:Method: GET 2024-03-04T17:50:29.393 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:29.393 DEBUG:boto:Data: 2024-03-04T17:50:29.393 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.393 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:29.393 DEBUG:boto:Port: 8001 2024-03-04T17:50:29.393 DEBUG:boto:Params: {} 2024-03-04T17:50:29.393 DEBUG:boto:Token: None 2024-03-04T17:50:29.393 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/ 2024-03-04T17:50:29.393 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA= 2024-03-04T17:50:29.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA=', 'Content-Length': '0'} 2024-03-04T17:50:29.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce24edd57c646427-0065e609e5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.399 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:29.399 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:29.399 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:29.400 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:29.400 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:29.400 DEBUG:boto:Method: GET 2024-03-04T17:50:29.400 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:29.400 DEBUG:boto:Data: 2024-03-04T17:50:29.400 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.400 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.400 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.400 DEBUG:boto:Params: {} 2024-03-04T17:50:29.400 DEBUG:boto:Token: None 2024-03-04T17:50:29.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/ 2024-03-04T17:50:29.401 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA= 2024-03-04T17:50:29.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA=', 'Content-Length': '0'} 2024-03-04T17:50:29.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000472aff8f9338f30-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.407 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:29.408 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.408 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.408 DEBUG:boto:Method: HEAD 2024-03-04T17:50:29.408 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.408 DEBUG:boto:Data: 2024-03-04T17:50:29.408 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.408 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:29.409 DEBUG:boto:Port: 8001 2024-03-04T17:50:29.409 DEBUG:boto:Params: {} 2024-03-04T17:50:29.409 DEBUG:boto:Token: None 2024-03-04T17:50:29.409 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.409 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zcRB6w2jvHGpppDNbyfhASCohpw= 2024-03-04T17:50:29.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zcRB6w2jvHGpppDNbyfhASCohpw=', 'Content-Length': '0'} 2024-03-04T17:50:29.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007baad64c2b0d620-0065e609e5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.413 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.413 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.413 DEBUG:boto:Method: HEAD 2024-03-04T17:50:29.413 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.413 DEBUG:boto:Data: 2024-03-04T17:50:29.413 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.414 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.414 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.414 DEBUG:boto:Params: {} 2024-03-04T17:50:29.414 DEBUG:boto:Token: None 2024-03-04T17:50:29.414 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.414 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zcRB6w2jvHGpppDNbyfhASCohpw= 2024-03-04T17:50:29.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zcRB6w2jvHGpppDNbyfhASCohpw=', 'Content-Length': '0'} 2024-03-04T17:50:29.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005011517081af74be-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:29.418 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.418 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.418 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.418 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.418 DEBUG:boto:Method: GET 2024-03-04T17:50:29.419 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.419 DEBUG:boto:Data: 2024-03-04T17:50:29.419 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.419 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:29.419 DEBUG:boto:Port: 8001 2024-03-04T17:50:29.419 DEBUG:boto:Params: {} 2024-03-04T17:50:29.419 DEBUG:boto:Token: None 2024-03-04T17:50:29.419 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.419 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GkEicoydvIMRX9uGuq9eDbN0Nao= 2024-03-04T17:50:29.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GkEicoydvIMRX9uGuq9eDbN0Nao=', 'Content-Length': '0'} 2024-03-04T17:50:29.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6fcfcff7a9b3cf4-0065e609e5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.422 DEBUG:boto:path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.423 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3 2024-03-04T17:50:29.423 DEBUG:boto:path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.423 DEBUG:boto:auth_path=/ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.423 DEBUG:boto:Method: GET 2024-03-04T17:50:29.423 DEBUG:boto:Path: /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.423 DEBUG:boto:Data: 2024-03-04T17:50:29.423 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.423 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.423 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.423 DEBUG:boto:Params: {} 2024-03-04T17:50:29.423 DEBUG:boto:Token: None 2024-03-04T17:50:29.424 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/obj-test-zone3?versionId=wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv 2024-03-04T17:50:29.424 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GkEicoydvIMRX9uGuq9eDbN0Nao= 2024-03-04T17:50:29.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GkEicoydvIMRX9uGuq9eDbN0Nao=', 'Content-Length': '0'} 2024-03-04T17:50:29.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'wZh0X8ZP7Ady3IwE3RhyE8b3It.BBFv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-19:d96636ac-054d-40e3-90f3-e3912da784b2.4349.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000556bfe46f45acad-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.468 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:29.468 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:29.468 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:29.469 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:29.469 DEBUG:boto:Method: GET 2024-03-04T17:50:29.469 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:29.469 DEBUG:boto:Data: 2024-03-04T17:50:29.470 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.470 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:29.470 DEBUG:boto:Port: 8001 2024-03-04T17:50:29.470 DEBUG:boto:Params: {} 2024-03-04T17:50:29.471 DEBUG:boto:Token: None 2024-03-04T17:50:29.471 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/ 2024-03-04T17:50:29.471 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA= 2024-03-04T17:50:29.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA=', 'Content-Length': '0'} 2024-03-04T17:50:29.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f49a9e0553924b20-0065e609e5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.476 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:29.477 DEBUG:boto:path=/ppfahn-19/ 2024-03-04T17:50:29.477 DEBUG:boto:auth_path=/ppfahn-19/ 2024-03-04T17:50:29.478 DEBUG:boto:path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:29.478 DEBUG:boto:auth_path=/ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:29.478 DEBUG:boto:Method: GET 2024-03-04T17:50:29.478 DEBUG:boto:Path: /ppfahn-19/?prefix=obj-test-zone3 2024-03-04T17:50:29.478 DEBUG:boto:Data: 2024-03-04T17:50:29.478 DEBUG:boto:Headers: {} 2024-03-04T17:50:29.478 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:29.478 DEBUG:boto:Port: 8000 2024-03-04T17:50:29.478 DEBUG:boto:Params: {} 2024-03-04T17:50:29.479 DEBUG:boto:Token: None 2024-03-04T17:50:29.479 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:29 GMT /ppfahn-19/ 2024-03-04T17:50:29.479 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA= 2024-03-04T17:50:29.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RnLp7cnYzODrjah600xsMeBYXIA=', 'Content-Length': '0'} 2024-03-04T17:50:29.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000094b026411e82816-0065e609e5-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:29.484 DEBUG:boto:b'ppfahn-19obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:29.485 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-19 zones={test-zone3, test-zone2} 2024-03-04T17:50:29.485 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-20 2024-03-04T17:50:29.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:29.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:29.486 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:29.536 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:29.536 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:29.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:29.599+0000 7f5bec6ec980 1 bucket sync caught up with source: 2024-03-04T17:50:29.599 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000011.79.6, , 00000000003.11.10, , 00000000008.76.10, , , , , ] 2024-03-04T17:50:29.599 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000011.79.6, , 00000000003.11.10, , 00000000008.76.10, , , , , ] 2024-03-04T17:50:29.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:29.599+0000 7f5bec6ec980 0 bucket checkpoint complete 2024-03-04T17:50:29.621 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-20 2024-03-04T17:50:29.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:29.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:29.622 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:29.672 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:29.672 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:29.733+0000 7febd58f8980 1 bucket sync caught up with source: 2024-03-04T17:50:29.733 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000011.79.6, , 00000000003.11.10, , 00000000008.76.10, , , , , ] 2024-03-04T17:50:29.733 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000011.79.6, , 00000000003.11.10, , 00000000008.76.10, , , , , ] 2024-03-04T17:50:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:29.733+0000 7febd58f8980 0 bucket checkpoint complete 2024-03-04T17:50:29.755 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-20 2024-03-04T17:50:29.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:29.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:29.756 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:29.797 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:29.797 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:29.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:29.857+0000 7f43489c6980 1 bucket sync caught up with source: 2024-03-04T17:50:29.857 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.58.10, , 00000000011.37.6, , 00000000008.64.10, , , , , ] 2024-03-04T17:50:29.857 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.58.10, , 00000000011.37.6, , 00000000008.64.10, , , , , ] 2024-03-04T17:50:29.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:29.857+0000 7f43489c6980 0 bucket checkpoint complete 2024-03-04T17:50:29.878 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-20 2024-03-04T17:50:29.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:29.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:29.879 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:29.969 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:29.969 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:30.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:30.033+0000 7f3c8d965980 1 bucket sync caught up with source: 2024-03-04T17:50:30.032 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.58.10, , 00000000011.37.6, , 00000000008.64.10, , , , , ] 2024-03-04T17:50:30.032 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.58.10, , 00000000011.37.6, , 00000000008.64.10, , , , , ] 2024-03-04T17:50:30.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:30.033+0000 7f3c8d965980 0 bucket checkpoint complete 2024-03-04T17:50:30.053 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-20 2024-03-04T17:50:30.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:30.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:30.053 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:30.094 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:30.094 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:30.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:30.149+0000 7f06f6291980 1 bucket sync caught up with source: 2024-03-04T17:50:30.150 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.124.10, , 00000000003.62.10, , 00000000014.127.6, , , , , ] 2024-03-04T17:50:30.150 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.124.10, , 00000000003.62.10, , 00000000014.127.6, , , , , ] 2024-03-04T17:50:30.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:30.149+0000 7f06f6291980 0 bucket checkpoint complete 2024-03-04T17:50:30.171 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-20 2024-03-04T17:50:30.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:30.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:30.172 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:30.221 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:30.222 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:30.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:30.295+0000 7f5ba0424980 1 bucket sync caught up with source: 2024-03-04T17:50:30.293 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000003.124.10, , 00000000003.62.10, , 00000000014.127.6, , , , , ] 2024-03-04T17:50:30.293 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000003.124.10, , 00000000003.62.10, , 00000000014.127.6, , , , , ] 2024-03-04T17:50:30.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:30.295+0000 7f5ba0424980 0 bucket checkpoint complete 2024-03-04T17:50:30.317 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-20 zones={test-zone2, test-zone1} 2024-03-04T17:50:30.318 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.318 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.318 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.318 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:50:30.318 DEBUG:boto:Data: 2024-03-04T17:50:30.318 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.318 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.318 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.318 DEBUG:boto:Params: {} 2024-03-04T17:50:30.319 DEBUG:boto:Token: None 2024-03-04T17:50:30.319 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/ 2024-03-04T17:50:30.319 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1LpIquJURGXoux0lsLDmfM4uIig= 2024-03-04T17:50:30.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1LpIquJURGXoux0lsLDmfM4uIig=', 'Content-Length': '0'} 2024-03-04T17:50:30.321 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062d25dcab6cf69a4-0065e609e6-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.322 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.322 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.322 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.322 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:50:30.322 DEBUG:boto:Data: 2024-03-04T17:50:30.322 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.322 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.322 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.322 DEBUG:boto:Params: {} 2024-03-04T17:50:30.323 DEBUG:boto:Token: None 2024-03-04T17:50:30.323 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/ 2024-03-04T17:50:30.323 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1LpIquJURGXoux0lsLDmfM4uIig= 2024-03-04T17:50:30.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1LpIquJURGXoux0lsLDmfM4uIig=', 'Content-Length': '0'} 2024-03-04T17:50:30.327 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d1ea3d338b10591-0065e609e6-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.327 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:50:30.327 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.327 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.327 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:30.327 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:30.327 DEBUG:boto:Method: GET 2024-03-04T17:50:30.327 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:30.327 DEBUG:boto:Data: 2024-03-04T17:50:30.327 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.328 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.328 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.328 DEBUG:boto:Params: {} 2024-03-04T17:50:30.328 DEBUG:boto:Token: None 2024-03-04T17:50:30.328 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/?versions 2024-03-04T17:50:30.328 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uy6+Npb6B2wkGorVqGxqm6wLMdE= 2024-03-04T17:50:30.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uy6+Npb6B2wkGorVqGxqm6wLMdE=', 'Content-Length': '0'} 2024-03-04T17:50:30.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e630f55597c8637-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.331 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.331 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:30.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:30.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:30.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:30.332 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:50:30.332 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.332 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.332 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:30.332 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:30.333 DEBUG:boto:Method: GET 2024-03-04T17:50:30.333 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:30.333 DEBUG:boto:Data: 2024-03-04T17:50:30.333 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.333 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.333 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.333 DEBUG:boto:Params: {} 2024-03-04T17:50:30.333 DEBUG:boto:Token: None 2024-03-04T17:50:30.334 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/?versions 2024-03-04T17:50:30.334 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uy6+Npb6B2wkGorVqGxqm6wLMdE= 2024-03-04T17:50:30.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uy6+Npb6B2wkGorVqGxqm6wLMdE=', 'Content-Length': '0'} 2024-03-04T17:50:30.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da54f27214a1db64-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.338 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.339 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:30.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:30.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:30.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:30.339 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.340 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.340 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:30.340 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:30.340 DEBUG:boto:Method: GET 2024-03-04T17:50:30.340 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:30.340 DEBUG:boto:Data: 2024-03-04T17:50:30.341 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.341 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.342 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.342 DEBUG:boto:Params: {} 2024-03-04T17:50:30.342 DEBUG:boto:Token: None 2024-03-04T17:50:30.342 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/?versions 2024-03-04T17:50:30.342 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uy6+Npb6B2wkGorVqGxqm6wLMdE= 2024-03-04T17:50:30.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uy6+Npb6B2wkGorVqGxqm6wLMdE=', 'Content-Length': '0'} 2024-03-04T17:50:30.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dd1cc145a15d870-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.345 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.346 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.346 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.346 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:30.346 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:30.346 DEBUG:boto:Method: GET 2024-03-04T17:50:30.346 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:30.346 DEBUG:boto:Data: 2024-03-04T17:50:30.346 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.346 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.346 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.346 DEBUG:boto:Params: {} 2024-03-04T17:50:30.347 DEBUG:boto:Token: None 2024-03-04T17:50:30.347 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/?versions 2024-03-04T17:50:30.347 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uy6+Npb6B2wkGorVqGxqm6wLMdE= 2024-03-04T17:50:30.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uy6+Npb6B2wkGorVqGxqm6wLMdE=', 'Content-Length': '0'} 2024-03-04T17:50:30.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebfb11fe30cbf19c-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.351 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:30.352 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:30.352 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:30.352 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.352 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.353 DEBUG:boto:Method: GET 2024-03-04T17:50:30.353 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.353 DEBUG:boto:Data: 2024-03-04T17:50:30.353 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.353 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.353 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.353 DEBUG:boto:Params: {} 2024-03-04T17:50:30.353 DEBUG:boto:Token: None 2024-03-04T17:50:30.353 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.353 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DaIF3sRaOk+T9n08aoElBvze76w= 2024-03-04T17:50:30.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DaIF3sRaOk+T9n08aoElBvze76w=', 'Content-Length': '0'} 2024-03-04T17:50:30.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e9188a28efdab98-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.356 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:30.356 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:30.356 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.356 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.356 DEBUG:boto:Method: GET 2024-03-04T17:50:30.356 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.356 DEBUG:boto:Data: 2024-03-04T17:50:30.357 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.357 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.357 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.357 DEBUG:boto:Params: {} 2024-03-04T17:50:30.357 DEBUG:boto:Token: None 2024-03-04T17:50:30.357 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.357 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DaIF3sRaOk+T9n08aoElBvze76w= 2024-03-04T17:50:30.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DaIF3sRaOk+T9n08aoElBvze76w=', 'Content-Length': '0'} 2024-03-04T17:50:30.360 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006f9b3c5b32c3be06-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.360 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:30.361 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:30.361 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.361 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.361 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.361 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.361 DEBUG:boto:Data: 2024-03-04T17:50:30.361 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.361 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.361 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.361 DEBUG:boto:Params: {} 2024-03-04T17:50:30.362 DEBUG:boto:Token: None 2024-03-04T17:50:30.362 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.362 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XKTaZGPVbF464M2YXBov7kmjUrg= 2024-03-04T17:50:30.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XKTaZGPVbF464M2YXBov7kmjUrg=', 'Content-Length': '0'} 2024-03-04T17:50:30.364 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aebecc3aa5abb8d7-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.364 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:30.364 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:30.364 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.364 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.365 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.365 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.365 DEBUG:boto:Data: 2024-03-04T17:50:30.365 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.365 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.365 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.365 DEBUG:boto:Params: {} 2024-03-04T17:50:30.365 DEBUG:boto:Token: None 2024-03-04T17:50:30.365 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.365 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XKTaZGPVbF464M2YXBov7kmjUrg= 2024-03-04T17:50:30.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XKTaZGPVbF464M2YXBov7kmjUrg=', 'Content-Length': '0'} 2024-03-04T17:50:30.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a9f533351a17a7d7-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:30.369 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:30.369 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:30.369 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.369 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.369 DEBUG:boto:Method: GET 2024-03-04T17:50:30.369 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.369 DEBUG:boto:Data: 2024-03-04T17:50:30.369 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.369 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.369 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.370 DEBUG:boto:Params: {} 2024-03-04T17:50:30.370 DEBUG:boto:Token: None 2024-03-04T17:50:30.370 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.370 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DaIF3sRaOk+T9n08aoElBvze76w= 2024-03-04T17:50:30.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DaIF3sRaOk+T9n08aoElBvze76w=', 'Content-Length': '0'} 2024-03-04T17:50:30.372 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000010e7c7357df048f4-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.372 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:30.373 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:30.373 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.373 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.373 DEBUG:boto:Method: GET 2024-03-04T17:50:30.373 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.373 DEBUG:boto:Data: 2024-03-04T17:50:30.373 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.373 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.373 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.373 DEBUG:boto:Params: {} 2024-03-04T17:50:30.374 DEBUG:boto:Token: None 2024-03-04T17:50:30.374 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.374 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DaIF3sRaOk+T9n08aoElBvze76w= 2024-03-04T17:50:30.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DaIF3sRaOk+T9n08aoElBvze76w=', 'Content-Length': '0'} 2024-03-04T17:50:30.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084b4204008946629-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.377 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.377 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.377 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:30.377 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:30.377 DEBUG:boto:Method: GET 2024-03-04T17:50:30.377 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:50:30.377 DEBUG:boto:Data: 2024-03-04T17:50:30.377 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.378 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.378 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.378 DEBUG:boto:Params: {} 2024-03-04T17:50:30.378 DEBUG:boto:Token: None 2024-03-04T17:50:30.378 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/ 2024-03-04T17:50:30.378 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4= 2024-03-04T17:50:30.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4=', 'Content-Length': '0'} 2024-03-04T17:50:30.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f14d61645952030-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.424 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.425 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.425 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.425 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:30.425 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:30.425 DEBUG:boto:Method: GET 2024-03-04T17:50:30.425 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:50:30.425 DEBUG:boto:Data: 2024-03-04T17:50:30.425 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.425 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.425 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.426 DEBUG:boto:Params: {} 2024-03-04T17:50:30.426 DEBUG:boto:Token: None 2024-03-04T17:50:30.426 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/ 2024-03-04T17:50:30.426 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4= 2024-03-04T17:50:30.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4=', 'Content-Length': '0'} 2024-03-04T17:50:30.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d5ab26805837be1-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.431 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.431 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:30.431 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:30.431 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.431 DEBUG:boto:Path: /ppfahn-20/dummy 2024-03-04T17:50:30.432 DEBUG:boto:Data: 2024-03-04T17:50:30.432 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.432 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.432 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.432 DEBUG:boto:Params: {} 2024-03-04T17:50:30.432 DEBUG:boto:Token: None 2024-03-04T17:50:30.432 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/dummy 2024-03-04T17:50:30.432 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Ioc4vOjtbArBNOgGQDLoMpyxpPo= 2024-03-04T17:50:30.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Ioc4vOjtbArBNOgGQDLoMpyxpPo=', 'Content-Length': '0'} 2024-03-04T17:50:30.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b64edf090f143f4-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.437 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:30.437 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:30.437 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.437 DEBUG:boto:Path: /ppfahn-20/dummy 2024-03-04T17:50:30.437 DEBUG:boto:Data: 2024-03-04T17:50:30.437 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.437 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.437 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.437 DEBUG:boto:Params: {} 2024-03-04T17:50:30.437 DEBUG:boto:Token: None 2024-03-04T17:50:30.438 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/dummy 2024-03-04T17:50:30.438 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Ioc4vOjtbArBNOgGQDLoMpyxpPo= 2024-03-04T17:50:30.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Ioc4vOjtbArBNOgGQDLoMpyxpPo=', 'Content-Length': '0'} 2024-03-04T17:50:30.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000070251404e3f813b8-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:30.442 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:30.442 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:30.442 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.442 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.442 DEBUG:boto:Method: GET 2024-03-04T17:50:30.443 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.443 DEBUG:boto:Data: 2024-03-04T17:50:30.443 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.443 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.443 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.443 DEBUG:boto:Params: {} 2024-03-04T17:50:30.443 DEBUG:boto:Token: None 2024-03-04T17:50:30.443 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.443 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DaIF3sRaOk+T9n08aoElBvze76w= 2024-03-04T17:50:30.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DaIF3sRaOk+T9n08aoElBvze76w=', 'Content-Length': '0'} 2024-03-04T17:50:30.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007188bef9cb42def7-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.446 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:30.447 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:30.447 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.447 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.447 DEBUG:boto:Method: GET 2024-03-04T17:50:30.447 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.447 DEBUG:boto:Data: 2024-03-04T17:50:30.447 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.447 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.447 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.447 DEBUG:boto:Params: {} 2024-03-04T17:50:30.448 DEBUG:boto:Token: None 2024-03-04T17:50:30.448 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:30.448 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DaIF3sRaOk+T9n08aoElBvze76w= 2024-03-04T17:50:30.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DaIF3sRaOk+T9n08aoElBvze76w=', 'Content-Length': '0'} 2024-03-04T17:50:30.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fc887fa658caeecc-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.451 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.451 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.451 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:30.451 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:30.451 DEBUG:boto:Method: GET 2024-03-04T17:50:30.451 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:50:30.451 DEBUG:boto:Data: 2024-03-04T17:50:30.451 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.452 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.452 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.452 DEBUG:boto:Params: {} 2024-03-04T17:50:30.452 DEBUG:boto:Token: None 2024-03-04T17:50:30.452 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/ 2024-03-04T17:50:30.452 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4= 2024-03-04T17:50:30.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4=', 'Content-Length': '0'} 2024-03-04T17:50:30.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dddaa6e304a4023-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.456 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.457 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.457 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.457 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:30.457 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:30.457 DEBUG:boto:Method: GET 2024-03-04T17:50:30.457 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:50:30.457 DEBUG:boto:Data: 2024-03-04T17:50:30.457 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.457 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.457 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.458 DEBUG:boto:Params: {} 2024-03-04T17:50:30.458 DEBUG:boto:Token: None 2024-03-04T17:50:30.458 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/ 2024-03-04T17:50:30.458 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4= 2024-03-04T17:50:30.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4=', 'Content-Length': '0'} 2024-03-04T17:50:30.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000591f6c57578779ce-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.463 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:30.463 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.463 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.463 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.464 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.464 DEBUG:boto:Method: GET 2024-03-04T17:50:30.464 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.464 DEBUG:boto:Data: 2024-03-04T17:50:30.464 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.464 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.464 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.464 DEBUG:boto:Params: {} 2024-03-04T17:50:30.464 DEBUG:boto:Token: None 2024-03-04T17:50:30.465 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.465 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Gr75ucid1laVd1MkwUtoJOVingk= 2024-03-04T17:50:30.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Gr75ucid1laVd1MkwUtoJOVingk=', 'Content-Length': '0'} 2024-03-04T17:50:30.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e43159c537377e25-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.516 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.516 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.516 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.516 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.517 DEBUG:boto:Method: GET 2024-03-04T17:50:30.517 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.517 DEBUG:boto:Data: 2024-03-04T17:50:30.517 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.517 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.517 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.517 DEBUG:boto:Params: {} 2024-03-04T17:50:30.517 DEBUG:boto:Token: None 2024-03-04T17:50:30.517 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.518 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Gr75ucid1laVd1MkwUtoJOVingk= 2024-03-04T17:50:30.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Gr75ucid1laVd1MkwUtoJOVingk=', 'Content-Length': '0'} 2024-03-04T17:50:30.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bad502a781f147bf-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.521 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.521 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.521 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.522 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.522 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.522 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.522 DEBUG:boto:Data: 2024-03-04T17:50:30.522 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.522 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.522 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.522 DEBUG:boto:Params: {} 2024-03-04T17:50:30.522 DEBUG:boto:Token: None 2024-03-04T17:50:30.522 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.523 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H30hi96t7A3Q7ZRYk/0UwZcF8R8= 2024-03-04T17:50:30.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H30hi96t7A3Q7ZRYk/0UwZcF8R8=', 'Content-Length': '0'} 2024-03-04T17:50:30.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca8c3e4dd06f0b26-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.526 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.526 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.526 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.526 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.526 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.526 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.527 DEBUG:boto:Data: 2024-03-04T17:50:30.527 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.527 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.527 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.527 DEBUG:boto:Params: {} 2024-03-04T17:50:30.527 DEBUG:boto:Token: None 2024-03-04T17:50:30.527 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.527 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H30hi96t7A3Q7ZRYk/0UwZcF8R8= 2024-03-04T17:50:30.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H30hi96t7A3Q7ZRYk/0UwZcF8R8=', 'Content-Length': '0'} 2024-03-04T17:50:30.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e50f5fc69477fff8-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:30.531 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.531 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.531 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.532 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.532 DEBUG:boto:Method: GET 2024-03-04T17:50:30.532 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.532 DEBUG:boto:Data: 2024-03-04T17:50:30.532 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.532 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.532 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.532 DEBUG:boto:Params: {} 2024-03-04T17:50:30.532 DEBUG:boto:Token: None 2024-03-04T17:50:30.532 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.533 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Gr75ucid1laVd1MkwUtoJOVingk= 2024-03-04T17:50:30.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Gr75ucid1laVd1MkwUtoJOVingk=', 'Content-Length': '0'} 2024-03-04T17:50:30.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000cd86906744d818-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.576 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.576 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.576 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.576 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.576 DEBUG:boto:Method: GET 2024-03-04T17:50:30.577 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.577 DEBUG:boto:Data: 2024-03-04T17:50:30.577 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.577 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.577 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.577 DEBUG:boto:Params: {} 2024-03-04T17:50:30.577 DEBUG:boto:Token: None 2024-03-04T17:50:30.577 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.577 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Gr75ucid1laVd1MkwUtoJOVingk= 2024-03-04T17:50:30.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Gr75ucid1laVd1MkwUtoJOVingk=', 'Content-Length': '0'} 2024-03-04T17:50:30.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000992d881e93d35cfa-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.581 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.581 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.581 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:30.581 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:30.582 DEBUG:boto:Method: GET 2024-03-04T17:50:30.582 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:30.582 DEBUG:boto:Data: 2024-03-04T17:50:30.582 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.582 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.582 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.582 DEBUG:boto:Params: {} 2024-03-04T17:50:30.582 DEBUG:boto:Token: None 2024-03-04T17:50:30.582 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/ 2024-03-04T17:50:30.583 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4= 2024-03-04T17:50:30.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4=', 'Content-Length': '0'} 2024-03-04T17:50:30.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088b701e52ae112f8-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.628 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.628 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.629 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.629 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:30.629 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:30.629 DEBUG:boto:Method: GET 2024-03-04T17:50:30.629 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:30.629 DEBUG:boto:Data: 2024-03-04T17:50:30.629 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.629 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.629 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.629 DEBUG:boto:Params: {} 2024-03-04T17:50:30.630 DEBUG:boto:Token: None 2024-03-04T17:50:30.630 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/ 2024-03-04T17:50:30.630 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4= 2024-03-04T17:50:30.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4=', 'Content-Length': '0'} 2024-03-04T17:50:30.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000033a28dbb1b866e8-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.636 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.636 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.636 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.636 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.636 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.636 DEBUG:boto:Data: 2024-03-04T17:50:30.637 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.637 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.637 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.637 DEBUG:boto:Params: {} 2024-03-04T17:50:30.637 DEBUG:boto:Token: None 2024-03-04T17:50:30.637 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.637 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vg7uI+dhLQ2y2kt42OZREazsb9I= 2024-03-04T17:50:30.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vg7uI+dhLQ2y2kt42OZREazsb9I=', 'Content-Length': '0'} 2024-03-04T17:50:30.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cadbe3a7291adf2-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.656 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.656 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.656 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.656 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.656 DEBUG:boto:Data: 2024-03-04T17:50:30.657 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.657 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.657 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.657 DEBUG:boto:Params: {} 2024-03-04T17:50:30.657 DEBUG:boto:Token: None 2024-03-04T17:50:30.657 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.657 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vg7uI+dhLQ2y2kt42OZREazsb9I= 2024-03-04T17:50:30.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vg7uI+dhLQ2y2kt42OZREazsb9I=', 'Content-Length': '0'} 2024-03-04T17:50:30.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd978acb79d7a4b2-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:30.661 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.661 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.661 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.661 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.662 DEBUG:boto:Method: GET 2024-03-04T17:50:30.662 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.662 DEBUG:boto:Data: 2024-03-04T17:50:30.662 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.662 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.662 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.662 DEBUG:boto:Params: {} 2024-03-04T17:50:30.662 DEBUG:boto:Token: None 2024-03-04T17:50:30.662 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.662 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Gr75ucid1laVd1MkwUtoJOVingk= 2024-03-04T17:50:30.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Gr75ucid1laVd1MkwUtoJOVingk=', 'Content-Length': '0'} 2024-03-04T17:50:30.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd65df7a80d1ac15-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.708 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.708 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:30.708 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.708 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.709 DEBUG:boto:Method: GET 2024-03-04T17:50:30.709 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.709 DEBUG:boto:Data: 2024-03-04T17:50:30.709 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.709 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.709 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.709 DEBUG:boto:Params: {} 2024-03-04T17:50:30.709 DEBUG:boto:Token: None 2024-03-04T17:50:30.709 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:30.710 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Gr75ucid1laVd1MkwUtoJOVingk= 2024-03-04T17:50:30.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Gr75ucid1laVd1MkwUtoJOVingk=', 'Content-Length': '0'} 2024-03-04T17:50:30.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e87a6781d65cf087-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.713 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.713 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.713 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:30.713 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:30.713 DEBUG:boto:Method: GET 2024-03-04T17:50:30.714 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:30.714 DEBUG:boto:Data: 2024-03-04T17:50:30.714 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.714 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.714 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.714 DEBUG:boto:Params: {} 2024-03-04T17:50:30.714 DEBUG:boto:Token: None 2024-03-04T17:50:30.714 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/ 2024-03-04T17:50:30.714 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4= 2024-03-04T17:50:30.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4=', 'Content-Length': '0'} 2024-03-04T17:50:30.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaf0404dda5d3d13-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.760 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.760 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.761 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.761 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:30.761 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:30.761 DEBUG:boto:Method: GET 2024-03-04T17:50:30.761 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:30.761 DEBUG:boto:Data: 2024-03-04T17:50:30.761 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.761 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.761 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.761 DEBUG:boto:Params: {} 2024-03-04T17:50:30.762 DEBUG:boto:Token: None 2024-03-04T17:50:30.762 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/ 2024-03-04T17:50:30.762 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4= 2024-03-04T17:50:30.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4=', 'Content-Length': '0'} 2024-03-04T17:50:30.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de0d05d859c4ee56-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.767 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:30.767 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.767 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.767 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.767 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.767 DEBUG:boto:Method: GET 2024-03-04T17:50:30.768 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.768 DEBUG:boto:Data: 2024-03-04T17:50:30.768 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.768 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.768 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.768 DEBUG:boto:Params: {} 2024-03-04T17:50:30.768 DEBUG:boto:Token: None 2024-03-04T17:50:30.768 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.768 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HvLJNGpPc70JNPPjSS3sfFOc79I= 2024-03-04T17:50:30.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HvLJNGpPc70JNPPjSS3sfFOc79I=', 'Content-Length': '0'} 2024-03-04T17:50:30.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037bf53eb05403f68-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.772 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.772 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.772 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.772 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.772 DEBUG:boto:Method: GET 2024-03-04T17:50:30.772 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.772 DEBUG:boto:Data: 2024-03-04T17:50:30.773 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.773 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.773 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.773 DEBUG:boto:Params: {} 2024-03-04T17:50:30.773 DEBUG:boto:Token: None 2024-03-04T17:50:30.773 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.773 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HvLJNGpPc70JNPPjSS3sfFOc79I= 2024-03-04T17:50:30.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HvLJNGpPc70JNPPjSS3sfFOc79I=', 'Content-Length': '0'} 2024-03-04T17:50:30.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005425bed9a393d447-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.820 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.820 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.820 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.820 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.820 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.821 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.821 DEBUG:boto:Data: 2024-03-04T17:50:30.821 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.821 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.821 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.821 DEBUG:boto:Params: {} 2024-03-04T17:50:30.821 DEBUG:boto:Token: None 2024-03-04T17:50:30.822 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.822 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4x/03Zoim+lPSdXWBP2Tr/tzu4k= 2024-03-04T17:50:30.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4x/03Zoim+lPSdXWBP2Tr/tzu4k=', 'Content-Length': '0'} 2024-03-04T17:50:30.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8010f1abb0bf8bc-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.825 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.825 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.825 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.825 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.825 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.826 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.826 DEBUG:boto:Data: 2024-03-04T17:50:30.826 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.826 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.826 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.826 DEBUG:boto:Params: {} 2024-03-04T17:50:30.826 DEBUG:boto:Token: None 2024-03-04T17:50:30.826 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.826 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4x/03Zoim+lPSdXWBP2Tr/tzu4k= 2024-03-04T17:50:30.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4x/03Zoim+lPSdXWBP2Tr/tzu4k=', 'Content-Length': '0'} 2024-03-04T17:50:30.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac5260fef77544a8-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:30.830 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.830 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.830 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.830 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.830 DEBUG:boto:Method: GET 2024-03-04T17:50:30.830 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.830 DEBUG:boto:Data: 2024-03-04T17:50:30.830 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.830 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.830 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.831 DEBUG:boto:Params: {} 2024-03-04T17:50:30.831 DEBUG:boto:Token: None 2024-03-04T17:50:30.831 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.831 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HvLJNGpPc70JNPPjSS3sfFOc79I= 2024-03-04T17:50:30.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HvLJNGpPc70JNPPjSS3sfFOc79I=', 'Content-Length': '0'} 2024-03-04T17:50:30.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003bd1d37da9e1e06-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.834 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.834 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.834 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.835 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.835 DEBUG:boto:Method: GET 2024-03-04T17:50:30.835 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.835 DEBUG:boto:Data: 2024-03-04T17:50:30.835 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.835 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.835 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.835 DEBUG:boto:Params: {} 2024-03-04T17:50:30.835 DEBUG:boto:Token: None 2024-03-04T17:50:30.835 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.836 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HvLJNGpPc70JNPPjSS3sfFOc79I= 2024-03-04T17:50:30.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HvLJNGpPc70JNPPjSS3sfFOc79I=', 'Content-Length': '0'} 2024-03-04T17:50:30.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000085632a704d66379-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.880 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.880 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.880 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:30.880 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:30.881 DEBUG:boto:Method: GET 2024-03-04T17:50:30.881 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:30.881 DEBUG:boto:Data: 2024-03-04T17:50:30.881 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.881 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.881 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.881 DEBUG:boto:Params: {} 2024-03-04T17:50:30.882 DEBUG:boto:Token: None 2024-03-04T17:50:30.882 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/ 2024-03-04T17:50:30.882 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4= 2024-03-04T17:50:30.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4=', 'Content-Length': '0'} 2024-03-04T17:50:30.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec9c2ce0d01fe0e0-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.886 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.887 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:30.887 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:30.887 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:30.887 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:30.887 DEBUG:boto:Method: GET 2024-03-04T17:50:30.887 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:30.888 DEBUG:boto:Data: 2024-03-04T17:50:30.888 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.888 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.888 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.888 DEBUG:boto:Params: {} 2024-03-04T17:50:30.888 DEBUG:boto:Token: None 2024-03-04T17:50:30.888 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/ 2024-03-04T17:50:30.888 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4= 2024-03-04T17:50:30.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UoF0P5jzUPeNpNSCWdR8NbWNeH4=', 'Content-Length': '0'} 2024-03-04T17:50:30.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f41e6b7621d4b50-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.893 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:30.893 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.893 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.894 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.894 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.894 DEBUG:boto:Data: 2024-03-04T17:50:30.894 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.894 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.894 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.894 DEBUG:boto:Params: {} 2024-03-04T17:50:30.895 DEBUG:boto:Token: None 2024-03-04T17:50:30.895 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.895 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hE4HJoMrXBOgM6c8zZMIF9/wzg4= 2024-03-04T17:50:30.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hE4HJoMrXBOgM6c8zZMIF9/wzg4=', 'Content-Length': '0'} 2024-03-04T17:50:30.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a0d9fb140a8bca7-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.899 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.899 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.899 DEBUG:boto:Method: HEAD 2024-03-04T17:50:30.899 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.899 DEBUG:boto:Data: 2024-03-04T17:50:30.899 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.899 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.899 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.900 DEBUG:boto:Params: {} 2024-03-04T17:50:30.900 DEBUG:boto:Token: None 2024-03-04T17:50:30.900 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.900 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hE4HJoMrXBOgM6c8zZMIF9/wzg4= 2024-03-04T17:50:30.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hE4HJoMrXBOgM6c8zZMIF9/wzg4=', 'Content-Length': '0'} 2024-03-04T17:50:30.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8714008e4847bb1-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:30.904 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.904 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.904 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.904 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.904 DEBUG:boto:Method: GET 2024-03-04T17:50:30.904 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.905 DEBUG:boto:Data: 2024-03-04T17:50:30.905 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.905 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:30.905 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.905 DEBUG:boto:Params: {} 2024-03-04T17:50:30.905 DEBUG:boto:Token: None 2024-03-04T17:50:30.905 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.905 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HvLJNGpPc70JNPPjSS3sfFOc79I= 2024-03-04T17:50:30.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HvLJNGpPc70JNPPjSS3sfFOc79I=', 'Content-Length': '0'} 2024-03-04T17:50:30.908 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004772e40471e586af-0065e609e6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:30.952 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.952 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:30.952 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.952 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.952 DEBUG:boto:Method: GET 2024-03-04T17:50:30.953 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.953 DEBUG:boto:Data: 2024-03-04T17:50:30.953 DEBUG:boto:Headers: {} 2024-03-04T17:50:30.953 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:30.953 DEBUG:boto:Port: 8000 2024-03-04T17:50:30.953 DEBUG:boto:Params: {} 2024-03-04T17:50:30.953 DEBUG:boto:Token: None 2024-03-04T17:50:30.953 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:30 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:30.953 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HvLJNGpPc70JNPPjSS3sfFOc79I= 2024-03-04T17:50:30.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HvLJNGpPc70JNPPjSS3sfFOc79I=', 'Content-Length': '0'} 2024-03-04T17:50:30.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0cbeda1f5905ff5-0065e609e6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.000 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.000 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.000 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:31.000 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:31.001 DEBUG:boto:Method: GET 2024-03-04T17:50:31.001 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:31.001 DEBUG:boto:Data: 2024-03-04T17:50:31.001 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.001 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:31.001 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.001 DEBUG:boto:Params: {} 2024-03-04T17:50:31.001 DEBUG:boto:Token: None 2024-03-04T17:50:31.001 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.001 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo= 2024-03-04T17:50:31.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo=', 'Content-Length': '0'} 2024-03-04T17:50:31.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000070f65906b6c9629-0065e609e7-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.006 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.007 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.007 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.007 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:31.007 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:31.007 DEBUG:boto:Method: GET 2024-03-04T17:50:31.007 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:31.007 DEBUG:boto:Data: 2024-03-04T17:50:31.008 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.008 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.008 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.008 DEBUG:boto:Params: {} 2024-03-04T17:50:31.008 DEBUG:boto:Token: None 2024-03-04T17:50:31.008 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.009 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo= 2024-03-04T17:50:31.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo=', 'Content-Length': '0'} 2024-03-04T17:50:31.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb7e70b9e63e9146-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.056 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:31.057 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.057 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.057 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.057 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.057 DEBUG:boto:Method: GET 2024-03-04T17:50:31.057 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.057 DEBUG:boto:Data: 2024-03-04T17:50:31.057 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.057 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:31.058 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.058 DEBUG:boto:Params: {} 2024-03-04T17:50:31.058 DEBUG:boto:Token: None 2024-03-04T17:50:31.058 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.058 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0Du1rrwxbNCrBi0hnqbE2VYnEoA= 2024-03-04T17:50:31.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0Du1rrwxbNCrBi0hnqbE2VYnEoA=', 'Content-Length': '0'} 2024-03-04T17:50:31.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002cf5ceb3c7d44bcc-0065e609e7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.104 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.104 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.104 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.104 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.104 DEBUG:boto:Method: GET 2024-03-04T17:50:31.105 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.105 DEBUG:boto:Data: 2024-03-04T17:50:31.105 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.105 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.105 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.105 DEBUG:boto:Params: {} 2024-03-04T17:50:31.105 DEBUG:boto:Token: None 2024-03-04T17:50:31.105 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.105 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0Du1rrwxbNCrBi0hnqbE2VYnEoA= 2024-03-04T17:50:31.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0Du1rrwxbNCrBi0hnqbE2VYnEoA=', 'Content-Length': '0'} 2024-03-04T17:50:31.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b43f7b5d967d479a-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.152 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.152 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.152 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.153 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.153 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.153 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.153 DEBUG:boto:Data: 2024-03-04T17:50:31.153 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.153 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:31.153 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.153 DEBUG:boto:Params: {} 2024-03-04T17:50:31.153 DEBUG:boto:Token: None 2024-03-04T17:50:31.154 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.154 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HXyJHkPsJ6+Dd0laKEY3oeI9/EQ= 2024-03-04T17:50:31.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HXyJHkPsJ6+Dd0laKEY3oeI9/EQ=', 'Content-Length': '0'} 2024-03-04T17:50:31.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8f365aa036532a2-0065e609e7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.157 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.157 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.157 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.158 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.158 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.158 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.158 DEBUG:boto:Data: 2024-03-04T17:50:31.158 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.158 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.158 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.158 DEBUG:boto:Params: {} 2024-03-04T17:50:31.158 DEBUG:boto:Token: None 2024-03-04T17:50:31.158 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.158 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HXyJHkPsJ6+Dd0laKEY3oeI9/EQ= 2024-03-04T17:50:31.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HXyJHkPsJ6+Dd0laKEY3oeI9/EQ=', 'Content-Length': '0'} 2024-03-04T17:50:31.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000a1fdcf93b4cec7-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:31.162 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.162 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.162 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.162 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.162 DEBUG:boto:Method: GET 2024-03-04T17:50:31.162 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.162 DEBUG:boto:Data: 2024-03-04T17:50:31.163 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.163 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:31.163 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.163 DEBUG:boto:Params: {} 2024-03-04T17:50:31.163 DEBUG:boto:Token: None 2024-03-04T17:50:31.163 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.163 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0Du1rrwxbNCrBi0hnqbE2VYnEoA= 2024-03-04T17:50:31.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0Du1rrwxbNCrBi0hnqbE2VYnEoA=', 'Content-Length': '0'} 2024-03-04T17:50:31.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000039a81991afb0692-0065e609e7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.208 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.208 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.208 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.208 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.208 DEBUG:boto:Method: GET 2024-03-04T17:50:31.208 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.209 DEBUG:boto:Data: 2024-03-04T17:50:31.209 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.209 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.209 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.209 DEBUG:boto:Params: {} 2024-03-04T17:50:31.209 DEBUG:boto:Token: None 2024-03-04T17:50:31.209 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.209 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0Du1rrwxbNCrBi0hnqbE2VYnEoA= 2024-03-04T17:50:31.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0Du1rrwxbNCrBi0hnqbE2VYnEoA=', 'Content-Length': '0'} 2024-03-04T17:50:31.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ab173f305a240dd-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.260 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.260 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.260 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:31.260 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:31.260 DEBUG:boto:Method: GET 2024-03-04T17:50:31.261 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:31.261 DEBUG:boto:Data: 2024-03-04T17:50:31.261 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.261 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:31.261 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.261 DEBUG:boto:Params: {} 2024-03-04T17:50:31.261 DEBUG:boto:Token: None 2024-03-04T17:50:31.261 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.261 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo= 2024-03-04T17:50:31.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo=', 'Content-Length': '0'} 2024-03-04T17:50:31.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2157e3919d0ff0b-0065e609e7-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.308 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.308 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.309 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.309 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:31.309 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:31.309 DEBUG:boto:Method: GET 2024-03-04T17:50:31.309 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:31.309 DEBUG:boto:Data: 2024-03-04T17:50:31.309 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.309 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.309 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.309 DEBUG:boto:Params: {} 2024-03-04T17:50:31.310 DEBUG:boto:Token: None 2024-03-04T17:50:31.310 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.310 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo= 2024-03-04T17:50:31.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo=', 'Content-Length': '0'} 2024-03-04T17:50:31.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000980c5453c2b9c3c9-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.356 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.356 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.357 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.357 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.357 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.357 DEBUG:boto:Data: 2024-03-04T17:50:31.357 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.357 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:31.357 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.357 DEBUG:boto:Params: {} 2024-03-04T17:50:31.357 DEBUG:boto:Token: None 2024-03-04T17:50:31.358 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.358 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/ikPdFWYvr9J5hGj4BJztQ+y3s= 2024-03-04T17:50:31.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/ikPdFWYvr9J5hGj4BJztQ+y3s=', 'Content-Length': '0'} 2024-03-04T17:50:31.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006942f3d3b02169d9-0065e609e7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.362 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.362 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.362 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.362 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.362 DEBUG:boto:Data: 2024-03-04T17:50:31.362 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.362 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.363 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.363 DEBUG:boto:Params: {} 2024-03-04T17:50:31.363 DEBUG:boto:Token: None 2024-03-04T17:50:31.363 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.363 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:U/ikPdFWYvr9J5hGj4BJztQ+y3s= 2024-03-04T17:50:31.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:U/ikPdFWYvr9J5hGj4BJztQ+y3s=', 'Content-Length': '0'} 2024-03-04T17:50:31.367 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b600cda3058fb65-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:31.368 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.368 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.368 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.368 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.368 DEBUG:boto:Method: GET 2024-03-04T17:50:31.368 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.368 DEBUG:boto:Data: 2024-03-04T17:50:31.368 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.369 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:31.369 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.369 DEBUG:boto:Params: {} 2024-03-04T17:50:31.369 DEBUG:boto:Token: None 2024-03-04T17:50:31.369 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.369 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0Du1rrwxbNCrBi0hnqbE2VYnEoA= 2024-03-04T17:50:31.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0Du1rrwxbNCrBi0hnqbE2VYnEoA=', 'Content-Length': '0'} 2024-03-04T17:50:31.372 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9e00a7e802523a0-0065e609e7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.416 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.416 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:31.416 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.416 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.416 DEBUG:boto:Method: GET 2024-03-04T17:50:31.417 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.417 DEBUG:boto:Data: 2024-03-04T17:50:31.417 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.417 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.417 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.417 DEBUG:boto:Params: {} 2024-03-04T17:50:31.417 DEBUG:boto:Token: None 2024-03-04T17:50:31.417 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:31.417 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0Du1rrwxbNCrBi0hnqbE2VYnEoA= 2024-03-04T17:50:31.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0Du1rrwxbNCrBi0hnqbE2VYnEoA=', 'Content-Length': '0'} 2024-03-04T17:50:31.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae63e86888d167c6-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.464 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.464 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.464 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:31.464 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:31.465 DEBUG:boto:Method: GET 2024-03-04T17:50:31.465 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:31.465 DEBUG:boto:Data: 2024-03-04T17:50:31.465 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.465 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:31.465 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.465 DEBUG:boto:Params: {} 2024-03-04T17:50:31.465 DEBUG:boto:Token: None 2024-03-04T17:50:31.465 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.466 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo= 2024-03-04T17:50:31.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo=', 'Content-Length': '0'} 2024-03-04T17:50:31.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d092d0eaeb5c0b5c-0065e609e7-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.470 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.471 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.471 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.471 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:31.471 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:31.471 DEBUG:boto:Method: GET 2024-03-04T17:50:31.471 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:31.471 DEBUG:boto:Data: 2024-03-04T17:50:31.471 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.471 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.471 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.472 DEBUG:boto:Params: {} 2024-03-04T17:50:31.472 DEBUG:boto:Token: None 2024-03-04T17:50:31.472 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.472 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo= 2024-03-04T17:50:31.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo=', 'Content-Length': '0'} 2024-03-04T17:50:31.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb2e7ebb864df599-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.520 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.520 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-20 zones={test-zone2, test-zone1} 2024-03-04T17:50:31.521 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-20 zones={test-zone3, test-zone1} 2024-03-04T17:50:31.521 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.521 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.521 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.521 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:50:31.521 DEBUG:boto:Data: 2024-03-04T17:50:31.521 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.521 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.521 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.522 DEBUG:boto:Params: {} 2024-03-04T17:50:31.522 DEBUG:boto:Token: None 2024-03-04T17:50:31.522 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.522 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BFimg4YqasrzHp5wOtVoic9epy4= 2024-03-04T17:50:31.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BFimg4YqasrzHp5wOtVoic9epy4=', 'Content-Length': '0'} 2024-03-04T17:50:31.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000189713916aaf776e-0065e609e7-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.528 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.528 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.528 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.529 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:50:31.529 DEBUG:boto:Data: 2024-03-04T17:50:31.529 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.529 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.529 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.529 DEBUG:boto:Params: {} 2024-03-04T17:50:31.529 DEBUG:boto:Token: None 2024-03-04T17:50:31.529 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.529 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BFimg4YqasrzHp5wOtVoic9epy4= 2024-03-04T17:50:31.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BFimg4YqasrzHp5wOtVoic9epy4=', 'Content-Length': '0'} 2024-03-04T17:50:31.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003dd45a048973572a-0065e609e7-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.533 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:50:31.533 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.533 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.534 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:31.534 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:31.534 DEBUG:boto:Method: GET 2024-03-04T17:50:31.534 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:31.534 DEBUG:boto:Data: 2024-03-04T17:50:31.534 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.534 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.534 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.534 DEBUG:boto:Params: {} 2024-03-04T17:50:31.534 DEBUG:boto:Token: None 2024-03-04T17:50:31.535 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/?versions 2024-03-04T17:50:31.535 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:h0dGqtFVH3/rYanu8pamBQFcTWg= 2024-03-04T17:50:31.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:h0dGqtFVH3/rYanu8pamBQFcTWg=', 'Content-Length': '0'} 2024-03-04T17:50:31.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000210ddf8327634a11-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.580 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.580 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:31.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:31.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:31.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:31.581 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:50:31.581 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.581 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.581 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:31.581 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:31.581 DEBUG:boto:Method: GET 2024-03-04T17:50:31.582 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:31.582 DEBUG:boto:Data: 2024-03-04T17:50:31.582 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.582 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.582 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.582 DEBUG:boto:Params: {} 2024-03-04T17:50:31.582 DEBUG:boto:Token: None 2024-03-04T17:50:31.582 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/?versions 2024-03-04T17:50:31.582 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:h0dGqtFVH3/rYanu8pamBQFcTWg= 2024-03-04T17:50:31.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:h0dGqtFVH3/rYanu8pamBQFcTWg=', 'Content-Length': '0'} 2024-03-04T17:50:31.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cba95a0c880bce81-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.588 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.588 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:31.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:31.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:31.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:31.588 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.589 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.589 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:31.589 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:31.589 DEBUG:boto:Method: GET 2024-03-04T17:50:31.589 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:31.589 DEBUG:boto:Data: 2024-03-04T17:50:31.589 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.589 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.589 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.589 DEBUG:boto:Params: {} 2024-03-04T17:50:31.589 DEBUG:boto:Token: None 2024-03-04T17:50:31.590 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/?versions 2024-03-04T17:50:31.590 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:h0dGqtFVH3/rYanu8pamBQFcTWg= 2024-03-04T17:50:31.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:h0dGqtFVH3/rYanu8pamBQFcTWg=', 'Content-Length': '0'} 2024-03-04T17:50:31.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000015f7d1dc2a08615-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.595 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.595 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.595 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.595 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:31.595 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:31.595 DEBUG:boto:Method: GET 2024-03-04T17:50:31.596 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:31.596 DEBUG:boto:Data: 2024-03-04T17:50:31.596 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.596 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.596 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.596 DEBUG:boto:Params: {} 2024-03-04T17:50:31.596 DEBUG:boto:Token: None 2024-03-04T17:50:31.596 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/?versions 2024-03-04T17:50:31.596 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:h0dGqtFVH3/rYanu8pamBQFcTWg= 2024-03-04T17:50:31.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:h0dGqtFVH3/rYanu8pamBQFcTWg=', 'Content-Length': '0'} 2024-03-04T17:50:31.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080a298c4e32f05c9-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.602 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:31.602 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:31.602 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:31.603 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.603 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.603 DEBUG:boto:Method: GET 2024-03-04T17:50:31.603 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.603 DEBUG:boto:Data: 2024-03-04T17:50:31.603 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.603 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.603 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.603 DEBUG:boto:Params: {} 2024-03-04T17:50:31.603 DEBUG:boto:Token: None 2024-03-04T17:50:31.604 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.604 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6jKk/X1+kGo1YymWI2ZdJftxVkY= 2024-03-04T17:50:31.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jKk/X1+kGo1YymWI2ZdJftxVkY=', 'Content-Length': '0'} 2024-03-04T17:50:31.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000000de46a9fe6c7fd-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.607 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:31.607 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:31.607 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.607 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.607 DEBUG:boto:Method: GET 2024-03-04T17:50:31.607 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.607 DEBUG:boto:Data: 2024-03-04T17:50:31.607 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.608 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.608 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.608 DEBUG:boto:Params: {} 2024-03-04T17:50:31.608 DEBUG:boto:Token: None 2024-03-04T17:50:31.608 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.608 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6jKk/X1+kGo1YymWI2ZdJftxVkY= 2024-03-04T17:50:31.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jKk/X1+kGo1YymWI2ZdJftxVkY=', 'Content-Length': '0'} 2024-03-04T17:50:31.611 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a337dc49c3e1fe5a-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.611 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:31.611 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:31.611 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.611 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.611 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.612 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.612 DEBUG:boto:Data: 2024-03-04T17:50:31.612 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.612 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.612 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.612 DEBUG:boto:Params: {} 2024-03-04T17:50:31.612 DEBUG:boto:Token: None 2024-03-04T17:50:31.612 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.612 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UhgE9Yih6SVW5DYyw/plgGDIQok= 2024-03-04T17:50:31.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UhgE9Yih6SVW5DYyw/plgGDIQok=', 'Content-Length': '0'} 2024-03-04T17:50:31.615 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000677986aa990bdf2a-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.615 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:31.616 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:31.616 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.616 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.616 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.616 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.616 DEBUG:boto:Data: 2024-03-04T17:50:31.616 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.616 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.616 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.616 DEBUG:boto:Params: {} 2024-03-04T17:50:31.617 DEBUG:boto:Token: None 2024-03-04T17:50:31.617 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.617 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UhgE9Yih6SVW5DYyw/plgGDIQok= 2024-03-04T17:50:31.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UhgE9Yih6SVW5DYyw/plgGDIQok=', 'Content-Length': '0'} 2024-03-04T17:50:31.619 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a9962181588d6d6d-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:31.620 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:31.620 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:31.620 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.620 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.620 DEBUG:boto:Method: GET 2024-03-04T17:50:31.620 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.620 DEBUG:boto:Data: 2024-03-04T17:50:31.620 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.620 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.621 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.621 DEBUG:boto:Params: {} 2024-03-04T17:50:31.621 DEBUG:boto:Token: None 2024-03-04T17:50:31.621 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.621 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6jKk/X1+kGo1YymWI2ZdJftxVkY= 2024-03-04T17:50:31.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jKk/X1+kGo1YymWI2ZdJftxVkY=', 'Content-Length': '0'} 2024-03-04T17:50:31.624 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000018cb3bfd2a726adc-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.624 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:31.624 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:31.624 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.624 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.625 DEBUG:boto:Method: GET 2024-03-04T17:50:31.625 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.625 DEBUG:boto:Data: 2024-03-04T17:50:31.625 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.625 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.625 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.625 DEBUG:boto:Params: {} 2024-03-04T17:50:31.625 DEBUG:boto:Token: None 2024-03-04T17:50:31.625 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.626 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6jKk/X1+kGo1YymWI2ZdJftxVkY= 2024-03-04T17:50:31.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jKk/X1+kGo1YymWI2ZdJftxVkY=', 'Content-Length': '0'} 2024-03-04T17:50:31.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000959998a6d785fa44-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.629 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.629 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.629 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:31.629 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:31.629 DEBUG:boto:Method: GET 2024-03-04T17:50:31.629 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:50:31.629 DEBUG:boto:Data: 2024-03-04T17:50:31.630 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.630 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.630 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.630 DEBUG:boto:Params: {} 2024-03-04T17:50:31.630 DEBUG:boto:Token: None 2024-03-04T17:50:31.630 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.630 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo= 2024-03-04T17:50:31.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo=', 'Content-Length': '0'} 2024-03-04T17:50:31.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c81d0504b20deaa-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.634 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.635 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.635 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.635 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:31.635 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:31.635 DEBUG:boto:Method: GET 2024-03-04T17:50:31.635 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:50:31.635 DEBUG:boto:Data: 2024-03-04T17:50:31.635 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.635 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.636 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.636 DEBUG:boto:Params: {} 2024-03-04T17:50:31.636 DEBUG:boto:Token: None 2024-03-04T17:50:31.636 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.636 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo= 2024-03-04T17:50:31.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo=', 'Content-Length': '0'} 2024-03-04T17:50:31.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e14eeb30032d451e-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.684 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.684 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:31.684 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:31.685 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.685 DEBUG:boto:Path: /ppfahn-20/dummy 2024-03-04T17:50:31.685 DEBUG:boto:Data: 2024-03-04T17:50:31.685 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.685 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.685 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.685 DEBUG:boto:Params: {} 2024-03-04T17:50:31.685 DEBUG:boto:Token: None 2024-03-04T17:50:31.685 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/dummy 2024-03-04T17:50:31.685 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:13NNrQcTU+IutZGVDSnlFTIfE5k= 2024-03-04T17:50:31.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:13NNrQcTU+IutZGVDSnlFTIfE5k=', 'Content-Length': '0'} 2024-03-04T17:50:31.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed35d2921c5359a7-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.690 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:31.690 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:31.690 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.690 DEBUG:boto:Path: /ppfahn-20/dummy 2024-03-04T17:50:31.690 DEBUG:boto:Data: 2024-03-04T17:50:31.690 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.690 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.690 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.690 DEBUG:boto:Params: {} 2024-03-04T17:50:31.690 DEBUG:boto:Token: None 2024-03-04T17:50:31.691 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/dummy 2024-03-04T17:50:31.691 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:13NNrQcTU+IutZGVDSnlFTIfE5k= 2024-03-04T17:50:31.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:13NNrQcTU+IutZGVDSnlFTIfE5k=', 'Content-Length': '0'} 2024-03-04T17:50:31.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b192b6e36828413-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:31.695 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:31.695 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:31.695 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.695 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.695 DEBUG:boto:Method: GET 2024-03-04T17:50:31.695 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.695 DEBUG:boto:Data: 2024-03-04T17:50:31.695 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.695 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.696 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.696 DEBUG:boto:Params: {} 2024-03-04T17:50:31.696 DEBUG:boto:Token: None 2024-03-04T17:50:31.696 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.696 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6jKk/X1+kGo1YymWI2ZdJftxVkY= 2024-03-04T17:50:31.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jKk/X1+kGo1YymWI2ZdJftxVkY=', 'Content-Length': '0'} 2024-03-04T17:50:31.699 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000037333dc12d6a9a5a-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.699 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:31.699 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:31.699 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.699 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.700 DEBUG:boto:Method: GET 2024-03-04T17:50:31.700 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.700 DEBUG:boto:Data: 2024-03-04T17:50:31.700 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.700 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.700 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.700 DEBUG:boto:Params: {} 2024-03-04T17:50:31.700 DEBUG:boto:Token: None 2024-03-04T17:50:31.700 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:31.700 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6jKk/X1+kGo1YymWI2ZdJftxVkY= 2024-03-04T17:50:31.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jKk/X1+kGo1YymWI2ZdJftxVkY=', 'Content-Length': '0'} 2024-03-04T17:50:31.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef6c31759299342f-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.703 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.703 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.704 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:31.704 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:31.704 DEBUG:boto:Method: GET 2024-03-04T17:50:31.704 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:50:31.704 DEBUG:boto:Data: 2024-03-04T17:50:31.704 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.704 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.704 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.704 DEBUG:boto:Params: {} 2024-03-04T17:50:31.704 DEBUG:boto:Token: None 2024-03-04T17:50:31.705 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.705 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo= 2024-03-04T17:50:31.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo=', 'Content-Length': '0'} 2024-03-04T17:50:31.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000830446310f2b7aa-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.752 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.752 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.753 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.753 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:31.753 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:31.753 DEBUG:boto:Method: GET 2024-03-04T17:50:31.753 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:50:31.753 DEBUG:boto:Data: 2024-03-04T17:50:31.753 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.753 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.753 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.753 DEBUG:boto:Params: {} 2024-03-04T17:50:31.754 DEBUG:boto:Token: None 2024-03-04T17:50:31.754 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.754 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo= 2024-03-04T17:50:31.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo=', 'Content-Length': '0'} 2024-03-04T17:50:31.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000700c6abcbe370b29-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.758 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:31.759 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.759 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.759 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.759 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.759 DEBUG:boto:Method: GET 2024-03-04T17:50:31.759 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.759 DEBUG:boto:Data: 2024-03-04T17:50:31.760 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.760 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.760 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.760 DEBUG:boto:Params: {} 2024-03-04T17:50:31.760 DEBUG:boto:Token: None 2024-03-04T17:50:31.760 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.760 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SrVYncZq13UWIu6ANIc1D2N9D0A= 2024-03-04T17:50:31.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SrVYncZq13UWIu6ANIc1D2N9D0A=', 'Content-Length': '0'} 2024-03-04T17:50:31.763 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0a30b4dfdedf783-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.804 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.804 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.804 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.804 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.804 DEBUG:boto:Method: GET 2024-03-04T17:50:31.804 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.805 DEBUG:boto:Data: 2024-03-04T17:50:31.805 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.805 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.805 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.805 DEBUG:boto:Params: {} 2024-03-04T17:50:31.805 DEBUG:boto:Token: None 2024-03-04T17:50:31.805 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.805 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SrVYncZq13UWIu6ANIc1D2N9D0A= 2024-03-04T17:50:31.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SrVYncZq13UWIu6ANIc1D2N9D0A=', 'Content-Length': '0'} 2024-03-04T17:50:31.808 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5c9aecf60c3d033-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.856 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.856 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.856 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.857 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.857 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.857 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.857 DEBUG:boto:Data: 2024-03-04T17:50:31.857 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.857 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.857 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.857 DEBUG:boto:Params: {} 2024-03-04T17:50:31.857 DEBUG:boto:Token: None 2024-03-04T17:50:31.858 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.858 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yVlI4op9QU4W8ooEahHP8pTys5k= 2024-03-04T17:50:31.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yVlI4op9QU4W8ooEahHP8pTys5k=', 'Content-Length': '0'} 2024-03-04T17:50:31.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006459719746fcc9d2-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.861 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.861 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.861 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.861 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.861 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.861 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.861 DEBUG:boto:Data: 2024-03-04T17:50:31.862 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.862 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.862 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.862 DEBUG:boto:Params: {} 2024-03-04T17:50:31.862 DEBUG:boto:Token: None 2024-03-04T17:50:31.862 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.862 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yVlI4op9QU4W8ooEahHP8pTys5k= 2024-03-04T17:50:31.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yVlI4op9QU4W8ooEahHP8pTys5k=', 'Content-Length': '0'} 2024-03-04T17:50:31.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008618116f13e32c2c-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:31.866 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.866 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.866 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.866 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.866 DEBUG:boto:Method: GET 2024-03-04T17:50:31.866 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.866 DEBUG:boto:Data: 2024-03-04T17:50:31.866 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.866 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.866 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.866 DEBUG:boto:Params: {} 2024-03-04T17:50:31.867 DEBUG:boto:Token: None 2024-03-04T17:50:31.867 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.867 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SrVYncZq13UWIu6ANIc1D2N9D0A= 2024-03-04T17:50:31.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SrVYncZq13UWIu6ANIc1D2N9D0A=', 'Content-Length': '0'} 2024-03-04T17:50:31.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003237954ff579cfee-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.916 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.916 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.917 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.917 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.917 DEBUG:boto:Method: GET 2024-03-04T17:50:31.917 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.917 DEBUG:boto:Data: 2024-03-04T17:50:31.917 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.917 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.918 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.918 DEBUG:boto:Params: {} 2024-03-04T17:50:31.918 DEBUG:boto:Token: None 2024-03-04T17:50:31.918 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.918 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SrVYncZq13UWIu6ANIc1D2N9D0A= 2024-03-04T17:50:31.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SrVYncZq13UWIu6ANIc1D2N9D0A=', 'Content-Length': '0'} 2024-03-04T17:50:31.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044106dc314ebcfc7-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.923 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.923 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.923 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:31.923 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:31.923 DEBUG:boto:Method: GET 2024-03-04T17:50:31.924 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:31.924 DEBUG:boto:Data: 2024-03-04T17:50:31.924 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.924 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.924 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.924 DEBUG:boto:Params: {} 2024-03-04T17:50:31.925 DEBUG:boto:Token: None 2024-03-04T17:50:31.925 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.925 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo= 2024-03-04T17:50:31.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo=', 'Content-Length': '0'} 2024-03-04T17:50:31.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eff61b044d04206-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.929 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.930 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:31.930 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:31.930 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:31.930 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:31.930 DEBUG:boto:Method: GET 2024-03-04T17:50:31.930 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:31.930 DEBUG:boto:Data: 2024-03-04T17:50:31.930 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.930 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.930 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.931 DEBUG:boto:Params: {} 2024-03-04T17:50:31.931 DEBUG:boto:Token: None 2024-03-04T17:50:31.931 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/ 2024-03-04T17:50:31.931 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo= 2024-03-04T17:50:31.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AP4+u/7njDe6KEtVOTTzF9LAGBo=', 'Content-Length': '0'} 2024-03-04T17:50:31.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b250112903efb4f1-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.936 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:31.939 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.939 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.939 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.939 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.940 DEBUG:boto:Data: 2024-03-04T17:50:31.940 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.940 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.940 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.941 DEBUG:boto:Params: {} 2024-03-04T17:50:31.941 DEBUG:boto:Token: None 2024-03-04T17:50:31.941 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.941 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UYi14oi3szNQnBymXPs5ssculaY= 2024-03-04T17:50:31.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UYi14oi3szNQnBymXPs5ssculaY=', 'Content-Length': '0'} 2024-03-04T17:50:31.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000299e2e1779500747-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.945 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.946 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.946 DEBUG:boto:Method: HEAD 2024-03-04T17:50:31.946 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.946 DEBUG:boto:Data: 2024-03-04T17:50:31.946 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.946 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.946 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.946 DEBUG:boto:Params: {} 2024-03-04T17:50:31.947 DEBUG:boto:Token: None 2024-03-04T17:50:31.947 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.947 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UYi14oi3szNQnBymXPs5ssculaY= 2024-03-04T17:50:31.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UYi14oi3szNQnBymXPs5ssculaY=', 'Content-Length': '0'} 2024-03-04T17:50:31.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048275f1d810ee7a2-0065e609e7-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:31.951 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.951 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.951 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.952 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.952 DEBUG:boto:Method: GET 2024-03-04T17:50:31.952 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.952 DEBUG:boto:Data: 2024-03-04T17:50:31.952 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.952 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:31.952 DEBUG:boto:Port: 8001 2024-03-04T17:50:31.952 DEBUG:boto:Params: {} 2024-03-04T17:50:31.952 DEBUG:boto:Token: None 2024-03-04T17:50:31.953 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.953 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SrVYncZq13UWIu6ANIc1D2N9D0A= 2024-03-04T17:50:31.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SrVYncZq13UWIu6ANIc1D2N9D0A=', 'Content-Length': '0'} 2024-03-04T17:50:31.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000534fea16f9e4cbb3-0065e609e7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:31.996 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.996 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:31.996 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.996 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.996 DEBUG:boto:Method: GET 2024-03-04T17:50:31.997 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.997 DEBUG:boto:Data: 2024-03-04T17:50:31.997 DEBUG:boto:Headers: {} 2024-03-04T17:50:31.997 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:31.997 DEBUG:boto:Port: 8000 2024-03-04T17:50:31.997 DEBUG:boto:Params: {} 2024-03-04T17:50:31.998 DEBUG:boto:Token: None 2024-03-04T17:50:31.998 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:31 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:31.998 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SrVYncZq13UWIu6ANIc1D2N9D0A= 2024-03-04T17:50:31.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SrVYncZq13UWIu6ANIc1D2N9D0A=', 'Content-Length': '0'} 2024-03-04T17:50:32.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000257057514f285a1c-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.002 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.002 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.002 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:32.002 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:32.002 DEBUG:boto:Method: GET 2024-03-04T17:50:32.002 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:32.002 DEBUG:boto:Data: 2024-03-04T17:50:32.002 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.002 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.003 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.003 DEBUG:boto:Params: {} 2024-03-04T17:50:32.003 DEBUG:boto:Token: None 2024-03-04T17:50:32.003 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.003 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffa26e0e0349140b-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.007 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.008 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.008 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.008 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:32.008 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:32.008 DEBUG:boto:Method: GET 2024-03-04T17:50:32.009 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:32.009 DEBUG:boto:Data: 2024-03-04T17:50:32.009 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.009 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.009 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.009 DEBUG:boto:Params: {} 2024-03-04T17:50:32.009 DEBUG:boto:Token: None 2024-03-04T17:50:32.009 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.009 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9622ae712580b3d-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.014 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:32.015 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.015 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.016 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.016 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.016 DEBUG:boto:Method: GET 2024-03-04T17:50:32.016 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.016 DEBUG:boto:Data: 2024-03-04T17:50:32.017 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.017 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.017 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.017 DEBUG:boto:Params: {} 2024-03-04T17:50:32.017 DEBUG:boto:Token: None 2024-03-04T17:50:32.017 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.017 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FvLYoNqm8tRKD+hMcYO8iEf0ny8= 2024-03-04T17:50:32.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FvLYoNqm8tRKD+hMcYO8iEf0ny8=', 'Content-Length': '0'} 2024-03-04T17:50:32.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000259eb22315081100-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.021 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.021 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.021 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.021 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.021 DEBUG:boto:Method: GET 2024-03-04T17:50:32.022 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.022 DEBUG:boto:Data: 2024-03-04T17:50:32.022 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.022 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.022 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.023 DEBUG:boto:Params: {} 2024-03-04T17:50:32.023 DEBUG:boto:Token: None 2024-03-04T17:50:32.023 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.023 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FvLYoNqm8tRKD+hMcYO8iEf0ny8= 2024-03-04T17:50:32.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FvLYoNqm8tRKD+hMcYO8iEf0ny8=', 'Content-Length': '0'} 2024-03-04T17:50:32.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c35449e5983125af-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.068 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.068 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.068 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.068 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.069 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.069 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.069 DEBUG:boto:Data: 2024-03-04T17:50:32.069 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.069 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.069 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.069 DEBUG:boto:Params: {} 2024-03-04T17:50:32.069 DEBUG:boto:Token: None 2024-03-04T17:50:32.070 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.070 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aBI5e8CqoTehiyt5Xss6x/W3eHA= 2024-03-04T17:50:32.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aBI5e8CqoTehiyt5Xss6x/W3eHA=', 'Content-Length': '0'} 2024-03-04T17:50:32.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007329b79c2305a576-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.074 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.074 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.074 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.074 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.075 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.075 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.075 DEBUG:boto:Data: 2024-03-04T17:50:32.075 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.075 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.075 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.075 DEBUG:boto:Params: {} 2024-03-04T17:50:32.075 DEBUG:boto:Token: None 2024-03-04T17:50:32.075 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.076 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aBI5e8CqoTehiyt5Xss6x/W3eHA= 2024-03-04T17:50:32.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aBI5e8CqoTehiyt5Xss6x/W3eHA=', 'Content-Length': '0'} 2024-03-04T17:50:32.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbcfec0aefe74e23-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:32.079 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.079 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.080 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.080 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.080 DEBUG:boto:Method: GET 2024-03-04T17:50:32.080 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.080 DEBUG:boto:Data: 2024-03-04T17:50:32.080 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.080 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.080 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.080 DEBUG:boto:Params: {} 2024-03-04T17:50:32.081 DEBUG:boto:Token: None 2024-03-04T17:50:32.081 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.081 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FvLYoNqm8tRKD+hMcYO8iEf0ny8= 2024-03-04T17:50:32.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FvLYoNqm8tRKD+hMcYO8iEf0ny8=', 'Content-Length': '0'} 2024-03-04T17:50:32.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e4f3c13aea7b5a6-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.124 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.124 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.124 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.125 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.125 DEBUG:boto:Method: GET 2024-03-04T17:50:32.125 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.125 DEBUG:boto:Data: 2024-03-04T17:50:32.125 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.125 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.125 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.125 DEBUG:boto:Params: {} 2024-03-04T17:50:32.125 DEBUG:boto:Token: None 2024-03-04T17:50:32.125 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.126 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FvLYoNqm8tRKD+hMcYO8iEf0ny8= 2024-03-04T17:50:32.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FvLYoNqm8tRKD+hMcYO8iEf0ny8=', 'Content-Length': '0'} 2024-03-04T17:50:32.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021da764c93affda2-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.172 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.172 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.172 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:32.172 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:32.173 DEBUG:boto:Method: GET 2024-03-04T17:50:32.173 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:32.173 DEBUG:boto:Data: 2024-03-04T17:50:32.173 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.173 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.173 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.173 DEBUG:boto:Params: {} 2024-03-04T17:50:32.173 DEBUG:boto:Token: None 2024-03-04T17:50:32.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.174 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7b51f8ce983a318-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.178 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.178 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.178 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.178 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:32.179 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:32.179 DEBUG:boto:Method: GET 2024-03-04T17:50:32.179 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:32.179 DEBUG:boto:Data: 2024-03-04T17:50:32.179 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.179 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.179 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.179 DEBUG:boto:Params: {} 2024-03-04T17:50:32.179 DEBUG:boto:Token: None 2024-03-04T17:50:32.179 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.180 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bf7f01f158cfcf3-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.184 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.185 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.185 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.185 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.185 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.185 DEBUG:boto:Data: 2024-03-04T17:50:32.185 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.185 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.185 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.185 DEBUG:boto:Params: {} 2024-03-04T17:50:32.186 DEBUG:boto:Token: None 2024-03-04T17:50:32.186 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.186 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qQezsSL3bnLmQOHOhpMTsRoTPW4= 2024-03-04T17:50:32.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qQezsSL3bnLmQOHOhpMTsRoTPW4=', 'Content-Length': '0'} 2024-03-04T17:50:32.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aab78d87c67dfbf7-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.190 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.190 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.190 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.190 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.190 DEBUG:boto:Data: 2024-03-04T17:50:32.190 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.190 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.190 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.190 DEBUG:boto:Params: {} 2024-03-04T17:50:32.190 DEBUG:boto:Token: None 2024-03-04T17:50:32.191 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.191 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qQezsSL3bnLmQOHOhpMTsRoTPW4= 2024-03-04T17:50:32.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qQezsSL3bnLmQOHOhpMTsRoTPW4=', 'Content-Length': '0'} 2024-03-04T17:50:32.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085ceaae36f32250e-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:32.195 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.195 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.195 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.195 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.195 DEBUG:boto:Method: GET 2024-03-04T17:50:32.195 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.195 DEBUG:boto:Data: 2024-03-04T17:50:32.195 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.195 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.196 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.196 DEBUG:boto:Params: {} 2024-03-04T17:50:32.196 DEBUG:boto:Token: None 2024-03-04T17:50:32.196 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.196 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FvLYoNqm8tRKD+hMcYO8iEf0ny8= 2024-03-04T17:50:32.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FvLYoNqm8tRKD+hMcYO8iEf0ny8=', 'Content-Length': '0'} 2024-03-04T17:50:32.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000792a494ea9be625d-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.244 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.244 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:32.244 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.244 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.244 DEBUG:boto:Method: GET 2024-03-04T17:50:32.245 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.245 DEBUG:boto:Data: 2024-03-04T17:50:32.245 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.245 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.245 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.245 DEBUG:boto:Params: {} 2024-03-04T17:50:32.245 DEBUG:boto:Token: None 2024-03-04T17:50:32.245 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:32.245 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FvLYoNqm8tRKD+hMcYO8iEf0ny8= 2024-03-04T17:50:32.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FvLYoNqm8tRKD+hMcYO8iEf0ny8=', 'Content-Length': '0'} 2024-03-04T17:50:32.248 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc14b0d3347ac9df-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.292 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.292 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.292 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:32.292 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:32.293 DEBUG:boto:Method: GET 2024-03-04T17:50:32.293 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:32.293 DEBUG:boto:Data: 2024-03-04T17:50:32.293 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.293 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.293 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.293 DEBUG:boto:Params: {} 2024-03-04T17:50:32.293 DEBUG:boto:Token: None 2024-03-04T17:50:32.293 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.293 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064e8f0d25e92c273-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.340 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.340 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.341 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.341 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:32.341 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:32.341 DEBUG:boto:Method: GET 2024-03-04T17:50:32.341 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:32.341 DEBUG:boto:Data: 2024-03-04T17:50:32.341 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.341 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.341 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.341 DEBUG:boto:Params: {} 2024-03-04T17:50:32.341 DEBUG:boto:Token: None 2024-03-04T17:50:32.342 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.342 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a066cedcfd0424-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.347 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:32.347 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.347 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.347 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.348 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.348 DEBUG:boto:Method: GET 2024-03-04T17:50:32.348 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.348 DEBUG:boto:Data: 2024-03-04T17:50:32.348 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.348 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.348 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.348 DEBUG:boto:Params: {} 2024-03-04T17:50:32.348 DEBUG:boto:Token: None 2024-03-04T17:50:32.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.349 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Xhhh47xEF5/+xUCWfaULWRd0bj0= 2024-03-04T17:50:32.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Xhhh47xEF5/+xUCWfaULWRd0bj0=', 'Content-Length': '0'} 2024-03-04T17:50:32.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001945c1b40049759a-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.352 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.352 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.352 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.352 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.353 DEBUG:boto:Method: GET 2024-03-04T17:50:32.353 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.353 DEBUG:boto:Data: 2024-03-04T17:50:32.353 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.353 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.353 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.353 DEBUG:boto:Params: {} 2024-03-04T17:50:32.353 DEBUG:boto:Token: None 2024-03-04T17:50:32.353 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.353 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Xhhh47xEF5/+xUCWfaULWRd0bj0= 2024-03-04T17:50:32.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Xhhh47xEF5/+xUCWfaULWRd0bj0=', 'Content-Length': '0'} 2024-03-04T17:50:32.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078d7a01f45462bcd-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.400 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.400 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.400 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.400 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.400 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.400 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.401 DEBUG:boto:Data: 2024-03-04T17:50:32.401 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.401 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.401 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.401 DEBUG:boto:Params: {} 2024-03-04T17:50:32.401 DEBUG:boto:Token: None 2024-03-04T17:50:32.401 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.401 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IR9bBeTWduAFYATAxAYs0rsiO24= 2024-03-04T17:50:32.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IR9bBeTWduAFYATAxAYs0rsiO24=', 'Content-Length': '0'} 2024-03-04T17:50:32.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008327b5a43b6fbf7a-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.405 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.405 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.405 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.405 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.405 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.405 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.405 DEBUG:boto:Data: 2024-03-04T17:50:32.405 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.406 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.406 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.406 DEBUG:boto:Params: {} 2024-03-04T17:50:32.406 DEBUG:boto:Token: None 2024-03-04T17:50:32.406 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.406 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IR9bBeTWduAFYATAxAYs0rsiO24= 2024-03-04T17:50:32.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IR9bBeTWduAFYATAxAYs0rsiO24=', 'Content-Length': '0'} 2024-03-04T17:50:32.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2a62a728bc3909f-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:32.409 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.409 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.410 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.410 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.410 DEBUG:boto:Method: GET 2024-03-04T17:50:32.410 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.410 DEBUG:boto:Data: 2024-03-04T17:50:32.410 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.410 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.410 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.410 DEBUG:boto:Params: {} 2024-03-04T17:50:32.410 DEBUG:boto:Token: None 2024-03-04T17:50:32.411 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.411 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Xhhh47xEF5/+xUCWfaULWRd0bj0= 2024-03-04T17:50:32.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Xhhh47xEF5/+xUCWfaULWRd0bj0=', 'Content-Length': '0'} 2024-03-04T17:50:32.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014fec761f1ad1f83-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.414 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.414 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.414 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.414 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.414 DEBUG:boto:Method: GET 2024-03-04T17:50:32.414 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.414 DEBUG:boto:Data: 2024-03-04T17:50:32.414 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.414 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.415 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.415 DEBUG:boto:Params: {} 2024-03-04T17:50:32.415 DEBUG:boto:Token: None 2024-03-04T17:50:32.415 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.415 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Xhhh47xEF5/+xUCWfaULWRd0bj0= 2024-03-04T17:50:32.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Xhhh47xEF5/+xUCWfaULWRd0bj0=', 'Content-Length': '0'} 2024-03-04T17:50:32.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d6da5e65d97aa6f-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.460 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.460 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.460 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:32.460 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:32.460 DEBUG:boto:Method: GET 2024-03-04T17:50:32.461 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:32.461 DEBUG:boto:Data: 2024-03-04T17:50:32.461 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.461 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.461 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.461 DEBUG:boto:Params: {} 2024-03-04T17:50:32.461 DEBUG:boto:Token: None 2024-03-04T17:50:32.461 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.461 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000293952f5c006490b-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.466 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.466 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.467 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.467 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:32.467 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:32.467 DEBUG:boto:Method: GET 2024-03-04T17:50:32.467 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:32.467 DEBUG:boto:Data: 2024-03-04T17:50:32.467 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.467 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.467 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.467 DEBUG:boto:Params: {} 2024-03-04T17:50:32.468 DEBUG:boto:Token: None 2024-03-04T17:50:32.468 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.468 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ba87b4b35cba458-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.472 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.472 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.473 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.473 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.473 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.473 DEBUG:boto:Data: 2024-03-04T17:50:32.473 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.473 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.473 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.473 DEBUG:boto:Params: {} 2024-03-04T17:50:32.473 DEBUG:boto:Token: None 2024-03-04T17:50:32.473 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.474 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RksFYp4TpjIp+E4EP8FHsezTGMM= 2024-03-04T17:50:32.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RksFYp4TpjIp+E4EP8FHsezTGMM=', 'Content-Length': '0'} 2024-03-04T17:50:32.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efc370d851413027-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.478 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.478 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.478 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.478 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.478 DEBUG:boto:Data: 2024-03-04T17:50:32.478 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.478 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.478 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.479 DEBUG:boto:Params: {} 2024-03-04T17:50:32.479 DEBUG:boto:Token: None 2024-03-04T17:50:32.479 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.479 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RksFYp4TpjIp+E4EP8FHsezTGMM= 2024-03-04T17:50:32.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RksFYp4TpjIp+E4EP8FHsezTGMM=', 'Content-Length': '0'} 2024-03-04T17:50:32.483 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003338ba06d59812dc-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:32.483 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.483 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.484 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.484 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.484 DEBUG:boto:Method: GET 2024-03-04T17:50:32.484 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.484 DEBUG:boto:Data: 2024-03-04T17:50:32.484 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.484 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.484 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.484 DEBUG:boto:Params: {} 2024-03-04T17:50:32.484 DEBUG:boto:Token: None 2024-03-04T17:50:32.485 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.485 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Xhhh47xEF5/+xUCWfaULWRd0bj0= 2024-03-04T17:50:32.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Xhhh47xEF5/+xUCWfaULWRd0bj0=', 'Content-Length': '0'} 2024-03-04T17:50:32.488 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006449ca663fe86298-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.528 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.528 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:32.528 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.528 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.528 DEBUG:boto:Method: GET 2024-03-04T17:50:32.529 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.529 DEBUG:boto:Data: 2024-03-04T17:50:32.529 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.529 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.529 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.529 DEBUG:boto:Params: {} 2024-03-04T17:50:32.529 DEBUG:boto:Token: None 2024-03-04T17:50:32.529 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:32.529 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Xhhh47xEF5/+xUCWfaULWRd0bj0= 2024-03-04T17:50:32.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Xhhh47xEF5/+xUCWfaULWRd0bj0=', 'Content-Length': '0'} 2024-03-04T17:50:32.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075a5aa6fc81c554b-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.576 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.576 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.577 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:32.577 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:32.577 DEBUG:boto:Method: GET 2024-03-04T17:50:32.577 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:32.577 DEBUG:boto:Data: 2024-03-04T17:50:32.577 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.577 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.577 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.577 DEBUG:boto:Params: {} 2024-03-04T17:50:32.578 DEBUG:boto:Token: None 2024-03-04T17:50:32.578 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.578 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080b4c638b5b28564-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.582 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.583 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.583 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.583 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:32.583 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:32.583 DEBUG:boto:Method: GET 2024-03-04T17:50:32.583 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:32.583 DEBUG:boto:Data: 2024-03-04T17:50:32.584 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.584 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:32.584 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.584 DEBUG:boto:Params: {} 2024-03-04T17:50:32.584 DEBUG:boto:Token: None 2024-03-04T17:50:32.584 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.584 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdf3b3b09d695aca-0065e609e8-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.589 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.589 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-20 zones={test-zone3, test-zone1} 2024-03-04T17:50:32.589 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-20 zones={test-zone3, test-zone2} 2024-03-04T17:50:32.589 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.590 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.590 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.590 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:50:32.590 DEBUG:boto:Data: 2024-03-04T17:50:32.590 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.590 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.590 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.590 DEBUG:boto:Params: {} 2024-03-04T17:50:32.590 DEBUG:boto:Token: None 2024-03-04T17:50:32.591 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.591 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W3k7pyF87BK4CyQbwpOexFW8YOE= 2024-03-04T17:50:32.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W3k7pyF87BK4CyQbwpOexFW8YOE=', 'Content-Length': '0'} 2024-03-04T17:50:32.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000610af7e72a08898e-0065e609e8-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.594 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.595 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.595 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.595 DEBUG:boto:Path: /ppfahn-20/ 2024-03-04T17:50:32.595 DEBUG:boto:Data: 2024-03-04T17:50:32.595 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.595 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.595 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.595 DEBUG:boto:Params: {} 2024-03-04T17:50:32.595 DEBUG:boto:Token: None 2024-03-04T17:50:32.595 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.596 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W3k7pyF87BK4CyQbwpOexFW8YOE= 2024-03-04T17:50:32.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W3k7pyF87BK4CyQbwpOexFW8YOE=', 'Content-Length': '0'} 2024-03-04T17:50:32.600 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005cbc805ec42f1e2e-0065e609e8-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.600 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:50:32.600 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.600 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.601 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:32.601 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:32.601 DEBUG:boto:Method: GET 2024-03-04T17:50:32.601 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:32.601 DEBUG:boto:Data: 2024-03-04T17:50:32.601 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.601 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.601 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.601 DEBUG:boto:Params: {} 2024-03-04T17:50:32.601 DEBUG:boto:Token: None 2024-03-04T17:50:32.602 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/?versions 2024-03-04T17:50:32.602 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nzp7gymI8f1j8nboowdW90JKCWg= 2024-03-04T17:50:32.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nzp7gymI8f1j8nboowdW90JKCWg=', 'Content-Length': '0'} 2024-03-04T17:50:32.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0b9ef53db3a3ecf-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.606 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.607 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:32.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:32.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:32.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:32.607 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:50:32.607 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.607 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.607 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:32.607 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:32.608 DEBUG:boto:Method: GET 2024-03-04T17:50:32.608 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:32.608 DEBUG:boto:Data: 2024-03-04T17:50:32.608 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.608 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.608 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.608 DEBUG:boto:Params: {} 2024-03-04T17:50:32.608 DEBUG:boto:Token: None 2024-03-04T17:50:32.608 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/?versions 2024-03-04T17:50:32.609 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nzp7gymI8f1j8nboowdW90JKCWg= 2024-03-04T17:50:32.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nzp7gymI8f1j8nboowdW90JKCWg=', 'Content-Length': '0'} 2024-03-04T17:50:32.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020f612cb72bff4c8-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.613 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.614 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:32.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:32.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:32.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:32.614 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.614 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.614 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:32.614 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:32.615 DEBUG:boto:Method: GET 2024-03-04T17:50:32.615 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:32.615 DEBUG:boto:Data: 2024-03-04T17:50:32.615 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.615 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.615 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.615 DEBUG:boto:Params: {} 2024-03-04T17:50:32.615 DEBUG:boto:Token: None 2024-03-04T17:50:32.615 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/?versions 2024-03-04T17:50:32.616 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nzp7gymI8f1j8nboowdW90JKCWg= 2024-03-04T17:50:32.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nzp7gymI8f1j8nboowdW90JKCWg=', 'Content-Length': '0'} 2024-03-04T17:50:32.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013cfb40e032d8c04-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.621 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.621 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.621 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.621 DEBUG:boto:path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:32.621 DEBUG:boto:auth_path=/ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:32.621 DEBUG:boto:Method: GET 2024-03-04T17:50:32.622 DEBUG:boto:Path: /ppfahn-20/?versions&max-keys=999 2024-03-04T17:50:32.622 DEBUG:boto:Data: 2024-03-04T17:50:32.622 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.622 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.622 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.622 DEBUG:boto:Params: {} 2024-03-04T17:50:32.622 DEBUG:boto:Token: None 2024-03-04T17:50:32.623 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/?versions 2024-03-04T17:50:32.623 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nzp7gymI8f1j8nboowdW90JKCWg= 2024-03-04T17:50:32.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nzp7gymI8f1j8nboowdW90JKCWg=', 'Content-Length': '0'} 2024-03-04T17:50:32.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1c482f2d1d3ab4f-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.628 DEBUG:boto:b'ppfahn-20999falsedummy273mi4WSqIddPDF3PD9XAIwhFdNxv9btrue2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19Nn63Wf7xVnzErCBAt.4UYmUg.L.38Ltrue2024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-danbM6QbPvBQ8R9uWBluKiMLcLfHcOtrue2024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3.DJ9hLU-KeSyKdnhOoPa6kBqV55BraKtrue2024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:32.629 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:32.629 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:32.629 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.629 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.629 DEBUG:boto:Method: GET 2024-03-04T17:50:32.629 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.629 DEBUG:boto:Data: 2024-03-04T17:50:32.630 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.630 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.630 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.630 DEBUG:boto:Params: {} 2024-03-04T17:50:32.630 DEBUG:boto:Token: None 2024-03-04T17:50:32.630 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.630 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M3DAHpPdWFUhdBaUu1wKf4uJpRg= 2024-03-04T17:50:32.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M3DAHpPdWFUhdBaUu1wKf4uJpRg=', 'Content-Length': '0'} 2024-03-04T17:50:32.633 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003212bede9117be79-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.634 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:32.634 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:32.634 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.634 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.634 DEBUG:boto:Method: GET 2024-03-04T17:50:32.634 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.634 DEBUG:boto:Data: 2024-03-04T17:50:32.634 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.634 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.634 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.634 DEBUG:boto:Params: {} 2024-03-04T17:50:32.635 DEBUG:boto:Token: None 2024-03-04T17:50:32.635 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.635 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M3DAHpPdWFUhdBaUu1wKf4uJpRg= 2024-03-04T17:50:32.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M3DAHpPdWFUhdBaUu1wKf4uJpRg=', 'Content-Length': '0'} 2024-03-04T17:50:32.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008ce9c3c15ac0f684-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.638 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:32.638 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:32.638 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.638 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.638 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.638 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.638 DEBUG:boto:Data: 2024-03-04T17:50:32.639 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.639 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.639 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.639 DEBUG:boto:Params: {} 2024-03-04T17:50:32.639 DEBUG:boto:Token: None 2024-03-04T17:50:32.639 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.639 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3HTEkDVB9syesYndOODg0bcyoOU= 2024-03-04T17:50:32.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3HTEkDVB9syesYndOODg0bcyoOU=', 'Content-Length': '0'} 2024-03-04T17:50:32.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000528ac31fddf72283-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.643 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:32.643 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:32.643 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.643 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.643 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.643 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.643 DEBUG:boto:Data: 2024-03-04T17:50:32.643 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.643 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.643 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.643 DEBUG:boto:Params: {} 2024-03-04T17:50:32.643 DEBUG:boto:Token: None 2024-03-04T17:50:32.644 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.644 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3HTEkDVB9syesYndOODg0bcyoOU= 2024-03-04T17:50:32.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3HTEkDVB9syesYndOODg0bcyoOU=', 'Content-Length': '0'} 2024-03-04T17:50:32.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036ca4d0f5be9f07a-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:32.647 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:32.647 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:32.647 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.647 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.647 DEBUG:boto:Method: GET 2024-03-04T17:50:32.648 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.648 DEBUG:boto:Data: 2024-03-04T17:50:32.648 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.648 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.648 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.648 DEBUG:boto:Params: {} 2024-03-04T17:50:32.648 DEBUG:boto:Token: None 2024-03-04T17:50:32.648 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.648 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M3DAHpPdWFUhdBaUu1wKf4uJpRg= 2024-03-04T17:50:32.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M3DAHpPdWFUhdBaUu1wKf4uJpRg=', 'Content-Length': '0'} 2024-03-04T17:50:32.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023bc9ca6d8af2394-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.651 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:32.652 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:32.652 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.652 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.652 DEBUG:boto:Method: GET 2024-03-04T17:50:32.652 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.652 DEBUG:boto:Data: 2024-03-04T17:50:32.652 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.652 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.652 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.652 DEBUG:boto:Params: {} 2024-03-04T17:50:32.653 DEBUG:boto:Token: None 2024-03-04T17:50:32.653 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.653 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M3DAHpPdWFUhdBaUu1wKf4uJpRg= 2024-03-04T17:50:32.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M3DAHpPdWFUhdBaUu1wKf4uJpRg=', 'Content-Length': '0'} 2024-03-04T17:50:32.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c37dbf165b24d75e-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.656 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.656 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.656 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:32.656 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:32.657 DEBUG:boto:Method: GET 2024-03-04T17:50:32.657 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:50:32.657 DEBUG:boto:Data: 2024-03-04T17:50:32.657 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.657 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.657 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.657 DEBUG:boto:Params: {} 2024-03-04T17:50:32.657 DEBUG:boto:Token: None 2024-03-04T17:50:32.657 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.657 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b67f083dc1fd263-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.704 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.704 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.705 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.705 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:32.705 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:32.705 DEBUG:boto:Method: GET 2024-03-04T17:50:32.706 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:50:32.706 DEBUG:boto:Data: 2024-03-04T17:50:32.706 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.706 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.706 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.706 DEBUG:boto:Params: {} 2024-03-04T17:50:32.706 DEBUG:boto:Token: None 2024-03-04T17:50:32.706 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.706 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee5d59822bcd2c36-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.711 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.711 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:32.711 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:32.711 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.711 DEBUG:boto:Path: /ppfahn-20/dummy 2024-03-04T17:50:32.712 DEBUG:boto:Data: 2024-03-04T17:50:32.712 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.712 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.712 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.712 DEBUG:boto:Params: {} 2024-03-04T17:50:32.712 DEBUG:boto:Token: None 2024-03-04T17:50:32.712 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/dummy 2024-03-04T17:50:32.712 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3GQoJWdMCpipq5wn/WMYtuyi4h0= 2024-03-04T17:50:32.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3GQoJWdMCpipq5wn/WMYtuyi4h0=', 'Content-Length': '0'} 2024-03-04T17:50:32.716 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052c6eff9ba4fa7c5-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.717 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:32.717 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:32.717 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.717 DEBUG:boto:Path: /ppfahn-20/dummy 2024-03-04T17:50:32.717 DEBUG:boto:Data: 2024-03-04T17:50:32.717 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.717 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.717 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.717 DEBUG:boto:Params: {} 2024-03-04T17:50:32.717 DEBUG:boto:Token: None 2024-03-04T17:50:32.718 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/dummy 2024-03-04T17:50:32.718 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3GQoJWdMCpipq5wn/WMYtuyi4h0= 2024-03-04T17:50:32.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3GQoJWdMCpipq5wn/WMYtuyi4h0=', 'Content-Length': '0'} 2024-03-04T17:50:32.721 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007fec8249010b1135-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:32.722 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:32.722 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:32.722 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.722 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.722 DEBUG:boto:Method: GET 2024-03-04T17:50:32.722 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.722 DEBUG:boto:Data: 2024-03-04T17:50:32.722 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.722 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.723 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.723 DEBUG:boto:Params: {} 2024-03-04T17:50:32.723 DEBUG:boto:Token: None 2024-03-04T17:50:32.723 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.723 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M3DAHpPdWFUhdBaUu1wKf4uJpRg= 2024-03-04T17:50:32.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M3DAHpPdWFUhdBaUu1wKf4uJpRg=', 'Content-Length': '0'} 2024-03-04T17:50:32.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b0446fb5f8f8426-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.726 DEBUG:boto:path=/ppfahn-20/dummy 2024-03-04T17:50:32.726 DEBUG:boto:auth_path=/ppfahn-20/dummy 2024-03-04T17:50:32.726 DEBUG:boto:path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.726 DEBUG:boto:auth_path=/ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.726 DEBUG:boto:Method: GET 2024-03-04T17:50:32.727 DEBUG:boto:Path: /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.727 DEBUG:boto:Data: 2024-03-04T17:50:32.727 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.727 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.727 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.727 DEBUG:boto:Params: {} 2024-03-04T17:50:32.727 DEBUG:boto:Token: None 2024-03-04T17:50:32.727 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/dummy?versionId=273mi4WSqIddPDF3PD9XAIwhFdNxv9b 2024-03-04T17:50:32.727 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M3DAHpPdWFUhdBaUu1wKf4uJpRg= 2024-03-04T17:50:32.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M3DAHpPdWFUhdBaUu1wKf4uJpRg=', 'Content-Length': '0'} 2024-03-04T17:50:32.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:10 GMT'), ('x-amz-version-id', '273mi4WSqIddPDF3PD9XAIwhFdNxv9b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c8b16f3f4f3ac38e-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.730 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.731 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.731 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:32.731 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:32.731 DEBUG:boto:Method: GET 2024-03-04T17:50:32.732 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:50:32.732 DEBUG:boto:Data: 2024-03-04T17:50:32.732 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.732 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.732 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.732 DEBUG:boto:Params: {} 2024-03-04T17:50:32.732 DEBUG:boto:Token: None 2024-03-04T17:50:32.732 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.732 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000437a22f581330746-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.737 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.738 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.738 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.738 DEBUG:boto:path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:32.738 DEBUG:boto:auth_path=/ppfahn-20/?prefix=dummy 2024-03-04T17:50:32.738 DEBUG:boto:Method: GET 2024-03-04T17:50:32.738 DEBUG:boto:Path: /ppfahn-20/?prefix=dummy 2024-03-04T17:50:32.738 DEBUG:boto:Data: 2024-03-04T17:50:32.738 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.738 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.738 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.738 DEBUG:boto:Params: {} 2024-03-04T17:50:32.739 DEBUG:boto:Token: None 2024-03-04T17:50:32.739 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.739 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003960943681c37967-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.743 DEBUG:boto:b'ppfahn-20dummy1000falsedummy2024-03-04T17:46:10.174Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:32.744 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.744 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.744 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.744 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.744 DEBUG:boto:Method: GET 2024-03-04T17:50:32.744 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.744 DEBUG:boto:Data: 2024-03-04T17:50:32.744 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.744 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.744 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.745 DEBUG:boto:Params: {} 2024-03-04T17:50:32.745 DEBUG:boto:Token: None 2024-03-04T17:50:32.745 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.745 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QVOhNif7A9RRekrePeDJkhmm3Pk= 2024-03-04T17:50:32.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QVOhNif7A9RRekrePeDJkhmm3Pk=', 'Content-Length': '0'} 2024-03-04T17:50:32.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033b5da698155d755-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.788 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.788 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.788 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.789 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.789 DEBUG:boto:Method: GET 2024-03-04T17:50:32.789 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.789 DEBUG:boto:Data: 2024-03-04T17:50:32.789 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.789 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.789 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.789 DEBUG:boto:Params: {} 2024-03-04T17:50:32.789 DEBUG:boto:Token: None 2024-03-04T17:50:32.790 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.790 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QVOhNif7A9RRekrePeDJkhmm3Pk= 2024-03-04T17:50:32.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QVOhNif7A9RRekrePeDJkhmm3Pk=', 'Content-Length': '0'} 2024-03-04T17:50:32.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000066a9aca6806a644-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.793 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.793 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.794 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.794 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.794 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.794 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.794 DEBUG:boto:Data: 2024-03-04T17:50:32.794 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.794 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.794 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.794 DEBUG:boto:Params: {} 2024-03-04T17:50:32.794 DEBUG:boto:Token: None 2024-03-04T17:50:32.795 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.795 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xneWgIyqNfA3lhJyZdfpDzFokxI= 2024-03-04T17:50:32.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xneWgIyqNfA3lhJyZdfpDzFokxI=', 'Content-Length': '0'} 2024-03-04T17:50:32.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a599e05a9c4be0b6-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.798 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.798 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.798 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.798 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.798 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.798 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.798 DEBUG:boto:Data: 2024-03-04T17:50:32.798 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.799 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.799 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.799 DEBUG:boto:Params: {} 2024-03-04T17:50:32.799 DEBUG:boto:Token: None 2024-03-04T17:50:32.799 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.799 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xneWgIyqNfA3lhJyZdfpDzFokxI= 2024-03-04T17:50:32.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xneWgIyqNfA3lhJyZdfpDzFokxI=', 'Content-Length': '0'} 2024-03-04T17:50:32.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d53a4536744ec6d-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:32.802 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.803 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.803 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.803 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.803 DEBUG:boto:Method: GET 2024-03-04T17:50:32.803 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.803 DEBUG:boto:Data: 2024-03-04T17:50:32.803 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.803 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.803 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.803 DEBUG:boto:Params: {} 2024-03-04T17:50:32.804 DEBUG:boto:Token: None 2024-03-04T17:50:32.804 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.804 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QVOhNif7A9RRekrePeDJkhmm3Pk= 2024-03-04T17:50:32.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QVOhNif7A9RRekrePeDJkhmm3Pk=', 'Content-Length': '0'} 2024-03-04T17:50:32.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000461c7052d1b3fddb-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.848 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.848 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.848 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.848 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.848 DEBUG:boto:Method: GET 2024-03-04T17:50:32.849 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.849 DEBUG:boto:Data: 2024-03-04T17:50:32.849 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.849 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.849 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.849 DEBUG:boto:Params: {} 2024-03-04T17:50:32.849 DEBUG:boto:Token: None 2024-03-04T17:50:32.849 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.849 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QVOhNif7A9RRekrePeDJkhmm3Pk= 2024-03-04T17:50:32.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QVOhNif7A9RRekrePeDJkhmm3Pk=', 'Content-Length': '0'} 2024-03-04T17:50:32.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8a7ae632eb92474-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.900 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.900 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.900 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:32.900 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:32.900 DEBUG:boto:Method: GET 2024-03-04T17:50:32.901 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:32.901 DEBUG:boto:Data: 2024-03-04T17:50:32.901 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.901 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.901 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.901 DEBUG:boto:Params: {} 2024-03-04T17:50:32.901 DEBUG:boto:Token: None 2024-03-04T17:50:32.901 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.901 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d89c2e0188b37f9b-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.952 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.952 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:32.953 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:32.953 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:32.953 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:32.953 DEBUG:boto:Method: GET 2024-03-04T17:50:32.953 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:32.953 DEBUG:boto:Data: 2024-03-04T17:50:32.953 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.953 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.953 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.954 DEBUG:boto:Params: {} 2024-03-04T17:50:32.954 DEBUG:boto:Token: None 2024-03-04T17:50:32.954 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/ 2024-03-04T17:50:32.954 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw= 2024-03-04T17:50:32.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:874/kWgMJZ+ytdiaIwg9r3uV6Xw=', 'Content-Length': '0'} 2024-03-04T17:50:32.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004288a65f28a3ff4d-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.958 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:32.959 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.959 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.959 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.959 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.959 DEBUG:boto:Data: 2024-03-04T17:50:32.959 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.959 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.959 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.959 DEBUG:boto:Params: {} 2024-03-04T17:50:32.960 DEBUG:boto:Token: None 2024-03-04T17:50:32.960 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.960 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FE3uyppOACywBgRs4PT/FYqx2EQ= 2024-03-04T17:50:32.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FE3uyppOACywBgRs4PT/FYqx2EQ=', 'Content-Length': '0'} 2024-03-04T17:50:32.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001da12ae9098c440a-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.967 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.967 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.968 DEBUG:boto:Method: HEAD 2024-03-04T17:50:32.968 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.968 DEBUG:boto:Data: 2024-03-04T17:50:32.968 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.968 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:32.968 DEBUG:boto:Port: 8000 2024-03-04T17:50:32.968 DEBUG:boto:Params: {} 2024-03-04T17:50:32.968 DEBUG:boto:Token: None 2024-03-04T17:50:32.968 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.968 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FE3uyppOACywBgRs4PT/FYqx2EQ= 2024-03-04T17:50:32.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FE3uyppOACywBgRs4PT/FYqx2EQ=', 'Content-Length': '0'} 2024-03-04T17:50:32.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af2b25a8b6c0dd12-0065e609e8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:32.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:32.973 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.973 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:32.973 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.973 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.973 DEBUG:boto:Method: GET 2024-03-04T17:50:32.973 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.974 DEBUG:boto:Data: 2024-03-04T17:50:32.974 DEBUG:boto:Headers: {} 2024-03-04T17:50:32.974 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:32.974 DEBUG:boto:Port: 8001 2024-03-04T17:50:32.974 DEBUG:boto:Params: {} 2024-03-04T17:50:32.974 DEBUG:boto:Token: None 2024-03-04T17:50:32.974 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:32 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:32.974 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QVOhNif7A9RRekrePeDJkhmm3Pk= 2024-03-04T17:50:32.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QVOhNif7A9RRekrePeDJkhmm3Pk=', 'Content-Length': '0'} 2024-03-04T17:50:32.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073bda7c401ad899e-0065e609e8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.024 DEBUG:boto:path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:33.024 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1 2024-03-04T17:50:33.025 DEBUG:boto:path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:33.025 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:33.025 DEBUG:boto:Method: GET 2024-03-04T17:50:33.025 DEBUG:boto:Path: /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:33.025 DEBUG:boto:Data: 2024-03-04T17:50:33.025 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.025 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.025 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.025 DEBUG:boto:Params: {} 2024-03-04T17:50:33.026 DEBUG:boto:Token: None 2024-03-04T17:50:33.026 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone1?versionId=9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L 2024-03-04T17:50:33.026 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KITUUe/nvgWcKHQSPiP3F+9ji5w= 2024-03-04T17:50:33.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KITUUe/nvgWcKHQSPiP3F+9ji5w=', 'Content-Length': '0'} 2024-03-04T17:50:33.029 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '9Nn63Wf7xVnzErCBAt.4UYmUg.L.38L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000867eeb30530be15f-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.076 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:33.076 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:33.076 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:33.076 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:33.076 DEBUG:boto:Method: GET 2024-03-04T17:50:33.077 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:33.077 DEBUG:boto:Data: 2024-03-04T17:50:33.077 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.077 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.077 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.077 DEBUG:boto:Params: {} 2024-03-04T17:50:33.077 DEBUG:boto:Token: None 2024-03-04T17:50:33.077 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/ 2024-03-04T17:50:33.077 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU= 2024-03-04T17:50:33.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU=', 'Content-Length': '0'} 2024-03-04T17:50:33.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000235010ec2dcfd9a-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.124 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:33.125 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:33.125 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:33.125 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:33.125 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:33.125 DEBUG:boto:Method: GET 2024-03-04T17:50:33.125 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone1 2024-03-04T17:50:33.125 DEBUG:boto:Data: 2024-03-04T17:50:33.125 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.125 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.125 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.125 DEBUG:boto:Params: {} 2024-03-04T17:50:33.126 DEBUG:boto:Token: None 2024-03-04T17:50:33.126 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/ 2024-03-04T17:50:33.126 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU= 2024-03-04T17:50:33.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU=', 'Content-Length': '0'} 2024-03-04T17:50:33.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aef8743e37501495-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.130 DEBUG:boto:b'ppfahn-20obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:33.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:33.131 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.131 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.131 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.131 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.131 DEBUG:boto:Method: GET 2024-03-04T17:50:33.131 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.131 DEBUG:boto:Data: 2024-03-04T17:50:33.131 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.132 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.132 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.132 DEBUG:boto:Params: {} 2024-03-04T17:50:33.132 DEBUG:boto:Token: None 2024-03-04T17:50:33.132 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.132 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HAjwNZeBGYzbanGChu/6dqsEW9s= 2024-03-04T17:50:33.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HAjwNZeBGYzbanGChu/6dqsEW9s=', 'Content-Length': '0'} 2024-03-04T17:50:33.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005285e8526098ba97-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.135 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.136 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.136 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.136 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.136 DEBUG:boto:Method: GET 2024-03-04T17:50:33.136 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.136 DEBUG:boto:Data: 2024-03-04T17:50:33.136 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.136 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.136 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.136 DEBUG:boto:Params: {} 2024-03-04T17:50:33.137 DEBUG:boto:Token: None 2024-03-04T17:50:33.137 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.137 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HAjwNZeBGYzbanGChu/6dqsEW9s= 2024-03-04T17:50:33.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HAjwNZeBGYzbanGChu/6dqsEW9s=', 'Content-Length': '0'} 2024-03-04T17:50:33.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000850e928beb3612d8-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.140 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.140 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.140 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.140 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.140 DEBUG:boto:Method: HEAD 2024-03-04T17:50:33.140 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.141 DEBUG:boto:Data: 2024-03-04T17:50:33.141 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.141 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.141 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.141 DEBUG:boto:Params: {} 2024-03-04T17:50:33.141 DEBUG:boto:Token: None 2024-03-04T17:50:33.141 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.141 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/HuL9OBb1kGa+/UJHKCSFmZUGaA= 2024-03-04T17:50:33.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/HuL9OBb1kGa+/UJHKCSFmZUGaA=', 'Content-Length': '0'} 2024-03-04T17:50:33.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4e88d526b8d2a2a-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.144 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.144 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.145 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.145 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.145 DEBUG:boto:Method: HEAD 2024-03-04T17:50:33.145 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.145 DEBUG:boto:Data: 2024-03-04T17:50:33.145 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.145 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.145 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.145 DEBUG:boto:Params: {} 2024-03-04T17:50:33.145 DEBUG:boto:Token: None 2024-03-04T17:50:33.146 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.146 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/HuL9OBb1kGa+/UJHKCSFmZUGaA= 2024-03-04T17:50:33.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/HuL9OBb1kGa+/UJHKCSFmZUGaA=', 'Content-Length': '0'} 2024-03-04T17:50:33.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2b2065320b50110-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:33.149 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.149 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.149 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.149 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.149 DEBUG:boto:Method: GET 2024-03-04T17:50:33.149 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.149 DEBUG:boto:Data: 2024-03-04T17:50:33.149 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.149 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.149 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.150 DEBUG:boto:Params: {} 2024-03-04T17:50:33.150 DEBUG:boto:Token: None 2024-03-04T17:50:33.150 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.150 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HAjwNZeBGYzbanGChu/6dqsEW9s= 2024-03-04T17:50:33.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HAjwNZeBGYzbanGChu/6dqsEW9s=', 'Content-Length': '0'} 2024-03-04T17:50:33.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c757278d3896e01-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.200 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.200 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.200 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.200 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.201 DEBUG:boto:Method: GET 2024-03-04T17:50:33.201 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.201 DEBUG:boto:Data: 2024-03-04T17:50:33.201 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.201 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.201 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.201 DEBUG:boto:Params: {} 2024-03-04T17:50:33.201 DEBUG:boto:Token: None 2024-03-04T17:50:33.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.202 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HAjwNZeBGYzbanGChu/6dqsEW9s= 2024-03-04T17:50:33.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HAjwNZeBGYzbanGChu/6dqsEW9s=', 'Content-Length': '0'} 2024-03-04T17:50:33.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002864d39bc56e6a2d-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.205 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:33.205 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:33.205 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:33.205 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:33.205 DEBUG:boto:Method: GET 2024-03-04T17:50:33.205 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:33.205 DEBUG:boto:Data: 2024-03-04T17:50:33.206 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.206 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.206 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.206 DEBUG:boto:Params: {} 2024-03-04T17:50:33.206 DEBUG:boto:Token: None 2024-03-04T17:50:33.206 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/ 2024-03-04T17:50:33.206 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU= 2024-03-04T17:50:33.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU=', 'Content-Length': '0'} 2024-03-04T17:50:33.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a0631b3113a4944-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.252 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:33.253 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:33.253 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:33.253 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:33.253 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:33.253 DEBUG:boto:Method: GET 2024-03-04T17:50:33.253 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:33.253 DEBUG:boto:Data: 2024-03-04T17:50:33.253 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.253 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.253 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.253 DEBUG:boto:Params: {} 2024-03-04T17:50:33.254 DEBUG:boto:Token: None 2024-03-04T17:50:33.254 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/ 2024-03-04T17:50:33.254 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU= 2024-03-04T17:50:33.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU=', 'Content-Length': '0'} 2024-03-04T17:50:33.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b87bc62d2a335c13-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.258 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:33.259 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.259 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.259 DEBUG:boto:Method: HEAD 2024-03-04T17:50:33.259 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.259 DEBUG:boto:Data: 2024-03-04T17:50:33.259 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.259 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.259 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.259 DEBUG:boto:Params: {} 2024-03-04T17:50:33.260 DEBUG:boto:Token: None 2024-03-04T17:50:33.260 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.260 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ESklC8zQQI0QRQ4ywfrnmzYXdxI= 2024-03-04T17:50:33.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ESklC8zQQI0QRQ4ywfrnmzYXdxI=', 'Content-Length': '0'} 2024-03-04T17:50:33.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051cc5ff0ba2d692a-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.264 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.264 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.264 DEBUG:boto:Method: HEAD 2024-03-04T17:50:33.264 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.265 DEBUG:boto:Data: 2024-03-04T17:50:33.265 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.265 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.265 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.265 DEBUG:boto:Params: {} 2024-03-04T17:50:33.265 DEBUG:boto:Token: None 2024-03-04T17:50:33.265 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.265 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ESklC8zQQI0QRQ4ywfrnmzYXdxI= 2024-03-04T17:50:33.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ESklC8zQQI0QRQ4ywfrnmzYXdxI=', 'Content-Length': '0'} 2024-03-04T17:50:33.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066c05a0321ab3f81-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:33.269 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.269 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.269 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.270 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.270 DEBUG:boto:Method: GET 2024-03-04T17:50:33.270 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.270 DEBUG:boto:Data: 2024-03-04T17:50:33.270 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.270 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.270 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.270 DEBUG:boto:Params: {} 2024-03-04T17:50:33.270 DEBUG:boto:Token: None 2024-03-04T17:50:33.270 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.270 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HAjwNZeBGYzbanGChu/6dqsEW9s= 2024-03-04T17:50:33.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HAjwNZeBGYzbanGChu/6dqsEW9s=', 'Content-Length': '0'} 2024-03-04T17:50:33.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac62f828799b3338-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.316 DEBUG:boto:path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.316 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2 2024-03-04T17:50:33.316 DEBUG:boto:path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.316 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.316 DEBUG:boto:Method: GET 2024-03-04T17:50:33.317 DEBUG:boto:Path: /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.317 DEBUG:boto:Data: 2024-03-04T17:50:33.317 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.317 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.317 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.317 DEBUG:boto:Params: {} 2024-03-04T17:50:33.317 DEBUG:boto:Token: None 2024-03-04T17:50:33.317 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone2?versionId=-danbM6QbPvBQ8R9uWBluKiMLcLfHcO 2024-03-04T17:50:33.317 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HAjwNZeBGYzbanGChu/6dqsEW9s= 2024-03-04T17:50:33.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HAjwNZeBGYzbanGChu/6dqsEW9s=', 'Content-Length': '0'} 2024-03-04T17:50:33.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '-danbM6QbPvBQ8R9uWBluKiMLcLfHcO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013a2bf5ff4f9c78a-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.321 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:33.321 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:33.321 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:33.321 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:33.321 DEBUG:boto:Method: GET 2024-03-04T17:50:33.321 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:33.321 DEBUG:boto:Data: 2024-03-04T17:50:33.321 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.321 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.321 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.322 DEBUG:boto:Params: {} 2024-03-04T17:50:33.322 DEBUG:boto:Token: None 2024-03-04T17:50:33.322 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/ 2024-03-04T17:50:33.322 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU= 2024-03-04T17:50:33.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU=', 'Content-Length': '0'} 2024-03-04T17:50:33.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002860cc69b495bfb5-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.326 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:33.327 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:33.327 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:33.327 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:33.327 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:33.327 DEBUG:boto:Method: GET 2024-03-04T17:50:33.327 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone2 2024-03-04T17:50:33.327 DEBUG:boto:Data: 2024-03-04T17:50:33.327 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.327 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.327 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.328 DEBUG:boto:Params: {} 2024-03-04T17:50:33.328 DEBUG:boto:Token: None 2024-03-04T17:50:33.328 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/ 2024-03-04T17:50:33.328 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU= 2024-03-04T17:50:33.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU=', 'Content-Length': '0'} 2024-03-04T17:50:33.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb931ecf814de412-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.376 DEBUG:boto:b'ppfahn-20obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:33.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:33.376 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.377 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.377 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.377 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.377 DEBUG:boto:Method: GET 2024-03-04T17:50:33.377 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.377 DEBUG:boto:Data: 2024-03-04T17:50:33.377 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.377 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.377 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.377 DEBUG:boto:Params: {} 2024-03-04T17:50:33.378 DEBUG:boto:Token: None 2024-03-04T17:50:33.378 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.378 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TqP2DPHQAs7Krvskbul+66Ecd3k= 2024-03-04T17:50:33.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TqP2DPHQAs7Krvskbul+66Ecd3k=', 'Content-Length': '0'} 2024-03-04T17:50:33.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078b201b07a539290-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.381 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.381 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.381 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.381 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.381 DEBUG:boto:Method: GET 2024-03-04T17:50:33.382 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.382 DEBUG:boto:Data: 2024-03-04T17:50:33.382 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.382 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.382 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.382 DEBUG:boto:Params: {} 2024-03-04T17:50:33.382 DEBUG:boto:Token: None 2024-03-04T17:50:33.382 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.382 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TqP2DPHQAs7Krvskbul+66Ecd3k= 2024-03-04T17:50:33.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TqP2DPHQAs7Krvskbul+66Ecd3k=', 'Content-Length': '0'} 2024-03-04T17:50:33.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb631aece170e380-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.432 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.432 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.432 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.433 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.433 DEBUG:boto:Method: HEAD 2024-03-04T17:50:33.433 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.433 DEBUG:boto:Data: 2024-03-04T17:50:33.433 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.433 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.433 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.433 DEBUG:boto:Params: {} 2024-03-04T17:50:33.433 DEBUG:boto:Token: None 2024-03-04T17:50:33.434 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.434 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KjZ+WhBv23/fwwpXW9z48YIMwY0= 2024-03-04T17:50:33.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KjZ+WhBv23/fwwpXW9z48YIMwY0=', 'Content-Length': '0'} 2024-03-04T17:50:33.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab2779d72cc4f9e5-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.437 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.437 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.437 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.437 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.437 DEBUG:boto:Method: HEAD 2024-03-04T17:50:33.437 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.437 DEBUG:boto:Data: 2024-03-04T17:50:33.437 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.438 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.438 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.438 DEBUG:boto:Params: {} 2024-03-04T17:50:33.438 DEBUG:boto:Token: None 2024-03-04T17:50:33.438 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.438 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KjZ+WhBv23/fwwpXW9z48YIMwY0= 2024-03-04T17:50:33.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KjZ+WhBv23/fwwpXW9z48YIMwY0=', 'Content-Length': '0'} 2024-03-04T17:50:33.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006315de1cd4fc779d-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:33.442 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.442 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.442 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.442 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.442 DEBUG:boto:Method: GET 2024-03-04T17:50:33.442 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.442 DEBUG:boto:Data: 2024-03-04T17:50:33.442 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.442 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.442 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.443 DEBUG:boto:Params: {} 2024-03-04T17:50:33.443 DEBUG:boto:Token: None 2024-03-04T17:50:33.443 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.443 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TqP2DPHQAs7Krvskbul+66Ecd3k= 2024-03-04T17:50:33.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TqP2DPHQAs7Krvskbul+66Ecd3k=', 'Content-Length': '0'} 2024-03-04T17:50:33.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c80928668278a0a-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.488 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.488 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.488 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.488 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.488 DEBUG:boto:Method: GET 2024-03-04T17:50:33.489 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.489 DEBUG:boto:Data: 2024-03-04T17:50:33.489 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.489 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.489 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.489 DEBUG:boto:Params: {} 2024-03-04T17:50:33.489 DEBUG:boto:Token: None 2024-03-04T17:50:33.489 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.489 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TqP2DPHQAs7Krvskbul+66Ecd3k= 2024-03-04T17:50:33.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TqP2DPHQAs7Krvskbul+66Ecd3k=', 'Content-Length': '0'} 2024-03-04T17:50:33.492 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c88300c6ff6b537-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.536 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:33.536 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:33.536 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:33.536 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:33.536 DEBUG:boto:Method: GET 2024-03-04T17:50:33.537 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:33.537 DEBUG:boto:Data: 2024-03-04T17:50:33.537 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.537 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.537 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.537 DEBUG:boto:Params: {} 2024-03-04T17:50:33.537 DEBUG:boto:Token: None 2024-03-04T17:50:33.537 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/ 2024-03-04T17:50:33.537 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU= 2024-03-04T17:50:33.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU=', 'Content-Length': '0'} 2024-03-04T17:50:33.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000611eaa26c3f4259-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.542 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:33.542 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:33.543 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:33.543 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:33.543 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:33.543 DEBUG:boto:Method: GET 2024-03-04T17:50:33.543 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:33.543 DEBUG:boto:Data: 2024-03-04T17:50:33.543 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.543 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.543 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.543 DEBUG:boto:Params: {} 2024-03-04T17:50:33.543 DEBUG:boto:Token: None 2024-03-04T17:50:33.544 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/ 2024-03-04T17:50:33.544 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU= 2024-03-04T17:50:33.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU=', 'Content-Length': '0'} 2024-03-04T17:50:33.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009baafda37f3a66fa-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.548 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:33.549 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.549 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.549 DEBUG:boto:Method: HEAD 2024-03-04T17:50:33.549 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.549 DEBUG:boto:Data: 2024-03-04T17:50:33.549 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.549 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.550 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.550 DEBUG:boto:Params: {} 2024-03-04T17:50:33.550 DEBUG:boto:Token: None 2024-03-04T17:50:33.550 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.550 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:o5uPZGqL7n3zoIqXbx1oTaTcceQ= 2024-03-04T17:50:33.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:o5uPZGqL7n3zoIqXbx1oTaTcceQ=', 'Content-Length': '0'} 2024-03-04T17:50:33.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4b274628d27d91a-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.554 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.555 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.555 DEBUG:boto:Method: HEAD 2024-03-04T17:50:33.555 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.555 DEBUG:boto:Data: 2024-03-04T17:50:33.555 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.555 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.555 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.555 DEBUG:boto:Params: {} 2024-03-04T17:50:33.555 DEBUG:boto:Token: None 2024-03-04T17:50:33.555 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.556 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:o5uPZGqL7n3zoIqXbx1oTaTcceQ= 2024-03-04T17:50:33.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:o5uPZGqL7n3zoIqXbx1oTaTcceQ=', 'Content-Length': '0'} 2024-03-04T17:50:33.559 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000430e1225155b7912-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:33.560 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.560 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.560 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.560 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.560 DEBUG:boto:Method: GET 2024-03-04T17:50:33.560 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.560 DEBUG:boto:Data: 2024-03-04T17:50:33.560 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.561 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.561 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.561 DEBUG:boto:Params: {} 2024-03-04T17:50:33.561 DEBUG:boto:Token: None 2024-03-04T17:50:33.561 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.561 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TqP2DPHQAs7Krvskbul+66Ecd3k= 2024-03-04T17:50:33.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TqP2DPHQAs7Krvskbul+66Ecd3k=', 'Content-Length': '0'} 2024-03-04T17:50:33.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000740e4e6dde46b515-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.564 DEBUG:boto:path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.565 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3 2024-03-04T17:50:33.565 DEBUG:boto:path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.565 DEBUG:boto:auth_path=/ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.565 DEBUG:boto:Method: GET 2024-03-04T17:50:33.565 DEBUG:boto:Path: /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.565 DEBUG:boto:Data: 2024-03-04T17:50:33.565 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.565 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.565 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.566 DEBUG:boto:Params: {} 2024-03-04T17:50:33.566 DEBUG:boto:Token: None 2024-03-04T17:50:33.566 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/obj-test-zone3?versionId=.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK 2024-03-04T17:50:33.566 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TqP2DPHQAs7Krvskbul+66Ecd3k= 2024-03-04T17:50:33.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TqP2DPHQAs7Krvskbul+66Ecd3k=', 'Content-Length': '0'} 2024-03-04T17:50:33.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', '.DJ9hLU-KeSyKdnhOoPa6kBqV55BraK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-20:d96636ac-054d-40e3-90f3-e3912da784b2.4349.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000356430cd33effdf7-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.612 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:33.612 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:33.612 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:33.612 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:33.613 DEBUG:boto:Method: GET 2024-03-04T17:50:33.613 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:33.613 DEBUG:boto:Data: 2024-03-04T17:50:33.613 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.613 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:33.613 DEBUG:boto:Port: 8001 2024-03-04T17:50:33.613 DEBUG:boto:Params: {} 2024-03-04T17:50:33.613 DEBUG:boto:Token: None 2024-03-04T17:50:33.613 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/ 2024-03-04T17:50:33.614 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU= 2024-03-04T17:50:33.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU=', 'Content-Length': '0'} 2024-03-04T17:50:33.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000841964ed2dfec68c-0065e609e9-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.619 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:33.619 DEBUG:boto:path=/ppfahn-20/ 2024-03-04T17:50:33.619 DEBUG:boto:auth_path=/ppfahn-20/ 2024-03-04T17:50:33.619 DEBUG:boto:path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:33.620 DEBUG:boto:auth_path=/ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:33.620 DEBUG:boto:Method: GET 2024-03-04T17:50:33.620 DEBUG:boto:Path: /ppfahn-20/?prefix=obj-test-zone3 2024-03-04T17:50:33.620 DEBUG:boto:Data: 2024-03-04T17:50:33.620 DEBUG:boto:Headers: {} 2024-03-04T17:50:33.620 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:33.620 DEBUG:boto:Port: 8000 2024-03-04T17:50:33.620 DEBUG:boto:Params: {} 2024-03-04T17:50:33.620 DEBUG:boto:Token: None 2024-03-04T17:50:33.620 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:33 GMT /ppfahn-20/ 2024-03-04T17:50:33.621 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU= 2024-03-04T17:50:33.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zZzgrVGU4zQFeLrTxYAo8humcxU=', 'Content-Length': '0'} 2024-03-04T17:50:33.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f4c5f96e0feaadf-0065e609e9-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:33.672 DEBUG:boto:b'ppfahn-20obj-test-zone31000falseobj-test-zone32024-03-04T17:46:46.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:33.673 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-20 zones={test-zone3, test-zone2} 2024-03-04T17:50:33.673 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-21 2024-03-04T17:50:33.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:33.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:33.673 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:33.725 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:33.725 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:33.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:33.787+0000 7fb138024980 1 bucket sync caught up with source: 2024-03-04T17:50:33.787 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000011.59.6, , 00000000003.35.10, , 00000000008.22.10, , , , , ] 2024-03-04T17:50:33.787 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000011.59.6, , 00000000003.35.10, , 00000000008.22.10, , , , , ] 2024-03-04T17:50:33.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:33.787+0000 7fb138024980 0 bucket checkpoint complete 2024-03-04T17:50:33.809 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-21 2024-03-04T17:50:33.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:33.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:33.809 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:33.859 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:33.860 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:33.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:33.925+0000 7f2fe1584980 1 bucket sync caught up with source: 2024-03-04T17:50:33.919 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000011.59.6, , 00000000003.35.10, , 00000000008.22.10, , , , , ] 2024-03-04T17:50:33.919 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000011.59.6, , 00000000003.35.10, , 00000000008.22.10, , , , , ] 2024-03-04T17:50:33.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:33.925+0000 7f2fe1584980 0 bucket checkpoint complete 2024-03-04T17:50:33.940 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-21 2024-03-04T17:50:33.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:33.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:33.941 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:33.981 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:33.981 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:34.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:34.045+0000 7f1fde77e980 1 bucket sync caught up with source: 2024-03-04T17:50:34.040 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.42.10, , 00000000011.52.6, , 00000000008.32.10, , , , , ] 2024-03-04T17:50:34.040 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.42.10, , 00000000011.52.6, , 00000000008.32.10, , , , , ] 2024-03-04T17:50:34.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:34.045+0000 7f1fde77e980 0 bucket checkpoint complete 2024-03-04T17:50:34.061 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-21 2024-03-04T17:50:34.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:34.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:34.062 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:34.146 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:34.147 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:34.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:34.213+0000 7f1367fbc980 1 bucket sync caught up with source: 2024-03-04T17:50:34.208 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.42.10, , 00000000011.52.6, , 00000000008.32.10, , , , , ] 2024-03-04T17:50:34.208 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.42.10, , 00000000011.52.6, , 00000000008.32.10, , , , , ] 2024-03-04T17:50:34.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:34.213+0000 7f1367fbc980 0 bucket checkpoint complete 2024-03-04T17:50:34.229 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-21 2024-03-04T17:50:34.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:34.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:34.229 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:34.311 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:34.311 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:34.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:34.373+0000 7fa774521980 1 bucket sync caught up with source: 2024-03-04T17:50:34.370 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, 00000000003.107.10, , 00000000003.68.10, , 00000000017.94.6, , , , , ] 2024-03-04T17:50:34.370 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, 00000000003.107.10, , 00000000003.68.10, , 00000000017.94.6, , , , , ] 2024-03-04T17:50:34.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:34.373+0000 7fa774521980 0 bucket checkpoint complete 2024-03-04T17:50:34.392 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-21 2024-03-04T17:50:34.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:34.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:34.392 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:34.444 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:34.445 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:34.511+0000 7fd28a926980 1 bucket sync caught up with source: 2024-03-04T17:50:34.509 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, 00000000003.107.10, , 00000000003.68.10, , 00000000017.94.6, , , , , ] 2024-03-04T17:50:34.510 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, 00000000003.107.10, , 00000000003.68.10, , 00000000017.94.6, , , , , ] 2024-03-04T17:50:34.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:34.511+0000 7fd28a926980 0 bucket checkpoint complete 2024-03-04T17:50:34.532 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-21 zones={test-zone2, test-zone1} 2024-03-04T17:50:34.532 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.533 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.533 DEBUG:boto:Method: HEAD 2024-03-04T17:50:34.533 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:50:34.533 DEBUG:boto:Data: 2024-03-04T17:50:34.533 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.533 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.533 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.533 DEBUG:boto:Params: {} 2024-03-04T17:50:34.533 DEBUG:boto:Token: None 2024-03-04T17:50:34.534 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/ 2024-03-04T17:50:34.534 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lR25EKn7DvlAMGIf+rZJaMMeDfI= 2024-03-04T17:50:34.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lR25EKn7DvlAMGIf+rZJaMMeDfI=', 'Content-Length': '0'} 2024-03-04T17:50:34.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9848412cfe65fea-0065e609ea-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.536 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.537 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.537 DEBUG:boto:Method: HEAD 2024-03-04T17:50:34.537 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:50:34.537 DEBUG:boto:Data: 2024-03-04T17:50:34.537 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.537 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.537 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.537 DEBUG:boto:Params: {} 2024-03-04T17:50:34.537 DEBUG:boto:Token: None 2024-03-04T17:50:34.537 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/ 2024-03-04T17:50:34.538 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lR25EKn7DvlAMGIf+rZJaMMeDfI= 2024-03-04T17:50:34.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lR25EKn7DvlAMGIf+rZJaMMeDfI=', 'Content-Length': '0'} 2024-03-04T17:50:34.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acb23de6213554d0-0065e609ea-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.542 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:50:34.542 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.542 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.542 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:34.542 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:34.543 DEBUG:boto:Method: GET 2024-03-04T17:50:34.543 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:34.543 DEBUG:boto:Data: 2024-03-04T17:50:34.543 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.543 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.543 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.543 DEBUG:boto:Params: {} 2024-03-04T17:50:34.543 DEBUG:boto:Token: None 2024-03-04T17:50:34.544 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/?versions 2024-03-04T17:50:34.544 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:a1G2+tdffb0+1vsAoEEJTCW4ba4= 2024-03-04T17:50:34.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:a1G2+tdffb0+1vsAoEEJTCW4ba4=', 'Content-Length': '0'} 2024-03-04T17:50:34.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c338690e4c16714-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.546 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:34.547 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:34.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:34.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:34.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:34.547 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:50:34.547 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.548 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.548 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:34.548 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:34.548 DEBUG:boto:Method: GET 2024-03-04T17:50:34.548 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:34.548 DEBUG:boto:Data: 2024-03-04T17:50:34.548 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.548 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.548 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.548 DEBUG:boto:Params: {} 2024-03-04T17:50:34.549 DEBUG:boto:Token: None 2024-03-04T17:50:34.549 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/?versions 2024-03-04T17:50:34.549 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:a1G2+tdffb0+1vsAoEEJTCW4ba4= 2024-03-04T17:50:34.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:a1G2+tdffb0+1vsAoEEJTCW4ba4=', 'Content-Length': '0'} 2024-03-04T17:50:34.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffd0aad839c2049e-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.554 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:34.554 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:34.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:34.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:34.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:34.555 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.555 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.555 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:34.555 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:34.555 DEBUG:boto:Method: GET 2024-03-04T17:50:34.556 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:34.556 DEBUG:boto:Data: 2024-03-04T17:50:34.556 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.556 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.556 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.556 DEBUG:boto:Params: {} 2024-03-04T17:50:34.556 DEBUG:boto:Token: None 2024-03-04T17:50:34.556 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/?versions 2024-03-04T17:50:34.556 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:a1G2+tdffb0+1vsAoEEJTCW4ba4= 2024-03-04T17:50:34.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:a1G2+tdffb0+1vsAoEEJTCW4ba4=', 'Content-Length': '0'} 2024-03-04T17:50:34.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d052a74e8397a6b-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.559 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:34.560 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.560 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.560 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:34.560 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:34.560 DEBUG:boto:Method: GET 2024-03-04T17:50:34.560 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:34.560 DEBUG:boto:Data: 2024-03-04T17:50:34.560 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.560 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.560 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.561 DEBUG:boto:Params: {} 2024-03-04T17:50:34.561 DEBUG:boto:Token: None 2024-03-04T17:50:34.561 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/?versions 2024-03-04T17:50:34.561 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:a1G2+tdffb0+1vsAoEEJTCW4ba4= 2024-03-04T17:50:34.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:a1G2+tdffb0+1vsAoEEJTCW4ba4=', 'Content-Length': '0'} 2024-03-04T17:50:34.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005277bdb49cb18edc-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.565 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:34.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:34.566 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:34.566 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:34.566 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.566 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.566 DEBUG:boto:Method: GET 2024-03-04T17:50:34.567 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.567 DEBUG:boto:Data: 2024-03-04T17:50:34.567 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.567 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.567 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.567 DEBUG:boto:Params: {} 2024-03-04T17:50:34.567 DEBUG:boto:Token: None 2024-03-04T17:50:34.567 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.567 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V/W3Y9PGm11okM3ngFQGSvDowHs= 2024-03-04T17:50:34.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V/W3Y9PGm11okM3ngFQGSvDowHs=', 'Content-Length': '0'} 2024-03-04T17:50:34.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000547e8a112d3bf92d-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.570 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:34.570 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:34.570 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.570 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.570 DEBUG:boto:Method: GET 2024-03-04T17:50:34.570 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.570 DEBUG:boto:Data: 2024-03-04T17:50:34.570 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.570 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.570 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.570 DEBUG:boto:Params: {} 2024-03-04T17:50:34.571 DEBUG:boto:Token: None 2024-03-04T17:50:34.571 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.571 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V/W3Y9PGm11okM3ngFQGSvDowHs= 2024-03-04T17:50:34.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V/W3Y9PGm11okM3ngFQGSvDowHs=', 'Content-Length': '0'} 2024-03-04T17:50:34.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000936edd91d97024f9-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.574 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:34.574 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:34.574 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.574 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.574 DEBUG:boto:Method: HEAD 2024-03-04T17:50:34.574 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.574 DEBUG:boto:Data: 2024-03-04T17:50:34.575 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.575 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.575 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.575 DEBUG:boto:Params: {} 2024-03-04T17:50:34.575 DEBUG:boto:Token: None 2024-03-04T17:50:34.575 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.575 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:setoID+uVLarZUvH/N/UVvzkwXc= 2024-03-04T17:50:34.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:setoID+uVLarZUvH/N/UVvzkwXc=', 'Content-Length': '0'} 2024-03-04T17:50:34.577 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000012fa940c3631863b-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.577 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:34.577 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:34.577 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.577 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.577 DEBUG:boto:Method: HEAD 2024-03-04T17:50:34.578 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.578 DEBUG:boto:Data: 2024-03-04T17:50:34.578 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.578 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.578 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.578 DEBUG:boto:Params: {} 2024-03-04T17:50:34.578 DEBUG:boto:Token: None 2024-03-04T17:50:34.578 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.578 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:setoID+uVLarZUvH/N/UVvzkwXc= 2024-03-04T17:50:34.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:setoID+uVLarZUvH/N/UVvzkwXc=', 'Content-Length': '0'} 2024-03-04T17:50:34.581 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b2b4b2471f78a204-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:34.582 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:34.582 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:34.582 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.582 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.582 DEBUG:boto:Method: GET 2024-03-04T17:50:34.582 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.582 DEBUG:boto:Data: 2024-03-04T17:50:34.582 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.582 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.582 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.582 DEBUG:boto:Params: {} 2024-03-04T17:50:34.583 DEBUG:boto:Token: None 2024-03-04T17:50:34.583 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.583 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V/W3Y9PGm11okM3ngFQGSvDowHs= 2024-03-04T17:50:34.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V/W3Y9PGm11okM3ngFQGSvDowHs=', 'Content-Length': '0'} 2024-03-04T17:50:34.585 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b70eba12f37c4b73-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.585 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:34.585 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:34.585 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.585 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.586 DEBUG:boto:Method: GET 2024-03-04T17:50:34.586 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.586 DEBUG:boto:Data: 2024-03-04T17:50:34.586 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.586 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.586 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.586 DEBUG:boto:Params: {} 2024-03-04T17:50:34.586 DEBUG:boto:Token: None 2024-03-04T17:50:34.586 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.587 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V/W3Y9PGm11okM3ngFQGSvDowHs= 2024-03-04T17:50:34.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V/W3Y9PGm11okM3ngFQGSvDowHs=', 'Content-Length': '0'} 2024-03-04T17:50:34.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d76e88a155e190a6-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.589 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.590 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.590 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:34.590 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:34.590 DEBUG:boto:Method: GET 2024-03-04T17:50:34.590 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:50:34.590 DEBUG:boto:Data: 2024-03-04T17:50:34.590 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.590 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.590 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.590 DEBUG:boto:Params: {} 2024-03-04T17:50:34.590 DEBUG:boto:Token: None 2024-03-04T17:50:34.591 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/ 2024-03-04T17:50:34.591 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y= 2024-03-04T17:50:34.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y=', 'Content-Length': '0'} 2024-03-04T17:50:34.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c312c9cd00b594d7-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.636 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:34.637 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.637 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.637 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:34.637 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:34.637 DEBUG:boto:Method: GET 2024-03-04T17:50:34.637 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:50:34.637 DEBUG:boto:Data: 2024-03-04T17:50:34.637 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.637 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.637 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.637 DEBUG:boto:Params: {} 2024-03-04T17:50:34.638 DEBUG:boto:Token: None 2024-03-04T17:50:34.638 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/ 2024-03-04T17:50:34.638 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y= 2024-03-04T17:50:34.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y=', 'Content-Length': '0'} 2024-03-04T17:50:34.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd49aba8ea26efcb-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.642 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:34.643 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:34.643 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:34.643 DEBUG:boto:Method: HEAD 2024-03-04T17:50:34.643 DEBUG:boto:Path: /ppfahn-21/dummy 2024-03-04T17:50:34.643 DEBUG:boto:Data: 2024-03-04T17:50:34.643 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.643 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.644 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.644 DEBUG:boto:Params: {} 2024-03-04T17:50:34.644 DEBUG:boto:Token: None 2024-03-04T17:50:34.644 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/dummy 2024-03-04T17:50:34.644 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3RETOiLtljxFvKTZdlH3B1rsYas= 2024-03-04T17:50:34.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3RETOiLtljxFvKTZdlH3B1rsYas=', 'Content-Length': '0'} 2024-03-04T17:50:34.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b6aff1efd7ac87f4-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.648 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:34.648 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:34.648 DEBUG:boto:Method: HEAD 2024-03-04T17:50:34.648 DEBUG:boto:Path: /ppfahn-21/dummy 2024-03-04T17:50:34.648 DEBUG:boto:Data: 2024-03-04T17:50:34.649 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.649 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.649 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.649 DEBUG:boto:Params: {} 2024-03-04T17:50:34.649 DEBUG:boto:Token: None 2024-03-04T17:50:34.649 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/dummy 2024-03-04T17:50:34.649 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3RETOiLtljxFvKTZdlH3B1rsYas= 2024-03-04T17:50:34.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3RETOiLtljxFvKTZdlH3B1rsYas=', 'Content-Length': '0'} 2024-03-04T17:50:34.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017ba17be15f170e8-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:34.654 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:34.654 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:34.654 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.654 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.654 DEBUG:boto:Method: GET 2024-03-04T17:50:34.654 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.654 DEBUG:boto:Data: 2024-03-04T17:50:34.654 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.654 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.654 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.654 DEBUG:boto:Params: {} 2024-03-04T17:50:34.655 DEBUG:boto:Token: None 2024-03-04T17:50:34.655 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.655 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V/W3Y9PGm11okM3ngFQGSvDowHs= 2024-03-04T17:50:34.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V/W3Y9PGm11okM3ngFQGSvDowHs=', 'Content-Length': '0'} 2024-03-04T17:50:34.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008523177f712d5d76-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.658 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:34.658 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:34.658 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.658 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.658 DEBUG:boto:Method: GET 2024-03-04T17:50:34.658 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.659 DEBUG:boto:Data: 2024-03-04T17:50:34.659 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.659 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.659 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.659 DEBUG:boto:Params: {} 2024-03-04T17:50:34.659 DEBUG:boto:Token: None 2024-03-04T17:50:34.659 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:34.659 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V/W3Y9PGm11okM3ngFQGSvDowHs= 2024-03-04T17:50:34.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V/W3Y9PGm11okM3ngFQGSvDowHs=', 'Content-Length': '0'} 2024-03-04T17:50:34.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f37775a581bf7e35-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.662 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.663 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.663 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:34.663 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:34.663 DEBUG:boto:Method: GET 2024-03-04T17:50:34.663 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:50:34.663 DEBUG:boto:Data: 2024-03-04T17:50:34.663 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.663 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.663 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.663 DEBUG:boto:Params: {} 2024-03-04T17:50:34.664 DEBUG:boto:Token: None 2024-03-04T17:50:34.664 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/ 2024-03-04T17:50:34.664 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y= 2024-03-04T17:50:34.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y=', 'Content-Length': '0'} 2024-03-04T17:50:34.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005178566f2105233f-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.668 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:34.668 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.669 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.669 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:34.669 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:34.669 DEBUG:boto:Method: GET 2024-03-04T17:50:34.669 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:50:34.669 DEBUG:boto:Data: 2024-03-04T17:50:34.669 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.669 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.669 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.669 DEBUG:boto:Params: {} 2024-03-04T17:50:34.669 DEBUG:boto:Token: None 2024-03-04T17:50:34.670 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/ 2024-03-04T17:50:34.670 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y= 2024-03-04T17:50:34.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y=', 'Content-Length': '0'} 2024-03-04T17:50:34.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005368a9f62b3b5f42-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.720 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:34.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:34.721 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.721 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.721 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.721 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.721 DEBUG:boto:Method: GET 2024-03-04T17:50:34.721 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.721 DEBUG:boto:Data: 2024-03-04T17:50:34.721 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.721 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.722 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.722 DEBUG:boto:Params: {} 2024-03-04T17:50:34.722 DEBUG:boto:Token: None 2024-03-04T17:50:34.722 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.722 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ChvnQpFu3Bcg/2RI5OFX2Md/VCw= 2024-03-04T17:50:34.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ChvnQpFu3Bcg/2RI5OFX2Md/VCw=', 'Content-Length': '0'} 2024-03-04T17:50:34.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003207b35867042d2-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.772 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.772 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.772 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.772 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.773 DEBUG:boto:Method: GET 2024-03-04T17:50:34.773 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.773 DEBUG:boto:Data: 2024-03-04T17:50:34.773 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.773 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.773 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.773 DEBUG:boto:Params: {} 2024-03-04T17:50:34.773 DEBUG:boto:Token: None 2024-03-04T17:50:34.773 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.774 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ChvnQpFu3Bcg/2RI5OFX2Md/VCw= 2024-03-04T17:50:34.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ChvnQpFu3Bcg/2RI5OFX2Md/VCw=', 'Content-Length': '0'} 2024-03-04T17:50:34.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001455301d78405e2-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.777 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.777 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.777 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.778 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.778 DEBUG:boto:Method: HEAD 2024-03-04T17:50:34.778 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.778 DEBUG:boto:Data: 2024-03-04T17:50:34.778 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.778 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.778 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.778 DEBUG:boto:Params: {} 2024-03-04T17:50:34.778 DEBUG:boto:Token: None 2024-03-04T17:50:34.778 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.779 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:a3jVTV9CnO9iQR/U0jhXPpN0ecc= 2024-03-04T17:50:34.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:a3jVTV9CnO9iQR/U0jhXPpN0ecc=', 'Content-Length': '0'} 2024-03-04T17:50:34.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059ff18cc0d6ef1e0-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.782 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.782 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.782 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.782 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.782 DEBUG:boto:Method: HEAD 2024-03-04T17:50:34.782 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.782 DEBUG:boto:Data: 2024-03-04T17:50:34.782 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.782 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.783 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.783 DEBUG:boto:Params: {} 2024-03-04T17:50:34.783 DEBUG:boto:Token: None 2024-03-04T17:50:34.783 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.783 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:a3jVTV9CnO9iQR/U0jhXPpN0ecc= 2024-03-04T17:50:34.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:a3jVTV9CnO9iQR/U0jhXPpN0ecc=', 'Content-Length': '0'} 2024-03-04T17:50:34.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000506ed033f4942421-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:34.786 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.786 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.787 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.787 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.787 DEBUG:boto:Method: GET 2024-03-04T17:50:34.787 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.787 DEBUG:boto:Data: 2024-03-04T17:50:34.787 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.787 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.787 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.787 DEBUG:boto:Params: {} 2024-03-04T17:50:34.787 DEBUG:boto:Token: None 2024-03-04T17:50:34.788 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.788 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ChvnQpFu3Bcg/2RI5OFX2Md/VCw= 2024-03-04T17:50:34.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ChvnQpFu3Bcg/2RI5OFX2Md/VCw=', 'Content-Length': '0'} 2024-03-04T17:50:34.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ad3179a06196026-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.836 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.836 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.836 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.836 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.837 DEBUG:boto:Method: GET 2024-03-04T17:50:34.837 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.837 DEBUG:boto:Data: 2024-03-04T17:50:34.837 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.837 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.837 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.837 DEBUG:boto:Params: {} 2024-03-04T17:50:34.837 DEBUG:boto:Token: None 2024-03-04T17:50:34.838 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.838 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ChvnQpFu3Bcg/2RI5OFX2Md/VCw= 2024-03-04T17:50:34.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ChvnQpFu3Bcg/2RI5OFX2Md/VCw=', 'Content-Length': '0'} 2024-03-04T17:50:34.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b8ed5cc0eb2957a-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.841 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.841 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.842 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:34.842 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:34.842 DEBUG:boto:Method: GET 2024-03-04T17:50:34.842 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:34.842 DEBUG:boto:Data: 2024-03-04T17:50:34.842 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.842 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.842 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.842 DEBUG:boto:Params: {} 2024-03-04T17:50:34.842 DEBUG:boto:Token: None 2024-03-04T17:50:34.843 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/ 2024-03-04T17:50:34.843 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y= 2024-03-04T17:50:34.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y=', 'Content-Length': '0'} 2024-03-04T17:50:34.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a220b0d197f1048c-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.847 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:34.848 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.848 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.848 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:34.848 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:34.848 DEBUG:boto:Method: GET 2024-03-04T17:50:34.848 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:34.848 DEBUG:boto:Data: 2024-03-04T17:50:34.848 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.848 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.848 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.849 DEBUG:boto:Params: {} 2024-03-04T17:50:34.849 DEBUG:boto:Token: None 2024-03-04T17:50:34.849 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/ 2024-03-04T17:50:34.849 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y= 2024-03-04T17:50:34.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y=', 'Content-Length': '0'} 2024-03-04T17:50:34.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001571e9057d2b83bc-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.854 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:34.854 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.854 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.855 DEBUG:boto:Method: HEAD 2024-03-04T17:50:34.855 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.855 DEBUG:boto:Data: 2024-03-04T17:50:34.855 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.855 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.855 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.855 DEBUG:boto:Params: {} 2024-03-04T17:50:34.855 DEBUG:boto:Token: None 2024-03-04T17:50:34.855 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.856 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WuaH4INiobSYEUMI8xytyDTv8fw= 2024-03-04T17:50:34.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WuaH4INiobSYEUMI8xytyDTv8fw=', 'Content-Length': '0'} 2024-03-04T17:50:34.859 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000226e38c85aeb686c-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.860 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.860 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.860 DEBUG:boto:Method: HEAD 2024-03-04T17:50:34.860 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.860 DEBUG:boto:Data: 2024-03-04T17:50:34.860 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.860 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.860 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.860 DEBUG:boto:Params: {} 2024-03-04T17:50:34.861 DEBUG:boto:Token: None 2024-03-04T17:50:34.861 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.861 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WuaH4INiobSYEUMI8xytyDTv8fw= 2024-03-04T17:50:34.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WuaH4INiobSYEUMI8xytyDTv8fw=', 'Content-Length': '0'} 2024-03-04T17:50:34.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001525c8987295f40-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:34.865 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.865 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.865 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.865 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.865 DEBUG:boto:Method: GET 2024-03-04T17:50:34.865 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.865 DEBUG:boto:Data: 2024-03-04T17:50:34.866 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.866 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.866 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.866 DEBUG:boto:Params: {} 2024-03-04T17:50:34.866 DEBUG:boto:Token: None 2024-03-04T17:50:34.866 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.866 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ChvnQpFu3Bcg/2RI5OFX2Md/VCw= 2024-03-04T17:50:34.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ChvnQpFu3Bcg/2RI5OFX2Md/VCw=', 'Content-Length': '0'} 2024-03-04T17:50:34.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053fe1701cd3a0954-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.912 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.912 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:34.913 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.913 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.913 DEBUG:boto:Method: GET 2024-03-04T17:50:34.913 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.913 DEBUG:boto:Data: 2024-03-04T17:50:34.913 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.914 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.914 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.914 DEBUG:boto:Params: {} 2024-03-04T17:50:34.914 DEBUG:boto:Token: None 2024-03-04T17:50:34.914 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:34.915 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ChvnQpFu3Bcg/2RI5OFX2Md/VCw= 2024-03-04T17:50:34.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ChvnQpFu3Bcg/2RI5OFX2Md/VCw=', 'Content-Length': '0'} 2024-03-04T17:50:34.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7eba2095e0acf27-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.919 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.919 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.919 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:34.919 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:34.919 DEBUG:boto:Method: GET 2024-03-04T17:50:34.919 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:34.919 DEBUG:boto:Data: 2024-03-04T17:50:34.919 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.919 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.919 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.919 DEBUG:boto:Params: {} 2024-03-04T17:50:34.920 DEBUG:boto:Token: None 2024-03-04T17:50:34.920 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/ 2024-03-04T17:50:34.920 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y= 2024-03-04T17:50:34.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y=', 'Content-Length': '0'} 2024-03-04T17:50:34.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003de43b582e5833dd-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.968 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:34.969 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:34.969 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:34.969 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:34.969 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:34.969 DEBUG:boto:Method: GET 2024-03-04T17:50:34.969 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:34.969 DEBUG:boto:Data: 2024-03-04T17:50:34.969 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.969 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.970 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.970 DEBUG:boto:Params: {} 2024-03-04T17:50:34.970 DEBUG:boto:Token: None 2024-03-04T17:50:34.970 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/ 2024-03-04T17:50:34.970 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y= 2024-03-04T17:50:34.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wW63QyoGFkZYx6OXdKtZ4JSLO5Y=', 'Content-Length': '0'} 2024-03-04T17:50:34.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000716f8d1edf77648f-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.975 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:34.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:34.976 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:34.976 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:34.976 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:34.976 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:34.976 DEBUG:boto:Method: GET 2024-03-04T17:50:34.976 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:34.976 DEBUG:boto:Data: 2024-03-04T17:50:34.977 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.977 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:34.977 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.977 DEBUG:boto:Params: {} 2024-03-04T17:50:34.977 DEBUG:boto:Token: None 2024-03-04T17:50:34.977 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:34.977 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ofIgY8SgMu9JjuKlOQMqtF60iCA= 2024-03-04T17:50:34.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ofIgY8SgMu9JjuKlOQMqtF60iCA=', 'Content-Length': '0'} 2024-03-04T17:50:34.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a20282c01db83c2-0065e609ea-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:34.981 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:34.981 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:34.981 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:34.981 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:34.982 DEBUG:boto:Method: GET 2024-03-04T17:50:34.982 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:34.982 DEBUG:boto:Data: 2024-03-04T17:50:34.982 DEBUG:boto:Headers: {} 2024-03-04T17:50:34.982 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:34.982 DEBUG:boto:Port: 8000 2024-03-04T17:50:34.982 DEBUG:boto:Params: {} 2024-03-04T17:50:34.982 DEBUG:boto:Token: None 2024-03-04T17:50:34.982 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:34 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:34.983 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ofIgY8SgMu9JjuKlOQMqtF60iCA= 2024-03-04T17:50:34.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ofIgY8SgMu9JjuKlOQMqtF60iCA=', 'Content-Length': '0'} 2024-03-04T17:50:34.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002637956bfd881fb7-0065e609ea-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.028 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.028 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.028 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.029 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.029 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.029 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.029 DEBUG:boto:Data: 2024-03-04T17:50:35.029 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.029 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:35.030 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.030 DEBUG:boto:Params: {} 2024-03-04T17:50:35.030 DEBUG:boto:Token: None 2024-03-04T17:50:35.030 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.030 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Fv2XGolJXWSpBhQtQN80KP6Hl20= 2024-03-04T17:50:35.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Fv2XGolJXWSpBhQtQN80KP6Hl20=', 'Content-Length': '0'} 2024-03-04T17:50:35.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d29db08b92fffb3-0065e609eb-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.034 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.034 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.034 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.034 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.034 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.035 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.035 DEBUG:boto:Data: 2024-03-04T17:50:35.035 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.035 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.035 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.035 DEBUG:boto:Params: {} 2024-03-04T17:50:35.035 DEBUG:boto:Token: None 2024-03-04T17:50:35.035 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.035 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Fv2XGolJXWSpBhQtQN80KP6Hl20= 2024-03-04T17:50:35.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Fv2XGolJXWSpBhQtQN80KP6Hl20=', 'Content-Length': '0'} 2024-03-04T17:50:35.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076b4e31a6081f72a-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:35.039 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.039 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.039 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.039 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.039 DEBUG:boto:Method: GET 2024-03-04T17:50:35.039 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.040 DEBUG:boto:Data: 2024-03-04T17:50:35.040 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.040 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:35.040 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.040 DEBUG:boto:Params: {} 2024-03-04T17:50:35.040 DEBUG:boto:Token: None 2024-03-04T17:50:35.040 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.040 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YW/tb3EFpqY47flUXMrUFPLW0y4= 2024-03-04T17:50:35.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YW/tb3EFpqY47flUXMrUFPLW0y4=', 'Content-Length': '0'} 2024-03-04T17:50:35.043 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9e529292051406e-0065e609eb-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.043 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.043 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.044 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.044 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.044 DEBUG:boto:Method: GET 2024-03-04T17:50:35.044 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.044 DEBUG:boto:Data: 2024-03-04T17:50:35.045 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.045 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.045 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.045 DEBUG:boto:Params: {} 2024-03-04T17:50:35.045 DEBUG:boto:Token: None 2024-03-04T17:50:35.045 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.045 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YW/tb3EFpqY47flUXMrUFPLW0y4= 2024-03-04T17:50:35.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YW/tb3EFpqY47flUXMrUFPLW0y4=', 'Content-Length': '0'} 2024-03-04T17:50:35.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037f59f6e5d965e23-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.096 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.096 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.096 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:35.096 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:35.096 DEBUG:boto:Method: GET 2024-03-04T17:50:35.097 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:35.097 DEBUG:boto:Data: 2024-03-04T17:50:35.097 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.097 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:35.097 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.097 DEBUG:boto:Params: {} 2024-03-04T17:50:35.097 DEBUG:boto:Token: None 2024-03-04T17:50:35.097 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.098 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI= 2024-03-04T17:50:35.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI=', 'Content-Length': '0'} 2024-03-04T17:50:35.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047cc2b2b3b1ccbab-0065e609eb-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.103 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.103 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.103 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.103 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:35.104 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:35.104 DEBUG:boto:Method: GET 2024-03-04T17:50:35.104 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:35.104 DEBUG:boto:Data: 2024-03-04T17:50:35.104 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.104 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.104 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.104 DEBUG:boto:Params: {} 2024-03-04T17:50:35.105 DEBUG:boto:Token: None 2024-03-04T17:50:35.105 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.105 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI= 2024-03-04T17:50:35.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI=', 'Content-Length': '0'} 2024-03-04T17:50:35.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087abff18bb439d49-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.156 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.157 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.157 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.158 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.158 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.158 DEBUG:boto:Data: 2024-03-04T17:50:35.158 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.158 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:35.158 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.158 DEBUG:boto:Params: {} 2024-03-04T17:50:35.158 DEBUG:boto:Token: None 2024-03-04T17:50:35.158 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.159 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nkrIJgNYWCevYGldG6rPnbL0F90= 2024-03-04T17:50:35.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nkrIJgNYWCevYGldG6rPnbL0F90=', 'Content-Length': '0'} 2024-03-04T17:50:35.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e54742943daeb818-0065e609eb-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.163 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.163 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.163 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.163 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.163 DEBUG:boto:Data: 2024-03-04T17:50:35.163 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.163 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.163 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.163 DEBUG:boto:Params: {} 2024-03-04T17:50:35.164 DEBUG:boto:Token: None 2024-03-04T17:50:35.164 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.164 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nkrIJgNYWCevYGldG6rPnbL0F90= 2024-03-04T17:50:35.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nkrIJgNYWCevYGldG6rPnbL0F90=', 'Content-Length': '0'} 2024-03-04T17:50:35.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054bb7c69eedd9226-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:35.169 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.169 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.169 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.169 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.169 DEBUG:boto:Method: GET 2024-03-04T17:50:35.169 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.169 DEBUG:boto:Data: 2024-03-04T17:50:35.170 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.170 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:35.170 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.170 DEBUG:boto:Params: {} 2024-03-04T17:50:35.170 DEBUG:boto:Token: None 2024-03-04T17:50:35.170 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.170 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YW/tb3EFpqY47flUXMrUFPLW0y4= 2024-03-04T17:50:35.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YW/tb3EFpqY47flUXMrUFPLW0y4=', 'Content-Length': '0'} 2024-03-04T17:50:35.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc83828e49d5773f-0065e609eb-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.219 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.219 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:35.219 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.219 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.219 DEBUG:boto:Method: GET 2024-03-04T17:50:35.219 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.219 DEBUG:boto:Data: 2024-03-04T17:50:35.220 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.220 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.220 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.220 DEBUG:boto:Params: {} 2024-03-04T17:50:35.220 DEBUG:boto:Token: None 2024-03-04T17:50:35.220 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:35.220 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YW/tb3EFpqY47flUXMrUFPLW0y4= 2024-03-04T17:50:35.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YW/tb3EFpqY47flUXMrUFPLW0y4=', 'Content-Length': '0'} 2024-03-04T17:50:35.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000451d70320aed51f1-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.264 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.264 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.264 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:35.264 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:35.264 DEBUG:boto:Method: GET 2024-03-04T17:50:35.265 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:35.265 DEBUG:boto:Data: 2024-03-04T17:50:35.265 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.265 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:35.265 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.265 DEBUG:boto:Params: {} 2024-03-04T17:50:35.265 DEBUG:boto:Token: None 2024-03-04T17:50:35.265 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.266 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI= 2024-03-04T17:50:35.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI=', 'Content-Length': '0'} 2024-03-04T17:50:35.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000936e560693ec9425-0065e609eb-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.270 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.271 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.271 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.271 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:35.271 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:35.271 DEBUG:boto:Method: GET 2024-03-04T17:50:35.271 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:35.271 DEBUG:boto:Data: 2024-03-04T17:50:35.271 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.271 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.271 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.272 DEBUG:boto:Params: {} 2024-03-04T17:50:35.272 DEBUG:boto:Token: None 2024-03-04T17:50:35.272 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.272 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI= 2024-03-04T17:50:35.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI=', 'Content-Length': '0'} 2024-03-04T17:50:35.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4a546724ad5bb15-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.320 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:35.321 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.321 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.321 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.321 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.321 DEBUG:boto:Method: GET 2024-03-04T17:50:35.322 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.322 DEBUG:boto:Data: 2024-03-04T17:50:35.322 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.322 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:35.322 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.322 DEBUG:boto:Params: {} 2024-03-04T17:50:35.322 DEBUG:boto:Token: None 2024-03-04T17:50:35.323 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.323 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W321PaulM7D666jmTrqJLlLdvf8= 2024-03-04T17:50:35.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W321PaulM7D666jmTrqJLlLdvf8=', 'Content-Length': '0'} 2024-03-04T17:50:35.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a36371d71e02a1ee-0065e609eb-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.372 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.372 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.372 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.373 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.373 DEBUG:boto:Method: GET 2024-03-04T17:50:35.373 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.373 DEBUG:boto:Data: 2024-03-04T17:50:35.373 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.373 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.373 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.373 DEBUG:boto:Params: {} 2024-03-04T17:50:35.373 DEBUG:boto:Token: None 2024-03-04T17:50:35.373 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.374 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W321PaulM7D666jmTrqJLlLdvf8= 2024-03-04T17:50:35.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W321PaulM7D666jmTrqJLlLdvf8=', 'Content-Length': '0'} 2024-03-04T17:50:35.377 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052062f0fdfe0255b-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.420 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.423 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.423 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.423 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.423 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.424 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.424 DEBUG:boto:Data: 2024-03-04T17:50:35.424 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.424 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:35.424 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.424 DEBUG:boto:Params: {} 2024-03-04T17:50:35.424 DEBUG:boto:Token: None 2024-03-04T17:50:35.424 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.424 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7p6vCD4plcW1MZSijiiBaWhBA2s= 2024-03-04T17:50:35.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7p6vCD4plcW1MZSijiiBaWhBA2s=', 'Content-Length': '0'} 2024-03-04T17:50:35.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa6c4a31ad4782ce-0065e609eb-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.428 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.428 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.428 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.428 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.428 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.428 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.428 DEBUG:boto:Data: 2024-03-04T17:50:35.429 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.429 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.429 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.429 DEBUG:boto:Params: {} 2024-03-04T17:50:35.429 DEBUG:boto:Token: None 2024-03-04T17:50:35.429 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.429 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7p6vCD4plcW1MZSijiiBaWhBA2s= 2024-03-04T17:50:35.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7p6vCD4plcW1MZSijiiBaWhBA2s=', 'Content-Length': '0'} 2024-03-04T17:50:35.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d21ad24b7560d453-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:35.433 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.433 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.433 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.433 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.433 DEBUG:boto:Method: GET 2024-03-04T17:50:35.433 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.433 DEBUG:boto:Data: 2024-03-04T17:50:35.433 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.433 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:35.434 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.434 DEBUG:boto:Params: {} 2024-03-04T17:50:35.434 DEBUG:boto:Token: None 2024-03-04T17:50:35.434 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.434 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W321PaulM7D666jmTrqJLlLdvf8= 2024-03-04T17:50:35.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W321PaulM7D666jmTrqJLlLdvf8=', 'Content-Length': '0'} 2024-03-04T17:50:35.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090fc72e997c91441-0065e609eb-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.480 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.480 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.480 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.480 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.480 DEBUG:boto:Method: GET 2024-03-04T17:50:35.481 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.481 DEBUG:boto:Data: 2024-03-04T17:50:35.481 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.481 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.481 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.481 DEBUG:boto:Params: {} 2024-03-04T17:50:35.481 DEBUG:boto:Token: None 2024-03-04T17:50:35.481 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.481 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W321PaulM7D666jmTrqJLlLdvf8= 2024-03-04T17:50:35.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W321PaulM7D666jmTrqJLlLdvf8=', 'Content-Length': '0'} 2024-03-04T17:50:35.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b128182cba0dce1-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.532 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.532 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.533 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:35.533 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:35.533 DEBUG:boto:Method: GET 2024-03-04T17:50:35.533 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:35.533 DEBUG:boto:Data: 2024-03-04T17:50:35.533 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.533 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:35.533 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.533 DEBUG:boto:Params: {} 2024-03-04T17:50:35.533 DEBUG:boto:Token: None 2024-03-04T17:50:35.534 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.534 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI= 2024-03-04T17:50:35.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI=', 'Content-Length': '0'} 2024-03-04T17:50:35.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cdf195842765ed1-0065e609eb-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.538 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.539 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.539 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.539 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:35.539 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:35.539 DEBUG:boto:Method: GET 2024-03-04T17:50:35.539 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:35.539 DEBUG:boto:Data: 2024-03-04T17:50:35.539 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.539 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.539 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.540 DEBUG:boto:Params: {} 2024-03-04T17:50:35.540 DEBUG:boto:Token: None 2024-03-04T17:50:35.540 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.540 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI= 2024-03-04T17:50:35.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI=', 'Content-Length': '0'} 2024-03-04T17:50:35.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e66544baeb754d84-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.588 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.588 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.589 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.589 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.589 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.589 DEBUG:boto:Data: 2024-03-04T17:50:35.589 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.589 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:35.589 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.589 DEBUG:boto:Params: {} 2024-03-04T17:50:35.589 DEBUG:boto:Token: None 2024-03-04T17:50:35.589 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.590 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yYVpTgctRDCHnv9BkdNmMuesoz0= 2024-03-04T17:50:35.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yYVpTgctRDCHnv9BkdNmMuesoz0=', 'Content-Length': '0'} 2024-03-04T17:50:35.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d4a7c0314ff09cd-0065e609eb-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.593 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.594 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.594 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.594 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.594 DEBUG:boto:Data: 2024-03-04T17:50:35.594 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.594 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.594 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.594 DEBUG:boto:Params: {} 2024-03-04T17:50:35.594 DEBUG:boto:Token: None 2024-03-04T17:50:35.594 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.595 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yYVpTgctRDCHnv9BkdNmMuesoz0= 2024-03-04T17:50:35.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yYVpTgctRDCHnv9BkdNmMuesoz0=', 'Content-Length': '0'} 2024-03-04T17:50:35.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ce7cf7afa2f5072-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:35.601 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.601 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.601 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.601 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.601 DEBUG:boto:Method: GET 2024-03-04T17:50:35.602 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.602 DEBUG:boto:Data: 2024-03-04T17:50:35.602 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.602 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:35.602 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.602 DEBUG:boto:Params: {} 2024-03-04T17:50:35.602 DEBUG:boto:Token: None 2024-03-04T17:50:35.602 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.602 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W321PaulM7D666jmTrqJLlLdvf8= 2024-03-04T17:50:35.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W321PaulM7D666jmTrqJLlLdvf8=', 'Content-Length': '0'} 2024-03-04T17:50:35.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045b00062b513d20c-0065e609eb-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.648 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.648 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:35.648 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.648 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.648 DEBUG:boto:Method: GET 2024-03-04T17:50:35.649 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.649 DEBUG:boto:Data: 2024-03-04T17:50:35.649 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.649 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.649 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.649 DEBUG:boto:Params: {} 2024-03-04T17:50:35.649 DEBUG:boto:Token: None 2024-03-04T17:50:35.649 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:35.649 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W321PaulM7D666jmTrqJLlLdvf8= 2024-03-04T17:50:35.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W321PaulM7D666jmTrqJLlLdvf8=', 'Content-Length': '0'} 2024-03-04T17:50:35.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c361160328573ec3-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.700 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.700 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.700 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:35.700 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:35.700 DEBUG:boto:Method: GET 2024-03-04T17:50:35.701 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:35.701 DEBUG:boto:Data: 2024-03-04T17:50:35.701 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.701 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:35.701 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.701 DEBUG:boto:Params: {} 2024-03-04T17:50:35.701 DEBUG:boto:Token: None 2024-03-04T17:50:35.701 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.701 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI= 2024-03-04T17:50:35.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI=', 'Content-Length': '0'} 2024-03-04T17:50:35.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e32b23a0bdc8992-0065e609eb-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.706 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.706 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.706 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.706 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:35.706 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:35.706 DEBUG:boto:Method: GET 2024-03-04T17:50:35.706 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:35.706 DEBUG:boto:Data: 2024-03-04T17:50:35.706 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.707 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.707 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.707 DEBUG:boto:Params: {} 2024-03-04T17:50:35.707 DEBUG:boto:Token: None 2024-03-04T17:50:35.707 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.707 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI= 2024-03-04T17:50:35.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI=', 'Content-Length': '0'} 2024-03-04T17:50:35.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f84f698de9fbfde2-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.756 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.757 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-21 zones={test-zone2, test-zone1} 2024-03-04T17:50:35.757 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-21 zones={test-zone3, test-zone1} 2024-03-04T17:50:35.757 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.757 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.757 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.757 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:50:35.757 DEBUG:boto:Data: 2024-03-04T17:50:35.757 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.757 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:35.758 DEBUG:boto:Port: 8001 2024-03-04T17:50:35.758 DEBUG:boto:Params: {} 2024-03-04T17:50:35.758 DEBUG:boto:Token: None 2024-03-04T17:50:35.758 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.758 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:coF67iMEF5ZzG/bI+4sAj0+BWRc= 2024-03-04T17:50:35.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:coF67iMEF5ZzG/bI+4sAj0+BWRc=', 'Content-Length': '0'} 2024-03-04T17:50:35.762 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d41e1a3e3824fba2-0065e609eb-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.762 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.762 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.763 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.763 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:50:35.763 DEBUG:boto:Data: 2024-03-04T17:50:35.763 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.763 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.763 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.763 DEBUG:boto:Params: {} 2024-03-04T17:50:35.763 DEBUG:boto:Token: None 2024-03-04T17:50:35.763 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.763 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:coF67iMEF5ZzG/bI+4sAj0+BWRc= 2024-03-04T17:50:35.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:coF67iMEF5ZzG/bI+4sAj0+BWRc=', 'Content-Length': '0'} 2024-03-04T17:50:35.767 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db2e4a678b77e9ba-0065e609eb-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.768 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:50:35.768 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.768 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.768 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:35.768 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:35.768 DEBUG:boto:Method: GET 2024-03-04T17:50:35.768 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:35.768 DEBUG:boto:Data: 2024-03-04T17:50:35.768 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.769 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:35.769 DEBUG:boto:Port: 8001 2024-03-04T17:50:35.769 DEBUG:boto:Params: {} 2024-03-04T17:50:35.769 DEBUG:boto:Token: None 2024-03-04T17:50:35.769 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/?versions 2024-03-04T17:50:35.769 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yfPoG0oNh4ALkuQEX3VxHPF31So= 2024-03-04T17:50:35.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yfPoG0oNh4ALkuQEX3VxHPF31So=', 'Content-Length': '0'} 2024-03-04T17:50:35.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ce9b528734483b3-0065e609eb-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.774 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.774 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:35.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:35.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:35.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:35.775 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:50:35.775 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.775 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.775 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:35.775 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:35.775 DEBUG:boto:Method: GET 2024-03-04T17:50:35.775 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:35.776 DEBUG:boto:Data: 2024-03-04T17:50:35.776 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.776 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.776 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.776 DEBUG:boto:Params: {} 2024-03-04T17:50:35.776 DEBUG:boto:Token: None 2024-03-04T17:50:35.776 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/?versions 2024-03-04T17:50:35.776 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yfPoG0oNh4ALkuQEX3VxHPF31So= 2024-03-04T17:50:35.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yfPoG0oNh4ALkuQEX3VxHPF31So=', 'Content-Length': '0'} 2024-03-04T17:50:35.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f497dfed6b81102d-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.781 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.782 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:35.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:35.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:35.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:35.782 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.782 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.783 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:35.783 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:35.783 DEBUG:boto:Method: GET 2024-03-04T17:50:35.783 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:35.783 DEBUG:boto:Data: 2024-03-04T17:50:35.783 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.783 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:35.783 DEBUG:boto:Port: 8001 2024-03-04T17:50:35.783 DEBUG:boto:Params: {} 2024-03-04T17:50:35.783 DEBUG:boto:Token: None 2024-03-04T17:50:35.784 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/?versions 2024-03-04T17:50:35.784 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yfPoG0oNh4ALkuQEX3VxHPF31So= 2024-03-04T17:50:35.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yfPoG0oNh4ALkuQEX3VxHPF31So=', 'Content-Length': '0'} 2024-03-04T17:50:35.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001029937d727e2d01-0065e609eb-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.788 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.788 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.789 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.789 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:35.789 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:35.789 DEBUG:boto:Method: GET 2024-03-04T17:50:35.789 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:35.789 DEBUG:boto:Data: 2024-03-04T17:50:35.789 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.789 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.789 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.790 DEBUG:boto:Params: {} 2024-03-04T17:50:35.790 DEBUG:boto:Token: None 2024-03-04T17:50:35.790 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/?versions 2024-03-04T17:50:35.790 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yfPoG0oNh4ALkuQEX3VxHPF31So= 2024-03-04T17:50:35.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yfPoG0oNh4ALkuQEX3VxHPF31So=', 'Content-Length': '0'} 2024-03-04T17:50:35.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005166e66eafaa8188-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.795 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:35.796 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:35.796 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:35.796 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.796 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.796 DEBUG:boto:Method: GET 2024-03-04T17:50:35.796 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.796 DEBUG:boto:Data: 2024-03-04T17:50:35.797 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.797 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:35.797 DEBUG:boto:Port: 8001 2024-03-04T17:50:35.797 DEBUG:boto:Params: {} 2024-03-04T17:50:35.797 DEBUG:boto:Token: None 2024-03-04T17:50:35.797 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.797 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YtAH190a7m23eZWwo4Tq0CUs+UE= 2024-03-04T17:50:35.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YtAH190a7m23eZWwo4Tq0CUs+UE=', 'Content-Length': '0'} 2024-03-04T17:50:35.800 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000428a74401583a559-0065e609eb-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.800 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:35.800 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:35.801 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.801 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.801 DEBUG:boto:Method: GET 2024-03-04T17:50:35.801 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.801 DEBUG:boto:Data: 2024-03-04T17:50:35.801 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.801 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.801 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.801 DEBUG:boto:Params: {} 2024-03-04T17:50:35.801 DEBUG:boto:Token: None 2024-03-04T17:50:35.802 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.802 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YtAH190a7m23eZWwo4Tq0CUs+UE= 2024-03-04T17:50:35.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YtAH190a7m23eZWwo4Tq0CUs+UE=', 'Content-Length': '0'} 2024-03-04T17:50:35.804 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f964b824215bba00-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.805 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:35.805 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:35.805 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.805 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.805 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.805 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.805 DEBUG:boto:Data: 2024-03-04T17:50:35.805 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.805 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:35.805 DEBUG:boto:Port: 8001 2024-03-04T17:50:35.806 DEBUG:boto:Params: {} 2024-03-04T17:50:35.806 DEBUG:boto:Token: None 2024-03-04T17:50:35.806 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.806 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PuZQOMPR/at9pXMVG+PObPzTRvU= 2024-03-04T17:50:35.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PuZQOMPR/at9pXMVG+PObPzTRvU=', 'Content-Length': '0'} 2024-03-04T17:50:35.808 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef69ec056868d1cd-0065e609eb-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.809 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:35.809 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:35.809 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.809 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.809 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.809 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.809 DEBUG:boto:Data: 2024-03-04T17:50:35.809 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.810 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.810 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.810 DEBUG:boto:Params: {} 2024-03-04T17:50:35.810 DEBUG:boto:Token: None 2024-03-04T17:50:35.810 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.810 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PuZQOMPR/at9pXMVG+PObPzTRvU= 2024-03-04T17:50:35.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PuZQOMPR/at9pXMVG+PObPzTRvU=', 'Content-Length': '0'} 2024-03-04T17:50:35.813 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e1ef7768d956b797-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:35.813 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:35.813 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:35.813 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.813 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.813 DEBUG:boto:Method: GET 2024-03-04T17:50:35.814 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.814 DEBUG:boto:Data: 2024-03-04T17:50:35.814 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.814 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:35.814 DEBUG:boto:Port: 8001 2024-03-04T17:50:35.814 DEBUG:boto:Params: {} 2024-03-04T17:50:35.814 DEBUG:boto:Token: None 2024-03-04T17:50:35.814 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.814 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YtAH190a7m23eZWwo4Tq0CUs+UE= 2024-03-04T17:50:35.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YtAH190a7m23eZWwo4Tq0CUs+UE=', 'Content-Length': '0'} 2024-03-04T17:50:35.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d335d6fd9ab0aa1-0065e609eb-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.817 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:35.818 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:35.818 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.818 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.818 DEBUG:boto:Method: GET 2024-03-04T17:50:35.818 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.818 DEBUG:boto:Data: 2024-03-04T17:50:35.818 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.818 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.818 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.818 DEBUG:boto:Params: {} 2024-03-04T17:50:35.819 DEBUG:boto:Token: None 2024-03-04T17:50:35.819 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.819 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YtAH190a7m23eZWwo4Tq0CUs+UE= 2024-03-04T17:50:35.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YtAH190a7m23eZWwo4Tq0CUs+UE=', 'Content-Length': '0'} 2024-03-04T17:50:35.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e0beeb9129bf0394-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.822 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.822 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.822 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:35.822 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:35.822 DEBUG:boto:Method: GET 2024-03-04T17:50:35.822 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:50:35.822 DEBUG:boto:Data: 2024-03-04T17:50:35.822 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.822 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:35.823 DEBUG:boto:Port: 8001 2024-03-04T17:50:35.823 DEBUG:boto:Params: {} 2024-03-04T17:50:35.823 DEBUG:boto:Token: None 2024-03-04T17:50:35.823 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.823 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI= 2024-03-04T17:50:35.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI=', 'Content-Length': '0'} 2024-03-04T17:50:35.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3810c1ecc84d7cf-0065e609eb-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.868 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.868 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.868 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.869 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:35.869 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:35.869 DEBUG:boto:Method: GET 2024-03-04T17:50:35.869 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:50:35.869 DEBUG:boto:Data: 2024-03-04T17:50:35.869 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.869 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.869 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.869 DEBUG:boto:Params: {} 2024-03-04T17:50:35.870 DEBUG:boto:Token: None 2024-03-04T17:50:35.870 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.870 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI= 2024-03-04T17:50:35.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI=', 'Content-Length': '0'} 2024-03-04T17:50:35.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c9df8319badc0a5-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.875 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.875 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:35.876 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:35.876 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.876 DEBUG:boto:Path: /ppfahn-21/dummy 2024-03-04T17:50:35.876 DEBUG:boto:Data: 2024-03-04T17:50:35.876 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.876 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:35.876 DEBUG:boto:Port: 8001 2024-03-04T17:50:35.876 DEBUG:boto:Params: {} 2024-03-04T17:50:35.876 DEBUG:boto:Token: None 2024-03-04T17:50:35.877 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/dummy 2024-03-04T17:50:35.877 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:i8TUIbATRnFmTZ/giGBLDogqC7s= 2024-03-04T17:50:35.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:i8TUIbATRnFmTZ/giGBLDogqC7s=', 'Content-Length': '0'} 2024-03-04T17:50:35.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b5f1e1075e14aa7c-0065e609eb-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.881 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:35.881 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:35.881 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.881 DEBUG:boto:Path: /ppfahn-21/dummy 2024-03-04T17:50:35.881 DEBUG:boto:Data: 2024-03-04T17:50:35.881 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.881 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.881 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.882 DEBUG:boto:Params: {} 2024-03-04T17:50:35.882 DEBUG:boto:Token: None 2024-03-04T17:50:35.882 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/dummy 2024-03-04T17:50:35.882 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:i8TUIbATRnFmTZ/giGBLDogqC7s= 2024-03-04T17:50:35.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:i8TUIbATRnFmTZ/giGBLDogqC7s=', 'Content-Length': '0'} 2024-03-04T17:50:35.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca8f81587caf0d58-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:35.886 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:35.886 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:35.887 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.887 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.887 DEBUG:boto:Method: GET 2024-03-04T17:50:35.887 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.887 DEBUG:boto:Data: 2024-03-04T17:50:35.887 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.887 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:35.887 DEBUG:boto:Port: 8001 2024-03-04T17:50:35.887 DEBUG:boto:Params: {} 2024-03-04T17:50:35.887 DEBUG:boto:Token: None 2024-03-04T17:50:35.888 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.888 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YtAH190a7m23eZWwo4Tq0CUs+UE= 2024-03-04T17:50:35.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YtAH190a7m23eZWwo4Tq0CUs+UE=', 'Content-Length': '0'} 2024-03-04T17:50:35.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000607a298a66934b17-0065e609eb-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.891 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:35.891 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:35.891 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.891 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.891 DEBUG:boto:Method: GET 2024-03-04T17:50:35.891 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.891 DEBUG:boto:Data: 2024-03-04T17:50:35.891 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.891 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.891 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.892 DEBUG:boto:Params: {} 2024-03-04T17:50:35.892 DEBUG:boto:Token: None 2024-03-04T17:50:35.892 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:35.892 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YtAH190a7m23eZWwo4Tq0CUs+UE= 2024-03-04T17:50:35.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YtAH190a7m23eZWwo4Tq0CUs+UE=', 'Content-Length': '0'} 2024-03-04T17:50:35.895 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1bbe7f108042229-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.896 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.896 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.896 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:35.896 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:35.896 DEBUG:boto:Method: GET 2024-03-04T17:50:35.896 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:50:35.896 DEBUG:boto:Data: 2024-03-04T17:50:35.896 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.896 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:35.896 DEBUG:boto:Port: 8001 2024-03-04T17:50:35.897 DEBUG:boto:Params: {} 2024-03-04T17:50:35.897 DEBUG:boto:Token: None 2024-03-04T17:50:35.897 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.897 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI= 2024-03-04T17:50:35.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI=', 'Content-Length': '0'} 2024-03-04T17:50:35.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4909dc5128c1fef-0065e609eb-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.902 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.902 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:35.903 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:35.903 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:35.903 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:35.903 DEBUG:boto:Method: GET 2024-03-04T17:50:35.903 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:50:35.903 DEBUG:boto:Data: 2024-03-04T17:50:35.903 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.903 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.903 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.904 DEBUG:boto:Params: {} 2024-03-04T17:50:35.904 DEBUG:boto:Token: None 2024-03-04T17:50:35.904 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/ 2024-03-04T17:50:35.904 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI= 2024-03-04T17:50:35.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wlbt3QH/cEpeI8AE1lccVHtg3aI=', 'Content-Length': '0'} 2024-03-04T17:50:35.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000523801e26d2a997-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.909 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:35.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:35.910 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:35.910 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:35.910 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.910 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.910 DEBUG:boto:Method: GET 2024-03-04T17:50:35.910 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.910 DEBUG:boto:Data: 2024-03-04T17:50:35.910 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.911 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:35.911 DEBUG:boto:Port: 8001 2024-03-04T17:50:35.911 DEBUG:boto:Params: {} 2024-03-04T17:50:35.911 DEBUG:boto:Token: None 2024-03-04T17:50:35.911 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.911 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/Sbgh6IxO9XnGuehSLrP9O+QuC0= 2024-03-04T17:50:35.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/Sbgh6IxO9XnGuehSLrP9O+QuC0=', 'Content-Length': '0'} 2024-03-04T17:50:35.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a370ac42e6ba878e-0065e609eb-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.960 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:35.960 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:35.960 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.960 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.960 DEBUG:boto:Method: GET 2024-03-04T17:50:35.960 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.961 DEBUG:boto:Data: 2024-03-04T17:50:35.961 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.961 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.961 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.961 DEBUG:boto:Params: {} 2024-03-04T17:50:35.961 DEBUG:boto:Token: None 2024-03-04T17:50:35.961 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.961 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/Sbgh6IxO9XnGuehSLrP9O+QuC0= 2024-03-04T17:50:35.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/Sbgh6IxO9XnGuehSLrP9O+QuC0=', 'Content-Length': '0'} 2024-03-04T17:50:35.964 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1d2d7098aff6cf5-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.965 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:35.965 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:35.965 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.965 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.965 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.965 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.965 DEBUG:boto:Data: 2024-03-04T17:50:35.965 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.966 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:35.966 DEBUG:boto:Port: 8001 2024-03-04T17:50:35.966 DEBUG:boto:Params: {} 2024-03-04T17:50:35.966 DEBUG:boto:Token: None 2024-03-04T17:50:35.966 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.966 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:L9wZkgyVvgxdjETIldwDz0S6who= 2024-03-04T17:50:35.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:L9wZkgyVvgxdjETIldwDz0S6who=', 'Content-Length': '0'} 2024-03-04T17:50:35.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c93754297a13b45-0065e609eb-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.969 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:35.969 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:35.969 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.969 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.970 DEBUG:boto:Method: HEAD 2024-03-04T17:50:35.970 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.970 DEBUG:boto:Data: 2024-03-04T17:50:35.970 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.970 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:35.970 DEBUG:boto:Port: 8000 2024-03-04T17:50:35.970 DEBUG:boto:Params: {} 2024-03-04T17:50:35.970 DEBUG:boto:Token: None 2024-03-04T17:50:35.970 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.970 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:L9wZkgyVvgxdjETIldwDz0S6who= 2024-03-04T17:50:35.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:L9wZkgyVvgxdjETIldwDz0S6who=', 'Content-Length': '0'} 2024-03-04T17:50:35.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2fe21cc28c3dd58-0065e609eb-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:35.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:35.974 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:35.974 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:35.974 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.974 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.974 DEBUG:boto:Method: GET 2024-03-04T17:50:35.974 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.974 DEBUG:boto:Data: 2024-03-04T17:50:35.974 DEBUG:boto:Headers: {} 2024-03-04T17:50:35.974 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:35.974 DEBUG:boto:Port: 8001 2024-03-04T17:50:35.975 DEBUG:boto:Params: {} 2024-03-04T17:50:35.975 DEBUG:boto:Token: None 2024-03-04T17:50:35.975 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:35 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:35.975 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/Sbgh6IxO9XnGuehSLrP9O+QuC0= 2024-03-04T17:50:35.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/Sbgh6IxO9XnGuehSLrP9O+QuC0=', 'Content-Length': '0'} 2024-03-04T17:50:35.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d78c77d12fee44a5-0065e609eb-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.020 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.020 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.020 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.020 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.020 DEBUG:boto:Method: GET 2024-03-04T17:50:36.021 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.021 DEBUG:boto:Data: 2024-03-04T17:50:36.021 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.021 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.021 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.021 DEBUG:boto:Params: {} 2024-03-04T17:50:36.021 DEBUG:boto:Token: None 2024-03-04T17:50:36.021 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.021 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:84M7kWpeA6YwllDro/XSuJ6bh1w= 2024-03-04T17:50:36.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:84M7kWpeA6YwllDro/XSuJ6bh1w=', 'Content-Length': '0'} 2024-03-04T17:50:36.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087fff218c8e0db9f-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.025 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.025 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.025 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:36.025 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:36.025 DEBUG:boto:Method: GET 2024-03-04T17:50:36.025 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:36.025 DEBUG:boto:Data: 2024-03-04T17:50:36.026 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.026 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.026 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.026 DEBUG:boto:Params: {} 2024-03-04T17:50:36.026 DEBUG:boto:Token: None 2024-03-04T17:50:36.026 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.026 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041dacc15ffaf8a81-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.072 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.072 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.073 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.073 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:36.073 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:36.073 DEBUG:boto:Method: GET 2024-03-04T17:50:36.073 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:36.073 DEBUG:boto:Data: 2024-03-04T17:50:36.073 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.073 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.073 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.073 DEBUG:boto:Params: {} 2024-03-04T17:50:36.074 DEBUG:boto:Token: None 2024-03-04T17:50:36.074 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.074 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056d9ec659ef7a585-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.079 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.080 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.080 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.080 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.080 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.080 DEBUG:boto:Data: 2024-03-04T17:50:36.080 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.080 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.081 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.081 DEBUG:boto:Params: {} 2024-03-04T17:50:36.081 DEBUG:boto:Token: None 2024-03-04T17:50:36.081 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.081 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vGYjHD8F2pTdRtrdtd7tmcD7cO0= 2024-03-04T17:50:36.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vGYjHD8F2pTdRtrdtd7tmcD7cO0=', 'Content-Length': '0'} 2024-03-04T17:50:36.085 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003641754aadf203a1-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.085 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.086 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.086 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.086 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.086 DEBUG:boto:Data: 2024-03-04T17:50:36.086 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.086 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.086 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.086 DEBUG:boto:Params: {} 2024-03-04T17:50:36.086 DEBUG:boto:Token: None 2024-03-04T17:50:36.086 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.087 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vGYjHD8F2pTdRtrdtd7tmcD7cO0= 2024-03-04T17:50:36.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vGYjHD8F2pTdRtrdtd7tmcD7cO0=', 'Content-Length': '0'} 2024-03-04T17:50:36.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068f88fd28973fe43-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:36.091 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.091 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.091 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.091 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.091 DEBUG:boto:Method: GET 2024-03-04T17:50:36.091 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.091 DEBUG:boto:Data: 2024-03-04T17:50:36.091 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.091 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.091 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.091 DEBUG:boto:Params: {} 2024-03-04T17:50:36.092 DEBUG:boto:Token: None 2024-03-04T17:50:36.092 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.092 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:84M7kWpeA6YwllDro/XSuJ6bh1w= 2024-03-04T17:50:36.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:84M7kWpeA6YwllDro/XSuJ6bh1w=', 'Content-Length': '0'} 2024-03-04T17:50:36.095 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecef2066dab7d7f1-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.140 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.140 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.141 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.141 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.141 DEBUG:boto:Method: GET 2024-03-04T17:50:36.141 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.141 DEBUG:boto:Data: 2024-03-04T17:50:36.141 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.141 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.141 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.141 DEBUG:boto:Params: {} 2024-03-04T17:50:36.142 DEBUG:boto:Token: None 2024-03-04T17:50:36.142 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.142 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:84M7kWpeA6YwllDro/XSuJ6bh1w= 2024-03-04T17:50:36.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:84M7kWpeA6YwllDro/XSuJ6bh1w=', 'Content-Length': '0'} 2024-03-04T17:50:36.146 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021eb18959b8a41cd-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.146 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.146 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.146 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:36.147 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:36.147 DEBUG:boto:Method: GET 2024-03-04T17:50:36.147 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:36.147 DEBUG:boto:Data: 2024-03-04T17:50:36.147 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.147 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.147 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.147 DEBUG:boto:Params: {} 2024-03-04T17:50:36.147 DEBUG:boto:Token: None 2024-03-04T17:50:36.147 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.147 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055e6385ea3660585-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.196 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.197 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.197 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.197 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:36.197 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:36.198 DEBUG:boto:Method: GET 2024-03-04T17:50:36.198 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:36.198 DEBUG:boto:Data: 2024-03-04T17:50:36.198 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.198 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.199 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.199 DEBUG:boto:Params: {} 2024-03-04T17:50:36.199 DEBUG:boto:Token: None 2024-03-04T17:50:36.199 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.200 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8a9fc301d32e49a-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.205 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:36.206 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.206 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.206 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.206 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.206 DEBUG:boto:Method: GET 2024-03-04T17:50:36.206 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.206 DEBUG:boto:Data: 2024-03-04T17:50:36.206 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.206 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.207 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.207 DEBUG:boto:Params: {} 2024-03-04T17:50:36.207 DEBUG:boto:Token: None 2024-03-04T17:50:36.207 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.207 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZnEaDYswQNORGokridq6BwwPlWI= 2024-03-04T17:50:36.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZnEaDYswQNORGokridq6BwwPlWI=', 'Content-Length': '0'} 2024-03-04T17:50:36.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020116dc5b0366fc7-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.211 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.211 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.211 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.211 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.211 DEBUG:boto:Method: GET 2024-03-04T17:50:36.211 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.211 DEBUG:boto:Data: 2024-03-04T17:50:36.212 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.212 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.212 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.212 DEBUG:boto:Params: {} 2024-03-04T17:50:36.212 DEBUG:boto:Token: None 2024-03-04T17:50:36.212 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.213 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZnEaDYswQNORGokridq6BwwPlWI= 2024-03-04T17:50:36.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZnEaDYswQNORGokridq6BwwPlWI=', 'Content-Length': '0'} 2024-03-04T17:50:36.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b1f7b7f734d556b-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.264 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.264 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.264 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.264 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.265 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.265 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.265 DEBUG:boto:Data: 2024-03-04T17:50:36.265 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.265 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.265 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.265 DEBUG:boto:Params: {} 2024-03-04T17:50:36.265 DEBUG:boto:Token: None 2024-03-04T17:50:36.265 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.266 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fbK2u+Um9LYmZSM3i4RSJp6qYmE= 2024-03-04T17:50:36.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fbK2u+Um9LYmZSM3i4RSJp6qYmE=', 'Content-Length': '0'} 2024-03-04T17:50:36.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b9f9f0a0a8871a6-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.269 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.269 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.269 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.269 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.269 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.269 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.269 DEBUG:boto:Data: 2024-03-04T17:50:36.269 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.270 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.270 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.270 DEBUG:boto:Params: {} 2024-03-04T17:50:36.270 DEBUG:boto:Token: None 2024-03-04T17:50:36.270 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.270 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fbK2u+Um9LYmZSM3i4RSJp6qYmE= 2024-03-04T17:50:36.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fbK2u+Um9LYmZSM3i4RSJp6qYmE=', 'Content-Length': '0'} 2024-03-04T17:50:36.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd4dfe6221bbc5ef-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:36.273 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.273 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.273 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.273 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.274 DEBUG:boto:Method: GET 2024-03-04T17:50:36.274 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.274 DEBUG:boto:Data: 2024-03-04T17:50:36.274 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.274 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.274 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.274 DEBUG:boto:Params: {} 2024-03-04T17:50:36.274 DEBUG:boto:Token: None 2024-03-04T17:50:36.274 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.274 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZnEaDYswQNORGokridq6BwwPlWI= 2024-03-04T17:50:36.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZnEaDYswQNORGokridq6BwwPlWI=', 'Content-Length': '0'} 2024-03-04T17:50:36.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0144012c2a806e3-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.320 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.320 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.320 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.320 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.320 DEBUG:boto:Method: GET 2024-03-04T17:50:36.321 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.321 DEBUG:boto:Data: 2024-03-04T17:50:36.321 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.321 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.321 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.321 DEBUG:boto:Params: {} 2024-03-04T17:50:36.321 DEBUG:boto:Token: None 2024-03-04T17:50:36.321 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.322 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZnEaDYswQNORGokridq6BwwPlWI= 2024-03-04T17:50:36.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZnEaDYswQNORGokridq6BwwPlWI=', 'Content-Length': '0'} 2024-03-04T17:50:36.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088f4479f9745aaa0-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.368 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.368 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.368 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:36.368 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:36.369 DEBUG:boto:Method: GET 2024-03-04T17:50:36.369 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:36.369 DEBUG:boto:Data: 2024-03-04T17:50:36.369 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.369 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.369 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.369 DEBUG:boto:Params: {} 2024-03-04T17:50:36.370 DEBUG:boto:Token: None 2024-03-04T17:50:36.370 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.370 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000753a424fb2c7db9d-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.416 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.416 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.417 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.417 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:36.417 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:36.417 DEBUG:boto:Method: GET 2024-03-04T17:50:36.417 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:36.417 DEBUG:boto:Data: 2024-03-04T17:50:36.417 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.417 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.417 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.417 DEBUG:boto:Params: {} 2024-03-04T17:50:36.418 DEBUG:boto:Token: None 2024-03-04T17:50:36.418 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.418 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082c131c7a853cbd4-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.423 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.423 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.423 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.423 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.424 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.424 DEBUG:boto:Data: 2024-03-04T17:50:36.424 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.424 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.424 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.424 DEBUG:boto:Params: {} 2024-03-04T17:50:36.424 DEBUG:boto:Token: None 2024-03-04T17:50:36.424 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.424 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CtWiecsoIc0FsF5555rSa74nONw= 2024-03-04T17:50:36.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CtWiecsoIc0FsF5555rSa74nONw=', 'Content-Length': '0'} 2024-03-04T17:50:36.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081c2e5f43e04a52c-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.428 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.429 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.429 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.429 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.429 DEBUG:boto:Data: 2024-03-04T17:50:36.429 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.429 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.430 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.430 DEBUG:boto:Params: {} 2024-03-04T17:50:36.430 DEBUG:boto:Token: None 2024-03-04T17:50:36.430 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.430 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CtWiecsoIc0FsF5555rSa74nONw= 2024-03-04T17:50:36.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CtWiecsoIc0FsF5555rSa74nONw=', 'Content-Length': '0'} 2024-03-04T17:50:36.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a8eb940360ff839-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:36.434 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.434 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.434 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.435 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.435 DEBUG:boto:Method: GET 2024-03-04T17:50:36.435 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.435 DEBUG:boto:Data: 2024-03-04T17:50:36.435 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.435 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.435 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.435 DEBUG:boto:Params: {} 2024-03-04T17:50:36.435 DEBUG:boto:Token: None 2024-03-04T17:50:36.435 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.436 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZnEaDYswQNORGokridq6BwwPlWI= 2024-03-04T17:50:36.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZnEaDYswQNORGokridq6BwwPlWI=', 'Content-Length': '0'} 2024-03-04T17:50:36.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004edc27ac73f214d9-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.484 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.484 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:36.484 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.484 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.484 DEBUG:boto:Method: GET 2024-03-04T17:50:36.485 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.485 DEBUG:boto:Data: 2024-03-04T17:50:36.485 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.485 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.485 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.485 DEBUG:boto:Params: {} 2024-03-04T17:50:36.485 DEBUG:boto:Token: None 2024-03-04T17:50:36.485 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:36.485 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZnEaDYswQNORGokridq6BwwPlWI= 2024-03-04T17:50:36.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZnEaDYswQNORGokridq6BwwPlWI=', 'Content-Length': '0'} 2024-03-04T17:50:36.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c96b17a306109d35-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.536 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.537 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.537 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:36.537 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:36.537 DEBUG:boto:Method: GET 2024-03-04T17:50:36.537 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:36.537 DEBUG:boto:Data: 2024-03-04T17:50:36.537 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.537 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.537 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.537 DEBUG:boto:Params: {} 2024-03-04T17:50:36.538 DEBUG:boto:Token: None 2024-03-04T17:50:36.538 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.538 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019d2e47a61e1170c-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.584 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.584 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.585 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.585 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:36.585 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:36.585 DEBUG:boto:Method: GET 2024-03-04T17:50:36.585 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:36.585 DEBUG:boto:Data: 2024-03-04T17:50:36.585 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.585 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.585 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.585 DEBUG:boto:Params: {} 2024-03-04T17:50:36.586 DEBUG:boto:Token: None 2024-03-04T17:50:36.586 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.586 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041c6b105b0064627-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.591 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:36.592 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.592 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.592 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.592 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.593 DEBUG:boto:Method: GET 2024-03-04T17:50:36.593 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.593 DEBUG:boto:Data: 2024-03-04T17:50:36.593 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.593 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.593 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.593 DEBUG:boto:Params: {} 2024-03-04T17:50:36.593 DEBUG:boto:Token: None 2024-03-04T17:50:36.594 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.594 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SGh4s5XcS5eRxDXhjpsIvNgPdTQ= 2024-03-04T17:50:36.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SGh4s5XcS5eRxDXhjpsIvNgPdTQ=', 'Content-Length': '0'} 2024-03-04T17:50:36.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c1c3071acf1cc71-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.597 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.597 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.597 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.597 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.597 DEBUG:boto:Method: GET 2024-03-04T17:50:36.597 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.597 DEBUG:boto:Data: 2024-03-04T17:50:36.598 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.598 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.598 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.598 DEBUG:boto:Params: {} 2024-03-04T17:50:36.598 DEBUG:boto:Token: None 2024-03-04T17:50:36.598 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.598 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SGh4s5XcS5eRxDXhjpsIvNgPdTQ= 2024-03-04T17:50:36.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SGh4s5XcS5eRxDXhjpsIvNgPdTQ=', 'Content-Length': '0'} 2024-03-04T17:50:36.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000475b764e6dfbadfb-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.647 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.647 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.647 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.647 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.647 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.648 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.648 DEBUG:boto:Data: 2024-03-04T17:50:36.648 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.648 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.648 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.648 DEBUG:boto:Params: {} 2024-03-04T17:50:36.648 DEBUG:boto:Token: None 2024-03-04T17:50:36.648 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.649 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Eo3Hds/pnl0WgKgI5BXe2yBvPUU= 2024-03-04T17:50:36.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Eo3Hds/pnl0WgKgI5BXe2yBvPUU=', 'Content-Length': '0'} 2024-03-04T17:50:36.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6747d2879bb4342-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.652 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.653 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.653 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.653 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.653 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.653 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.653 DEBUG:boto:Data: 2024-03-04T17:50:36.653 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.653 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.653 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.653 DEBUG:boto:Params: {} 2024-03-04T17:50:36.653 DEBUG:boto:Token: None 2024-03-04T17:50:36.654 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.654 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Eo3Hds/pnl0WgKgI5BXe2yBvPUU= 2024-03-04T17:50:36.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Eo3Hds/pnl0WgKgI5BXe2yBvPUU=', 'Content-Length': '0'} 2024-03-04T17:50:36.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000350d27ea913b42ff-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:36.657 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.657 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.657 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.658 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.658 DEBUG:boto:Method: GET 2024-03-04T17:50:36.658 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.658 DEBUG:boto:Data: 2024-03-04T17:50:36.658 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.658 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.658 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.658 DEBUG:boto:Params: {} 2024-03-04T17:50:36.658 DEBUG:boto:Token: None 2024-03-04T17:50:36.659 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.659 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SGh4s5XcS5eRxDXhjpsIvNgPdTQ= 2024-03-04T17:50:36.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SGh4s5XcS5eRxDXhjpsIvNgPdTQ=', 'Content-Length': '0'} 2024-03-04T17:50:36.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000563dabb17cb0d4dd-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.662 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.662 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.662 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.662 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.662 DEBUG:boto:Method: GET 2024-03-04T17:50:36.662 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.662 DEBUG:boto:Data: 2024-03-04T17:50:36.662 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.663 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.663 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.663 DEBUG:boto:Params: {} 2024-03-04T17:50:36.663 DEBUG:boto:Token: None 2024-03-04T17:50:36.663 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.663 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SGh4s5XcS5eRxDXhjpsIvNgPdTQ= 2024-03-04T17:50:36.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SGh4s5XcS5eRxDXhjpsIvNgPdTQ=', 'Content-Length': '0'} 2024-03-04T17:50:36.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c897aecb9a53ed7-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.708 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.708 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.708 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:36.708 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:36.709 DEBUG:boto:Method: GET 2024-03-04T17:50:36.709 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:36.709 DEBUG:boto:Data: 2024-03-04T17:50:36.709 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.709 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.709 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.709 DEBUG:boto:Params: {} 2024-03-04T17:50:36.709 DEBUG:boto:Token: None 2024-03-04T17:50:36.710 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.710 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c36109c1220dedaf-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.715 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.715 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.715 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.715 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:36.715 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:36.716 DEBUG:boto:Method: GET 2024-03-04T17:50:36.716 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:36.716 DEBUG:boto:Data: 2024-03-04T17:50:36.716 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.716 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.716 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.716 DEBUG:boto:Params: {} 2024-03-04T17:50:36.716 DEBUG:boto:Token: None 2024-03-04T17:50:36.716 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.717 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036af60418cf412d2-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.722 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.723 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.723 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.723 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.723 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.723 DEBUG:boto:Data: 2024-03-04T17:50:36.723 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.723 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.723 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.723 DEBUG:boto:Params: {} 2024-03-04T17:50:36.724 DEBUG:boto:Token: None 2024-03-04T17:50:36.724 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.724 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Sww8vbuGJ/9MDhUmocN6wgrOqRg= 2024-03-04T17:50:36.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sww8vbuGJ/9MDhUmocN6wgrOqRg=', 'Content-Length': '0'} 2024-03-04T17:50:36.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001036a2fb2531dfd5-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.727 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.728 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.728 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.728 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.728 DEBUG:boto:Data: 2024-03-04T17:50:36.728 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.728 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.728 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.728 DEBUG:boto:Params: {} 2024-03-04T17:50:36.729 DEBUG:boto:Token: None 2024-03-04T17:50:36.729 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.729 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Sww8vbuGJ/9MDhUmocN6wgrOqRg= 2024-03-04T17:50:36.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sww8vbuGJ/9MDhUmocN6wgrOqRg=', 'Content-Length': '0'} 2024-03-04T17:50:36.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000950d145ac734d7bc-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:36.735 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.735 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.735 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.735 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.735 DEBUG:boto:Method: GET 2024-03-04T17:50:36.735 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.735 DEBUG:boto:Data: 2024-03-04T17:50:36.735 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.736 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.736 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.736 DEBUG:boto:Params: {} 2024-03-04T17:50:36.736 DEBUG:boto:Token: None 2024-03-04T17:50:36.736 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.736 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SGh4s5XcS5eRxDXhjpsIvNgPdTQ= 2024-03-04T17:50:36.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SGh4s5XcS5eRxDXhjpsIvNgPdTQ=', 'Content-Length': '0'} 2024-03-04T17:50:36.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ad4ede3b749d3ac-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.739 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.740 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:36.740 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.740 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.740 DEBUG:boto:Method: GET 2024-03-04T17:50:36.740 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.740 DEBUG:boto:Data: 2024-03-04T17:50:36.740 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.740 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.740 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.740 DEBUG:boto:Params: {} 2024-03-04T17:50:36.740 DEBUG:boto:Token: None 2024-03-04T17:50:36.741 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:36.741 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SGh4s5XcS5eRxDXhjpsIvNgPdTQ= 2024-03-04T17:50:36.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SGh4s5XcS5eRxDXhjpsIvNgPdTQ=', 'Content-Length': '0'} 2024-03-04T17:50:36.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052d82c2c1118a44a-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.784 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.784 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.784 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:36.784 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:36.785 DEBUG:boto:Method: GET 2024-03-04T17:50:36.785 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:36.785 DEBUG:boto:Data: 2024-03-04T17:50:36.785 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.785 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.785 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.785 DEBUG:boto:Params: {} 2024-03-04T17:50:36.785 DEBUG:boto:Token: None 2024-03-04T17:50:36.785 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.785 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9cd9a126b48cc9f-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.790 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.791 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.791 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.791 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:36.791 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:36.791 DEBUG:boto:Method: GET 2024-03-04T17:50:36.791 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:36.791 DEBUG:boto:Data: 2024-03-04T17:50:36.791 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.791 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:36.791 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.791 DEBUG:boto:Params: {} 2024-03-04T17:50:36.792 DEBUG:boto:Token: None 2024-03-04T17:50:36.792 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.792 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001ca1729ee4531fa-0065e609ec-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.797 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.798 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-21 zones={test-zone3, test-zone1} 2024-03-04T17:50:36.798 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-21 zones={test-zone3, test-zone2} 2024-03-04T17:50:36.798 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.798 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.798 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.798 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:50:36.799 DEBUG:boto:Data: 2024-03-04T17:50:36.799 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.799 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.799 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.799 DEBUG:boto:Params: {} 2024-03-04T17:50:36.799 DEBUG:boto:Token: None 2024-03-04T17:50:36.799 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.799 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ES1AUYLexJ9lKrkT5IYX03BNais= 2024-03-04T17:50:36.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ES1AUYLexJ9lKrkT5IYX03BNais=', 'Content-Length': '0'} 2024-03-04T17:50:36.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de1498f6bc3e10af-0065e609ec-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.804 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.804 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.804 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.804 DEBUG:boto:Path: /ppfahn-21/ 2024-03-04T17:50:36.804 DEBUG:boto:Data: 2024-03-04T17:50:36.805 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.805 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:36.805 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.805 DEBUG:boto:Params: {} 2024-03-04T17:50:36.805 DEBUG:boto:Token: None 2024-03-04T17:50:36.805 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.805 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ES1AUYLexJ9lKrkT5IYX03BNais= 2024-03-04T17:50:36.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ES1AUYLexJ9lKrkT5IYX03BNais=', 'Content-Length': '0'} 2024-03-04T17:50:36.809 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000a10db71892de31-0065e609ec-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.809 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:50:36.810 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.810 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.810 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:36.810 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:36.810 DEBUG:boto:Method: GET 2024-03-04T17:50:36.810 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:36.810 DEBUG:boto:Data: 2024-03-04T17:50:36.810 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.810 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.810 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.811 DEBUG:boto:Params: {} 2024-03-04T17:50:36.811 DEBUG:boto:Token: None 2024-03-04T17:50:36.811 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/?versions 2024-03-04T17:50:36.811 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1f5nc0nJPMgV33vN0h5TeDiKv9M= 2024-03-04T17:50:36.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1f5nc0nJPMgV33vN0h5TeDiKv9M=', 'Content-Length': '0'} 2024-03-04T17:50:36.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023d5dcbfcbc875e8-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.815 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.816 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:36.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:36.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:36.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:36.816 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:50:36.816 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.817 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.817 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:36.817 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:36.817 DEBUG:boto:Method: GET 2024-03-04T17:50:36.817 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:36.817 DEBUG:boto:Data: 2024-03-04T17:50:36.817 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.817 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:36.817 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.817 DEBUG:boto:Params: {} 2024-03-04T17:50:36.817 DEBUG:boto:Token: None 2024-03-04T17:50:36.818 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/?versions 2024-03-04T17:50:36.818 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1f5nc0nJPMgV33vN0h5TeDiKv9M= 2024-03-04T17:50:36.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1f5nc0nJPMgV33vN0h5TeDiKv9M=', 'Content-Length': '0'} 2024-03-04T17:50:36.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0323e3d84632e61-0065e609ec-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.823 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.823 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:50:36.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T17:50:36.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T17:50:36.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T17:50:36.824 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.824 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.824 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:36.824 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:36.824 DEBUG:boto:Method: GET 2024-03-04T17:50:36.824 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:36.824 DEBUG:boto:Data: 2024-03-04T17:50:36.824 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.824 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.824 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.825 DEBUG:boto:Params: {} 2024-03-04T17:50:36.825 DEBUG:boto:Token: None 2024-03-04T17:50:36.825 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/?versions 2024-03-04T17:50:36.825 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1f5nc0nJPMgV33vN0h5TeDiKv9M= 2024-03-04T17:50:36.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1f5nc0nJPMgV33vN0h5TeDiKv9M=', 'Content-Length': '0'} 2024-03-04T17:50:36.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016e453187485e7b6-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.830 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.830 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.830 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.831 DEBUG:boto:path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:36.831 DEBUG:boto:auth_path=/ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:36.831 DEBUG:boto:Method: GET 2024-03-04T17:50:36.831 DEBUG:boto:Path: /ppfahn-21/?versions&max-keys=999 2024-03-04T17:50:36.831 DEBUG:boto:Data: 2024-03-04T17:50:36.831 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.831 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:36.831 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.831 DEBUG:boto:Params: {} 2024-03-04T17:50:36.831 DEBUG:boto:Token: None 2024-03-04T17:50:36.832 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/?versions 2024-03-04T17:50:36.832 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1f5nc0nJPMgV33vN0h5TeDiKv9M= 2024-03-04T17:50:36.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1f5nc0nJPMgV33vN0h5TeDiKv9M=', 'Content-Length': '0'} 2024-03-04T17:50:36.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f4396c57fb1fd7d-0065e609ec-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.836 DEBUG:boto:b'ppfahn-21999falsedummy2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2true2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Kia8Fr4CXboylh9TkrKjo8GzQMzX9tHtrue2024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gR2Se9STTMjhGJgttYGgRTNmJpnVDYNtrue2024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone39QsZhAGXMFfHaJPzDaiErNYQ7QImfSStrue2024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:36.837 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:36.837 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:36.837 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.837 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.837 DEBUG:boto:Method: GET 2024-03-04T17:50:36.837 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.837 DEBUG:boto:Data: 2024-03-04T17:50:36.837 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.837 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.838 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.838 DEBUG:boto:Params: {} 2024-03-04T17:50:36.838 DEBUG:boto:Token: None 2024-03-04T17:50:36.838 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.838 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tftvtiv2noZDITFIuqDHhNY9hCM= 2024-03-04T17:50:36.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tftvtiv2noZDITFIuqDHhNY9hCM=', 'Content-Length': '0'} 2024-03-04T17:50:36.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a3d066e57ee7e40f-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.841 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:36.842 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:36.842 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.842 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.842 DEBUG:boto:Method: GET 2024-03-04T17:50:36.842 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.842 DEBUG:boto:Data: 2024-03-04T17:50:36.842 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.842 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:36.842 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.843 DEBUG:boto:Params: {} 2024-03-04T17:50:36.843 DEBUG:boto:Token: None 2024-03-04T17:50:36.843 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.843 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tftvtiv2noZDITFIuqDHhNY9hCM= 2024-03-04T17:50:36.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tftvtiv2noZDITFIuqDHhNY9hCM=', 'Content-Length': '0'} 2024-03-04T17:50:36.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f0f974244e8e095d-0065e609ec-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.846 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:36.846 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:36.846 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.846 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.847 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.847 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.847 DEBUG:boto:Data: 2024-03-04T17:50:36.847 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.847 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.847 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.847 DEBUG:boto:Params: {} 2024-03-04T17:50:36.847 DEBUG:boto:Token: None 2024-03-04T17:50:36.847 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.847 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nOd7lCIgL+XqNuq8sQW51qy5fNM= 2024-03-04T17:50:36.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nOd7lCIgL+XqNuq8sQW51qy5fNM=', 'Content-Length': '0'} 2024-03-04T17:50:36.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d9dc19149a8cf2e-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.851 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:36.852 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:36.852 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.852 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.852 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.852 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.852 DEBUG:boto:Data: 2024-03-04T17:50:36.852 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.852 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:36.852 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.852 DEBUG:boto:Params: {} 2024-03-04T17:50:36.853 DEBUG:boto:Token: None 2024-03-04T17:50:36.853 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.853 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nOd7lCIgL+XqNuq8sQW51qy5fNM= 2024-03-04T17:50:36.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nOd7lCIgL+XqNuq8sQW51qy5fNM=', 'Content-Length': '0'} 2024-03-04T17:50:36.856 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007cdbc1b38ba6a0b3-0065e609ec-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:36.856 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:36.856 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:36.856 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.857 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.857 DEBUG:boto:Method: GET 2024-03-04T17:50:36.857 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.857 DEBUG:boto:Data: 2024-03-04T17:50:36.857 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.857 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.857 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.857 DEBUG:boto:Params: {} 2024-03-04T17:50:36.857 DEBUG:boto:Token: None 2024-03-04T17:50:36.857 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.858 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tftvtiv2noZDITFIuqDHhNY9hCM= 2024-03-04T17:50:36.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tftvtiv2noZDITFIuqDHhNY9hCM=', 'Content-Length': '0'} 2024-03-04T17:50:36.861 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d2256925e3a4cf8-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.861 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:36.861 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:36.861 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.861 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.861 DEBUG:boto:Method: GET 2024-03-04T17:50:36.862 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.862 DEBUG:boto:Data: 2024-03-04T17:50:36.862 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.862 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:36.862 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.862 DEBUG:boto:Params: {} 2024-03-04T17:50:36.862 DEBUG:boto:Token: None 2024-03-04T17:50:36.862 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.862 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tftvtiv2noZDITFIuqDHhNY9hCM= 2024-03-04T17:50:36.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tftvtiv2noZDITFIuqDHhNY9hCM=', 'Content-Length': '0'} 2024-03-04T17:50:36.865 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ca5180ea309b4e7-0065e609ec-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.865 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.866 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.866 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:36.866 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:36.866 DEBUG:boto:Method: GET 2024-03-04T17:50:36.866 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:50:36.866 DEBUG:boto:Data: 2024-03-04T17:50:36.866 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.866 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.866 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.866 DEBUG:boto:Params: {} 2024-03-04T17:50:36.867 DEBUG:boto:Token: None 2024-03-04T17:50:36.867 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.867 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1caaee97c1bc5bc-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.912 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.912 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.913 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.913 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:36.913 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:36.913 DEBUG:boto:Method: GET 2024-03-04T17:50:36.913 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:50:36.913 DEBUG:boto:Data: 2024-03-04T17:50:36.913 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.913 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:36.913 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.914 DEBUG:boto:Params: {} 2024-03-04T17:50:36.914 DEBUG:boto:Token: None 2024-03-04T17:50:36.914 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.914 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000addc4a34fba951d8-0065e609ec-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.918 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.919 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:36.919 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:36.919 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.919 DEBUG:boto:Path: /ppfahn-21/dummy 2024-03-04T17:50:36.919 DEBUG:boto:Data: 2024-03-04T17:50:36.919 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.919 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.919 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.920 DEBUG:boto:Params: {} 2024-03-04T17:50:36.920 DEBUG:boto:Token: None 2024-03-04T17:50:36.920 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/dummy 2024-03-04T17:50:36.920 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T35zNSuk122hUcHkXmrov6WQ+eo= 2024-03-04T17:50:36.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T35zNSuk122hUcHkXmrov6WQ+eo=', 'Content-Length': '0'} 2024-03-04T17:50:36.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003bf199daf5924b16-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.924 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:36.925 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:36.925 DEBUG:boto:Method: HEAD 2024-03-04T17:50:36.925 DEBUG:boto:Path: /ppfahn-21/dummy 2024-03-04T17:50:36.925 DEBUG:boto:Data: 2024-03-04T17:50:36.925 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.925 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:36.925 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.925 DEBUG:boto:Params: {} 2024-03-04T17:50:36.925 DEBUG:boto:Token: None 2024-03-04T17:50:36.926 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/dummy 2024-03-04T17:50:36.926 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T35zNSuk122hUcHkXmrov6WQ+eo= 2024-03-04T17:50:36.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T35zNSuk122hUcHkXmrov6WQ+eo=', 'Content-Length': '0'} 2024-03-04T17:50:36.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fe671ca24d79458f-0065e609ec-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:50:36.930 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:36.930 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:36.930 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.930 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.931 DEBUG:boto:Method: GET 2024-03-04T17:50:36.931 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.931 DEBUG:boto:Data: 2024-03-04T17:50:36.931 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.931 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.931 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.931 DEBUG:boto:Params: {} 2024-03-04T17:50:36.931 DEBUG:boto:Token: None 2024-03-04T17:50:36.932 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.932 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tftvtiv2noZDITFIuqDHhNY9hCM= 2024-03-04T17:50:36.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tftvtiv2noZDITFIuqDHhNY9hCM=', 'Content-Length': '0'} 2024-03-04T17:50:36.935 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d3cd4e1648517fbf-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.935 DEBUG:boto:path=/ppfahn-21/dummy 2024-03-04T17:50:36.935 DEBUG:boto:auth_path=/ppfahn-21/dummy 2024-03-04T17:50:36.935 DEBUG:boto:path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.935 DEBUG:boto:auth_path=/ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.935 DEBUG:boto:Method: GET 2024-03-04T17:50:36.936 DEBUG:boto:Path: /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.936 DEBUG:boto:Data: 2024-03-04T17:50:36.936 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.936 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:36.936 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.936 DEBUG:boto:Params: {} 2024-03-04T17:50:36.936 DEBUG:boto:Token: None 2024-03-04T17:50:36.936 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/dummy?versionId=2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2 2024-03-04T17:50:36.937 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tftvtiv2noZDITFIuqDHhNY9hCM= 2024-03-04T17:50:36.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tftvtiv2noZDITFIuqDHhNY9hCM=', 'Content-Length': '0'} 2024-03-04T17:50:36.939 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:30 GMT'), ('x-amz-version-id', '2rTnsQmxetv18BnGN4MEWQV7H6Uqwh2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b8c5aa068f303dc-0065e609ec-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.940 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.940 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.940 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:36.940 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:36.940 DEBUG:boto:Method: GET 2024-03-04T17:50:36.940 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:50:36.940 DEBUG:boto:Data: 2024-03-04T17:50:36.941 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.941 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.941 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.941 DEBUG:boto:Params: {} 2024-03-04T17:50:36.941 DEBUG:boto:Token: None 2024-03-04T17:50:36.941 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.941 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084ab38adf781bfec-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.946 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.946 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:36.946 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:36.947 DEBUG:boto:path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:36.947 DEBUG:boto:auth_path=/ppfahn-21/?prefix=dummy 2024-03-04T17:50:36.947 DEBUG:boto:Method: GET 2024-03-04T17:50:36.947 DEBUG:boto:Path: /ppfahn-21/?prefix=dummy 2024-03-04T17:50:36.947 DEBUG:boto:Data: 2024-03-04T17:50:36.947 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.947 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:36.947 DEBUG:boto:Port: 8000 2024-03-04T17:50:36.947 DEBUG:boto:Params: {} 2024-03-04T17:50:36.948 DEBUG:boto:Token: None 2024-03-04T17:50:36.948 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/ 2024-03-04T17:50:36.948 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50= 2024-03-04T17:50:36.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWZAscEH/f1BMV79mLqvxeoRz50=', 'Content-Length': '0'} 2024-03-04T17:50:36.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062576947a825b248-0065e609ec-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:36.953 DEBUG:boto:b'ppfahn-21dummy1000falsedummy2024-03-04T17:46:30.536Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:36.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:36.953 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.953 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:36.953 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.954 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.954 DEBUG:boto:Method: GET 2024-03-04T17:50:36.954 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.954 DEBUG:boto:Data: 2024-03-04T17:50:36.954 DEBUG:boto:Headers: {} 2024-03-04T17:50:36.954 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:36.954 DEBUG:boto:Port: 8001 2024-03-04T17:50:36.954 DEBUG:boto:Params: {} 2024-03-04T17:50:36.954 DEBUG:boto:Token: None 2024-03-04T17:50:36.955 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:36 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:36.955 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:84M7kWpeA6YwllDro/XSuJ6bh1w= 2024-03-04T17:50:36.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:84M7kWpeA6YwllDro/XSuJ6bh1w=', 'Content-Length': '0'} 2024-03-04T17:50:36.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ceceb901107061e8-0065e609ec-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.000 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.000 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.000 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.001 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.001 DEBUG:boto:Method: GET 2024-03-04T17:50:37.001 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.001 DEBUG:boto:Data: 2024-03-04T17:50:37.001 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.001 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.001 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.001 DEBUG:boto:Params: {} 2024-03-04T17:50:37.001 DEBUG:boto:Token: None 2024-03-04T17:50:37.001 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.002 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gev/NpUFAtMjCmYa4FdZqDfiuJM= 2024-03-04T17:50:37.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gev/NpUFAtMjCmYa4FdZqDfiuJM=', 'Content-Length': '0'} 2024-03-04T17:50:37.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028d11f75cf8ce5a0-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.048 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.048 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.048 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.048 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.048 DEBUG:boto:Method: HEAD 2024-03-04T17:50:37.049 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.049 DEBUG:boto:Data: 2024-03-04T17:50:37.049 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.049 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.049 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.049 DEBUG:boto:Params: {} 2024-03-04T17:50:37.049 DEBUG:boto:Token: None 2024-03-04T17:50:37.049 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.050 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eTVmJs6MvCFOQ0oXrsYzLp7gwgs= 2024-03-04T17:50:37.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eTVmJs6MvCFOQ0oXrsYzLp7gwgs=', 'Content-Length': '0'} 2024-03-04T17:50:37.052 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e29bc2c4b5b494e0-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.053 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.053 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.053 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.053 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.053 DEBUG:boto:Method: HEAD 2024-03-04T17:50:37.053 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.053 DEBUG:boto:Data: 2024-03-04T17:50:37.054 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.054 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.054 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.054 DEBUG:boto:Params: {} 2024-03-04T17:50:37.054 DEBUG:boto:Token: None 2024-03-04T17:50:37.054 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.054 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eTVmJs6MvCFOQ0oXrsYzLp7gwgs= 2024-03-04T17:50:37.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eTVmJs6MvCFOQ0oXrsYzLp7gwgs=', 'Content-Length': '0'} 2024-03-04T17:50:37.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da6cf14c62fb5626-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:37.059 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.059 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.059 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.060 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.060 DEBUG:boto:Method: GET 2024-03-04T17:50:37.060 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.060 DEBUG:boto:Data: 2024-03-04T17:50:37.060 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.060 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.060 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.060 DEBUG:boto:Params: {} 2024-03-04T17:50:37.060 DEBUG:boto:Token: None 2024-03-04T17:50:37.061 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.061 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gev/NpUFAtMjCmYa4FdZqDfiuJM= 2024-03-04T17:50:37.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gev/NpUFAtMjCmYa4FdZqDfiuJM=', 'Content-Length': '0'} 2024-03-04T17:50:37.063 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003db112637198fdd6-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.104 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.104 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.104 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.104 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.105 DEBUG:boto:Method: GET 2024-03-04T17:50:37.105 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.105 DEBUG:boto:Data: 2024-03-04T17:50:37.105 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.105 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.105 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.105 DEBUG:boto:Params: {} 2024-03-04T17:50:37.105 DEBUG:boto:Token: None 2024-03-04T17:50:37.106 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.106 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gev/NpUFAtMjCmYa4FdZqDfiuJM= 2024-03-04T17:50:37.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gev/NpUFAtMjCmYa4FdZqDfiuJM=', 'Content-Length': '0'} 2024-03-04T17:50:37.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070bb2d4c5e4af97d-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.152 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:37.152 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:37.152 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:37.152 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:37.152 DEBUG:boto:Method: GET 2024-03-04T17:50:37.153 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:37.153 DEBUG:boto:Data: 2024-03-04T17:50:37.153 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.153 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.153 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.153 DEBUG:boto:Params: {} 2024-03-04T17:50:37.153 DEBUG:boto:Token: None 2024-03-04T17:50:37.153 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/ 2024-03-04T17:50:37.154 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k= 2024-03-04T17:50:37.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k=', 'Content-Length': '0'} 2024-03-04T17:50:37.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008528914440121dc8-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.158 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:37.159 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:37.159 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:37.159 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:37.159 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:37.159 DEBUG:boto:Method: GET 2024-03-04T17:50:37.159 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:37.159 DEBUG:boto:Data: 2024-03-04T17:50:37.160 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.160 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.160 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.160 DEBUG:boto:Params: {} 2024-03-04T17:50:37.160 DEBUG:boto:Token: None 2024-03-04T17:50:37.160 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/ 2024-03-04T17:50:37.160 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k= 2024-03-04T17:50:37.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k=', 'Content-Length': '0'} 2024-03-04T17:50:37.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041bc520f5e6eaf5b-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.165 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:37.166 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.166 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.166 DEBUG:boto:Method: HEAD 2024-03-04T17:50:37.166 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.166 DEBUG:boto:Data: 2024-03-04T17:50:37.166 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.166 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.166 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.166 DEBUG:boto:Params: {} 2024-03-04T17:50:37.167 DEBUG:boto:Token: None 2024-03-04T17:50:37.167 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.167 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sI4Yg2TFD/VsY393jPExwbzP71M= 2024-03-04T17:50:37.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sI4Yg2TFD/VsY393jPExwbzP71M=', 'Content-Length': '0'} 2024-03-04T17:50:37.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023f48ee409d0a215-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.171 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.171 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.171 DEBUG:boto:Method: HEAD 2024-03-04T17:50:37.171 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.171 DEBUG:boto:Data: 2024-03-04T17:50:37.171 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.171 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.172 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.172 DEBUG:boto:Params: {} 2024-03-04T17:50:37.172 DEBUG:boto:Token: None 2024-03-04T17:50:37.172 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.172 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sI4Yg2TFD/VsY393jPExwbzP71M= 2024-03-04T17:50:37.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sI4Yg2TFD/VsY393jPExwbzP71M=', 'Content-Length': '0'} 2024-03-04T17:50:37.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e28e3596428af9e1-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T17:50:37.176 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.177 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.177 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.177 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.177 DEBUG:boto:Method: GET 2024-03-04T17:50:37.177 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.177 DEBUG:boto:Data: 2024-03-04T17:50:37.177 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.177 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.177 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.177 DEBUG:boto:Params: {} 2024-03-04T17:50:37.178 DEBUG:boto:Token: None 2024-03-04T17:50:37.178 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.178 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gev/NpUFAtMjCmYa4FdZqDfiuJM= 2024-03-04T17:50:37.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gev/NpUFAtMjCmYa4FdZqDfiuJM=', 'Content-Length': '0'} 2024-03-04T17:50:37.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a1a39fdf7432070-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.228 DEBUG:boto:path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.228 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1 2024-03-04T17:50:37.228 DEBUG:boto:path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.228 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.229 DEBUG:boto:Method: GET 2024-03-04T17:50:37.229 DEBUG:boto:Path: /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.229 DEBUG:boto:Data: 2024-03-04T17:50:37.229 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.229 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.229 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.229 DEBUG:boto:Params: {} 2024-03-04T17:50:37.230 DEBUG:boto:Token: None 2024-03-04T17:50:37.230 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone1?versionId=Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH 2024-03-04T17:50:37.230 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gev/NpUFAtMjCmYa4FdZqDfiuJM= 2024-03-04T17:50:37.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gev/NpUFAtMjCmYa4FdZqDfiuJM=', 'Content-Length': '0'} 2024-03-04T17:50:37.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'Kia8Fr4CXboylh9TkrKjo8GzQMzX9tH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c12550beed1e292b-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.280 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:37.280 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:37.280 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:37.281 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:37.281 DEBUG:boto:Method: GET 2024-03-04T17:50:37.281 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:37.281 DEBUG:boto:Data: 2024-03-04T17:50:37.281 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.281 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.281 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.281 DEBUG:boto:Params: {} 2024-03-04T17:50:37.282 DEBUG:boto:Token: None 2024-03-04T17:50:37.282 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/ 2024-03-04T17:50:37.282 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k= 2024-03-04T17:50:37.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k=', 'Content-Length': '0'} 2024-03-04T17:50:37.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ae901520309c224-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.287 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:37.287 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:37.287 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:37.288 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:37.288 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:37.288 DEBUG:boto:Method: GET 2024-03-04T17:50:37.288 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone1 2024-03-04T17:50:37.288 DEBUG:boto:Data: 2024-03-04T17:50:37.288 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.288 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.288 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.288 DEBUG:boto:Params: {} 2024-03-04T17:50:37.289 DEBUG:boto:Token: None 2024-03-04T17:50:37.289 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/ 2024-03-04T17:50:37.289 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k= 2024-03-04T17:50:37.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k=', 'Content-Length': '0'} 2024-03-04T17:50:37.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cfc353950262b47-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.293 DEBUG:boto:b'ppfahn-21obj-test-zone11000falseobj-test-zone12024-03-04T17:46:46.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:37.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:37.294 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.294 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.294 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.294 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.294 DEBUG:boto:Method: GET 2024-03-04T17:50:37.294 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.294 DEBUG:boto:Data: 2024-03-04T17:50:37.295 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.295 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.295 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.295 DEBUG:boto:Params: {} 2024-03-04T17:50:37.295 DEBUG:boto:Token: None 2024-03-04T17:50:37.295 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.295 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dY0fu/0hi1ZPnoYVi10AuOeybqc= 2024-03-04T17:50:37.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dY0fu/0hi1ZPnoYVi10AuOeybqc=', 'Content-Length': '0'} 2024-03-04T17:50:37.299 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000592b2126586faab0-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.299 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.299 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.299 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.299 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.300 DEBUG:boto:Method: GET 2024-03-04T17:50:37.300 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.300 DEBUG:boto:Data: 2024-03-04T17:50:37.300 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.300 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.300 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.300 DEBUG:boto:Params: {} 2024-03-04T17:50:37.300 DEBUG:boto:Token: None 2024-03-04T17:50:37.301 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.301 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dY0fu/0hi1ZPnoYVi10AuOeybqc= 2024-03-04T17:50:37.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dY0fu/0hi1ZPnoYVi10AuOeybqc=', 'Content-Length': '0'} 2024-03-04T17:50:37.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000840a97c996be4fba-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.304 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.304 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.304 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.304 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.304 DEBUG:boto:Method: HEAD 2024-03-04T17:50:37.304 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.305 DEBUG:boto:Data: 2024-03-04T17:50:37.305 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.305 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.305 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.305 DEBUG:boto:Params: {} 2024-03-04T17:50:37.305 DEBUG:boto:Token: None 2024-03-04T17:50:37.305 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.305 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QQVKDxcJftS32xjlZhz2PIXWwEg= 2024-03-04T17:50:37.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QQVKDxcJftS32xjlZhz2PIXWwEg=', 'Content-Length': '0'} 2024-03-04T17:50:37.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e62eb94f7258fdb9-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.308 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.308 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.309 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.309 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.309 DEBUG:boto:Method: HEAD 2024-03-04T17:50:37.309 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.309 DEBUG:boto:Data: 2024-03-04T17:50:37.309 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.309 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.309 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.309 DEBUG:boto:Params: {} 2024-03-04T17:50:37.310 DEBUG:boto:Token: None 2024-03-04T17:50:37.310 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.310 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QQVKDxcJftS32xjlZhz2PIXWwEg= 2024-03-04T17:50:37.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QQVKDxcJftS32xjlZhz2PIXWwEg=', 'Content-Length': '0'} 2024-03-04T17:50:37.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fa960a5536c090f-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:37.313 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.313 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.313 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.313 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.314 DEBUG:boto:Method: GET 2024-03-04T17:50:37.314 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.314 DEBUG:boto:Data: 2024-03-04T17:50:37.314 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.314 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.314 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.314 DEBUG:boto:Params: {} 2024-03-04T17:50:37.314 DEBUG:boto:Token: None 2024-03-04T17:50:37.315 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.315 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dY0fu/0hi1ZPnoYVi10AuOeybqc= 2024-03-04T17:50:37.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dY0fu/0hi1ZPnoYVi10AuOeybqc=', 'Content-Length': '0'} 2024-03-04T17:50:37.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035ee7b091fe5e637-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.360 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.360 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.360 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.360 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.360 DEBUG:boto:Method: GET 2024-03-04T17:50:37.360 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.361 DEBUG:boto:Data: 2024-03-04T17:50:37.361 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.361 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.361 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.361 DEBUG:boto:Params: {} 2024-03-04T17:50:37.361 DEBUG:boto:Token: None 2024-03-04T17:50:37.361 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.361 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dY0fu/0hi1ZPnoYVi10AuOeybqc= 2024-03-04T17:50:37.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dY0fu/0hi1ZPnoYVi10AuOeybqc=', 'Content-Length': '0'} 2024-03-04T17:50:37.364 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c77294af4937e4a7-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.365 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:37.365 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:37.365 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:37.365 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:37.365 DEBUG:boto:Method: GET 2024-03-04T17:50:37.365 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:37.365 DEBUG:boto:Data: 2024-03-04T17:50:37.365 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.366 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.366 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.366 DEBUG:boto:Params: {} 2024-03-04T17:50:37.366 DEBUG:boto:Token: None 2024-03-04T17:50:37.366 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/ 2024-03-04T17:50:37.366 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k= 2024-03-04T17:50:37.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k=', 'Content-Length': '0'} 2024-03-04T17:50:37.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000167c67c729116ee2-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.412 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:37.413 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:37.413 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:37.413 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:37.413 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:37.413 DEBUG:boto:Method: GET 2024-03-04T17:50:37.413 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:37.413 DEBUG:boto:Data: 2024-03-04T17:50:37.413 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.413 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.414 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.414 DEBUG:boto:Params: {} 2024-03-04T17:50:37.414 DEBUG:boto:Token: None 2024-03-04T17:50:37.414 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/ 2024-03-04T17:50:37.414 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k= 2024-03-04T17:50:37.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k=', 'Content-Length': '0'} 2024-03-04T17:50:37.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000548874c6c334f3d4-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.419 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:37.419 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.419 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.419 DEBUG:boto:Method: HEAD 2024-03-04T17:50:37.419 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.420 DEBUG:boto:Data: 2024-03-04T17:50:37.420 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.420 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.420 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.420 DEBUG:boto:Params: {} 2024-03-04T17:50:37.420 DEBUG:boto:Token: None 2024-03-04T17:50:37.420 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.420 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:chnmKtKbIJSjks1BU0hNWVPFRcE= 2024-03-04T17:50:37.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:chnmKtKbIJSjks1BU0hNWVPFRcE=', 'Content-Length': '0'} 2024-03-04T17:50:37.424 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2668c23ab5e04a7-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.425 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.425 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.425 DEBUG:boto:Method: HEAD 2024-03-04T17:50:37.425 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.425 DEBUG:boto:Data: 2024-03-04T17:50:37.425 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.425 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.426 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.426 DEBUG:boto:Params: {} 2024-03-04T17:50:37.426 DEBUG:boto:Token: None 2024-03-04T17:50:37.426 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.426 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:chnmKtKbIJSjks1BU0hNWVPFRcE= 2024-03-04T17:50:37.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:chnmKtKbIJSjks1BU0hNWVPFRcE=', 'Content-Length': '0'} 2024-03-04T17:50:37.430 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000d23760cf1aa5e2-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T17:50:37.430 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.430 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.430 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.431 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.431 DEBUG:boto:Method: GET 2024-03-04T17:50:37.431 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.431 DEBUG:boto:Data: 2024-03-04T17:50:37.431 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.431 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.431 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.431 DEBUG:boto:Params: {} 2024-03-04T17:50:37.431 DEBUG:boto:Token: None 2024-03-04T17:50:37.432 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.432 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dY0fu/0hi1ZPnoYVi10AuOeybqc= 2024-03-04T17:50:37.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dY0fu/0hi1ZPnoYVi10AuOeybqc=', 'Content-Length': '0'} 2024-03-04T17:50:37.435 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b21bf6934b458505-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.480 DEBUG:boto:path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.480 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2 2024-03-04T17:50:37.480 DEBUG:boto:path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.480 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.481 DEBUG:boto:Method: GET 2024-03-04T17:50:37.481 DEBUG:boto:Path: /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.481 DEBUG:boto:Data: 2024-03-04T17:50:37.481 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.481 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.481 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.481 DEBUG:boto:Params: {} 2024-03-04T17:50:37.481 DEBUG:boto:Token: None 2024-03-04T17:50:37.481 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone2?versionId=gR2Se9STTMjhGJgttYGgRTNmJpnVDYN 2024-03-04T17:50:37.482 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dY0fu/0hi1ZPnoYVi10AuOeybqc= 2024-03-04T17:50:37.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dY0fu/0hi1ZPnoYVi10AuOeybqc=', 'Content-Length': '0'} 2024-03-04T17:50:37.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:46 GMT'), ('x-amz-version-id', 'gR2Se9STTMjhGJgttYGgRTNmJpnVDYN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcb0d87709e74cba-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.485 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:37.486 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:37.486 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:37.486 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:37.486 DEBUG:boto:Method: GET 2024-03-04T17:50:37.486 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:37.486 DEBUG:boto:Data: 2024-03-04T17:50:37.486 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.486 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.486 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.486 DEBUG:boto:Params: {} 2024-03-04T17:50:37.487 DEBUG:boto:Token: None 2024-03-04T17:50:37.487 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/ 2024-03-04T17:50:37.487 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k= 2024-03-04T17:50:37.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k=', 'Content-Length': '0'} 2024-03-04T17:50:37.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd8b474b8fc14f95-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.532 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:37.533 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:37.533 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:37.533 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:37.533 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:37.533 DEBUG:boto:Method: GET 2024-03-04T17:50:37.533 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone2 2024-03-04T17:50:37.533 DEBUG:boto:Data: 2024-03-04T17:50:37.533 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.533 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.533 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.534 DEBUG:boto:Params: {} 2024-03-04T17:50:37.534 DEBUG:boto:Token: None 2024-03-04T17:50:37.534 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/ 2024-03-04T17:50:37.534 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k= 2024-03-04T17:50:37.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k=', 'Content-Length': '0'} 2024-03-04T17:50:37.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ea3d60711e2ca48-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.539 DEBUG:boto:b'ppfahn-21obj-test-zone21000falseobj-test-zone22024-03-04T17:46:46.894Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:37.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:37.540 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.540 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.540 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.540 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.540 DEBUG:boto:Method: GET 2024-03-04T17:50:37.540 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.540 DEBUG:boto:Data: 2024-03-04T17:50:37.540 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.541 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.541 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.541 DEBUG:boto:Params: {} 2024-03-04T17:50:37.541 DEBUG:boto:Token: None 2024-03-04T17:50:37.541 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.541 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UT0QWWaHT29mjmZuT0DB1AaWxHg= 2024-03-04T17:50:37.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UT0QWWaHT29mjmZuT0DB1AaWxHg=', 'Content-Length': '0'} 2024-03-04T17:50:37.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cb0e518305fdcca-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.544 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.544 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.545 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.545 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.545 DEBUG:boto:Method: GET 2024-03-04T17:50:37.545 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.545 DEBUG:boto:Data: 2024-03-04T17:50:37.545 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.545 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.545 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.545 DEBUG:boto:Params: {} 2024-03-04T17:50:37.545 DEBUG:boto:Token: None 2024-03-04T17:50:37.546 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.546 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UT0QWWaHT29mjmZuT0DB1AaWxHg= 2024-03-04T17:50:37.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UT0QWWaHT29mjmZuT0DB1AaWxHg=', 'Content-Length': '0'} 2024-03-04T17:50:37.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002eea2873e263c0f3-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.592 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.592 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.592 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.593 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.593 DEBUG:boto:Method: HEAD 2024-03-04T17:50:37.593 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.593 DEBUG:boto:Data: 2024-03-04T17:50:37.593 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.593 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.593 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.593 DEBUG:boto:Params: {} 2024-03-04T17:50:37.593 DEBUG:boto:Token: None 2024-03-04T17:50:37.594 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.594 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zpiGGECSdPiSIPg2/qlHhBCMToI= 2024-03-04T17:50:37.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zpiGGECSdPiSIPg2/qlHhBCMToI=', 'Content-Length': '0'} 2024-03-04T17:50:37.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008254e2e11ca50944-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.597 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.597 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.597 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.597 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.598 DEBUG:boto:Method: HEAD 2024-03-04T17:50:37.598 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.598 DEBUG:boto:Data: 2024-03-04T17:50:37.598 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.598 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.598 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.598 DEBUG:boto:Params: {} 2024-03-04T17:50:37.598 DEBUG:boto:Token: None 2024-03-04T17:50:37.598 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.599 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zpiGGECSdPiSIPg2/qlHhBCMToI= 2024-03-04T17:50:37.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zpiGGECSdPiSIPg2/qlHhBCMToI=', 'Content-Length': '0'} 2024-03-04T17:50:37.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd2b4a51158ea369-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:37.602 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.602 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.602 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.602 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.602 DEBUG:boto:Method: GET 2024-03-04T17:50:37.602 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.602 DEBUG:boto:Data: 2024-03-04T17:50:37.602 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.603 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.603 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.603 DEBUG:boto:Params: {} 2024-03-04T17:50:37.603 DEBUG:boto:Token: None 2024-03-04T17:50:37.603 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.603 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UT0QWWaHT29mjmZuT0DB1AaWxHg= 2024-03-04T17:50:37.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UT0QWWaHT29mjmZuT0DB1AaWxHg=', 'Content-Length': '0'} 2024-03-04T17:50:37.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b09d56bc90e04a56-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.606 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.607 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.607 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.607 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.607 DEBUG:boto:Method: GET 2024-03-04T17:50:37.607 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.607 DEBUG:boto:Data: 2024-03-04T17:50:37.607 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.607 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.607 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.607 DEBUG:boto:Params: {} 2024-03-04T17:50:37.608 DEBUG:boto:Token: None 2024-03-04T17:50:37.608 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.608 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UT0QWWaHT29mjmZuT0DB1AaWxHg= 2024-03-04T17:50:37.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UT0QWWaHT29mjmZuT0DB1AaWxHg=', 'Content-Length': '0'} 2024-03-04T17:50:37.611 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd1c3a7d1c1af68a-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.656 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:37.656 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:37.656 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:37.656 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:37.657 DEBUG:boto:Method: GET 2024-03-04T17:50:37.657 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:37.657 DEBUG:boto:Data: 2024-03-04T17:50:37.657 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.657 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.657 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.657 DEBUG:boto:Params: {} 2024-03-04T17:50:37.657 DEBUG:boto:Token: None 2024-03-04T17:50:37.658 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/ 2024-03-04T17:50:37.658 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k= 2024-03-04T17:50:37.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k=', 'Content-Length': '0'} 2024-03-04T17:50:37.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6dad5979691b12a-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.662 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:37.663 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:37.663 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:37.663 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:37.663 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:37.664 DEBUG:boto:Method: GET 2024-03-04T17:50:37.664 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:37.664 DEBUG:boto:Data: 2024-03-04T17:50:37.664 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.664 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.664 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.664 DEBUG:boto:Params: {} 2024-03-04T17:50:37.664 DEBUG:boto:Token: None 2024-03-04T17:50:37.665 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/ 2024-03-04T17:50:37.665 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k= 2024-03-04T17:50:37.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k=', 'Content-Length': '0'} 2024-03-04T17:50:37.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eb4af6675fcda55-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.671 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:37.671 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.672 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.672 DEBUG:boto:Method: HEAD 2024-03-04T17:50:37.672 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.672 DEBUG:boto:Data: 2024-03-04T17:50:37.672 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.672 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.672 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.672 DEBUG:boto:Params: {} 2024-03-04T17:50:37.673 DEBUG:boto:Token: None 2024-03-04T17:50:37.673 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.673 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XF6cAjAe5ZFooNbI0OKzhSHYMKI= 2024-03-04T17:50:37.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XF6cAjAe5ZFooNbI0OKzhSHYMKI=', 'Content-Length': '0'} 2024-03-04T17:50:37.677 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcc87fcf2b5ff5e8-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.677 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.677 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.677 DEBUG:boto:Method: HEAD 2024-03-04T17:50:37.677 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.678 DEBUG:boto:Data: 2024-03-04T17:50:37.678 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.678 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.678 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.678 DEBUG:boto:Params: {} 2024-03-04T17:50:37.678 DEBUG:boto:Token: None 2024-03-04T17:50:37.678 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.678 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XF6cAjAe5ZFooNbI0OKzhSHYMKI= 2024-03-04T17:50:37.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XF6cAjAe5ZFooNbI0OKzhSHYMKI=', 'Content-Length': '0'} 2024-03-04T17:50:37.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fdff64051c29fad-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T17:50:37.683 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.683 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.683 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.683 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.683 DEBUG:boto:Method: GET 2024-03-04T17:50:37.683 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.683 DEBUG:boto:Data: 2024-03-04T17:50:37.684 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.684 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.684 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.684 DEBUG:boto:Params: {} 2024-03-04T17:50:37.684 DEBUG:boto:Token: None 2024-03-04T17:50:37.684 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.684 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UT0QWWaHT29mjmZuT0DB1AaWxHg= 2024-03-04T17:50:37.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UT0QWWaHT29mjmZuT0DB1AaWxHg=', 'Content-Length': '0'} 2024-03-04T17:50:37.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f836d32acdebdc2b-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.728 DEBUG:boto:path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.728 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3 2024-03-04T17:50:37.728 DEBUG:boto:path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.728 DEBUG:boto:auth_path=/ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.729 DEBUG:boto:Method: GET 2024-03-04T17:50:37.729 DEBUG:boto:Path: /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.729 DEBUG:boto:Data: 2024-03-04T17:50:37.729 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.729 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.729 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.729 DEBUG:boto:Params: {} 2024-03-04T17:50:37.729 DEBUG:boto:Token: None 2024-03-04T17:50:37.729 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/obj-test-zone3?versionId=9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS 2024-03-04T17:50:37.730 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UT0QWWaHT29mjmZuT0DB1AaWxHg= 2024-03-04T17:50:37.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UT0QWWaHT29mjmZuT0DB1AaWxHg=', 'Content-Length': '0'} 2024-03-04T17:50:37.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:46:47 GMT'), ('x-amz-version-id', '9QsZhAGXMFfHaJPzDaiErNYQ7QImfSS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-21:d96636ac-054d-40e3-90f3-e3912da784b2.4349.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a13db1082d6ef6e4-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.776 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:37.776 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:37.777 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:37.777 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:37.777 DEBUG:boto:Method: GET 2024-03-04T17:50:37.777 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:37.777 DEBUG:boto:Data: 2024-03-04T17:50:37.777 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.777 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:50:37.777 DEBUG:boto:Port: 8001 2024-03-04T17:50:37.777 DEBUG:boto:Params: {} 2024-03-04T17:50:37.777 DEBUG:boto:Token: None 2024-03-04T17:50:37.778 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/ 2024-03-04T17:50:37.778 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k= 2024-03-04T17:50:37.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k=', 'Content-Length': '0'} 2024-03-04T17:50:37.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006ae71e532b563b2-0065e609ed-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.782 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:37.783 DEBUG:boto:path=/ppfahn-21/ 2024-03-04T17:50:37.783 DEBUG:boto:auth_path=/ppfahn-21/ 2024-03-04T17:50:37.783 DEBUG:boto:path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:37.783 DEBUG:boto:auth_path=/ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:37.783 DEBUG:boto:Method: GET 2024-03-04T17:50:37.783 DEBUG:boto:Path: /ppfahn-21/?prefix=obj-test-zone3 2024-03-04T17:50:37.784 DEBUG:boto:Data: 2024-03-04T17:50:37.784 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.784 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:50:37.784 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.784 DEBUG:boto:Params: {} 2024-03-04T17:50:37.784 DEBUG:boto:Token: None 2024-03-04T17:50:37.784 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-21/ 2024-03-04T17:50:37.784 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k= 2024-03-04T17:50:37.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S6F7s2YOMjONzjwJAPsKqeL1z+k=', 'Content-Length': '0'} 2024-03-04T17:50:37.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a6cb63a44e1497-0065e609ed-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.789 DEBUG:boto:b'ppfahn-21obj-test-zone31000falseobj-test-zone32024-03-04T17:46:47.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:50:37.790 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-21 zones={test-zone3, test-zone2} 2024-03-04T17:50:37.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2024-03-04T17:50:37.793 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:50:37.793 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:50:37.794 DEBUG:boto:Method: PUT 2024-03-04T17:50:37.794 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:50:37.794 DEBUG:boto:Data: 2024-03-04T17:50:37.794 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.794 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:37.794 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.794 DEBUG:boto:Params: {} 2024-03-04T17:50:37.794 DEBUG:boto:Token: None 2024-03-04T17:50:37.794 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-22/ 2024-03-04T17:50:37.795 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6GgyM4ubWAlncIVcYxy8LmOokos= 2024-03-04T17:50:37.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6GgyM4ubWAlncIVcYxy8LmOokos=', 'Content-Length': '0'} 2024-03-04T17:50:37.830 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000130757fcc2263b4d-0065e609ed-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.830 DEBUG:rgw_multi.tests:created bucket=ppfahn-22 2024-03-04T17:50:37.830 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:50:37.830 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:50:37.830 DEBUG:boto:path=/ppfahn-22/?versioning 2024-03-04T17:50:37.830 DEBUG:boto:auth_path=/ppfahn-22/?versioning 2024-03-04T17:50:37.831 DEBUG:boto:Method: PUT 2024-03-04T17:50:37.831 DEBUG:boto:Path: /ppfahn-22/?versioning 2024-03-04T17:50:37.831 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:50:37.831 DEBUG:boto:Headers: {} 2024-03-04T17:50:37.831 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:37.831 DEBUG:boto:Port: 8000 2024-03-04T17:50:37.831 DEBUG:boto:Params: {} 2024-03-04T17:50:37.831 DEBUG:boto:Token: None 2024-03-04T17:50:37.831 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:50:37 GMT /ppfahn-22/?versioning 2024-03-04T17:50:37.832 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0HSX1l7rq3hCUfjdq6/mgWBkG5Q= 2024-03-04T17:50:37.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0HSX1l7rq3hCUfjdq6/mgWBkG5Q=', 'Content-Length': '228'} 2024-03-04T17:50:37.843 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004a05b1024f3a8319-0065e609ed-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:37.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:37.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:37.844 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:37.884 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:37.885 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:37.945 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:50:37.973 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709574637.848220_175.1","last_update":"2024-03-04T17:50:37.848220Z"}] 2024-03-04T17:50:37.973 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574637.848220_175.1'} 2024-03-04T17:50:37.973 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:50:37.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:37.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:37.974 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:38.014 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:38.015 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:38.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:38.071+0000 7fa33942c980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:38.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:38.071+0000 7fa33942c980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:38.094 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574637.816001_170.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:50:37.816001Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:38Z"} 2024-03-04T17:50:38.095 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574637.816001_170.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:50:37.816001Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:38Z"} 2024-03-04T17:50:38.095 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574637.816001_170.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:50:37.816001Z', 'realm_epoch': 2}}] 2024-03-04T17:50:38.095 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574637.848220_175.1'} 2024-03-04T17:50:38.095 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574637.816001_170.1'} 2024-03-04T17:50:38.095 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709574637.848220_175.1 target=1_1709574637.816001_170.1 2024-03-04T17:50:43.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:43.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:43.097 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:43.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:43.199+0000 7fe3a47c4980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:43.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:43.199+0000 7fe3a47c4980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:43.221 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574637.848220_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:50:37.848220Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:43Z"} 2024-03-04T17:50:43.221 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574637.848220_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:50:37.848220Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:43Z"} 2024-03-04T17:50:43.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574637.848220_175.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:50:37.848220Z', 'realm_epoch': 2}}] 2024-03-04T17:50:43.222 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574637.848220_175.1'} 2024-03-04T17:50:43.222 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574637.848220_175.1'} 2024-03-04T17:50:43.222 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:50:43.222 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:50:43.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:43.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:43.223 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:43.265 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:43.265 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:43.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:43.324+0000 7f9d42264980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:43.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:43.324+0000 7f9d42264980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:43.343 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574637.848220_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:50:37.848220Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:43Z"} 2024-03-04T17:50:43.343 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574637.848220_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:50:37.848220Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:43Z"} 2024-03-04T17:50:43.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574637.848220_175.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:50:37.848220Z', 'realm_epoch': 2}}] 2024-03-04T17:50:43.344 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574637.848220_175.1'} 2024-03-04T17:50:43.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574637.848220_175.1'} 2024-03-04T17:50:43.344 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:50:43.344 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:50:43.344 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:50:43.344 DEBUG:boto:Method: HEAD 2024-03-04T17:50:43.344 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:50:43.344 DEBUG:boto:Data: 2024-03-04T17:50:43.345 DEBUG:boto:Headers: {} 2024-03-04T17:50:43.345 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:43.345 DEBUG:boto:Port: 8000 2024-03-04T17:50:43.345 DEBUG:boto:Params: {} 2024-03-04T17:50:43.345 DEBUG:boto:Token: None 2024-03-04T17:50:43.345 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:50:43 GMT /ppfahn-22/ 2024-03-04T17:50:43.345 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XZwVItZw5UlIlvxk3qBPhwfmyuE= 2024-03-04T17:50:43.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:50:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XZwVItZw5UlIlvxk3qBPhwfmyuE=', 'Content-Length': '0'} 2024-03-04T17:50:43.348 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c6cd621a90b13b3-0065e609f3-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:43.348 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:50:43.348 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:50:43.348 DEBUG:boto:Method: PUT 2024-03-04T17:50:43.348 DEBUG:boto:Path: /ppfahn-22/dummy 2024-03-04T17:50:43.349 DEBUG:boto:Data: 2024-03-04T17:50:43.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T17:50:43.349 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:50:43.349 DEBUG:boto:Port: 8000 2024-03-04T17:50:43.349 DEBUG:boto:Params: {} 2024-03-04T17:50:43.349 DEBUG:boto:Token: None 2024-03-04T17:50:43.349 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 17:50:43 GMT /ppfahn-22/dummy 2024-03-04T17:50:43.349 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M5mYTgJXGziV4ddpEZYCU4ur4DI= 2024-03-04T17:50:43.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:50:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M5mYTgJXGziV4ddpEZYCU4ur4DI='} 2024-03-04T17:50:43.361 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-amz-request-id', 'tx000007776dd83dbb856f0-0065e609f3-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:50:43.361 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-22 2024-03-04T17:50:43.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:43.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:43.361 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:43.413 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:43.413 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:43.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:43.475+0000 7fa95468c980 1 waiting to reach incremental sync.. 2024-03-04T17:50:48.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:48.483+0000 7fa95468c980 1 waiting to reach incremental sync.. 2024-03-04T17:50:53.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:53.483+0000 7fa95468c980 1 bucket sync caught up with source: 2024-03-04T17:50:53.481 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000002.37.11, , , , , ] 2024-03-04T17:50:53.481 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000002.37.11, , , , , ] 2024-03-04T17:50:53.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:50:53.483+0000 7fa95468c980 0 bucket checkpoint complete 2024-03-04T17:50:53.521 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-22 2024-03-04T17:50:53.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:53.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:53.521 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:53.572 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:53.572 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:53.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:53.636+0000 7fe34c7e8980 1 bucket sync caught up with source: 2024-03-04T17:50:53.633 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000002.37.11, , , , , ] 2024-03-04T17:50:53.633 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000002.37.11, , , , , ] 2024-03-04T17:50:53.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:53.636+0000 7fe34c7e8980 0 bucket checkpoint complete 2024-03-04T17:50:53.655 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-22 2024-03-04T17:50:53.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:53.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:50:53.655 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:50:53.696 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:53.696 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:53.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:53.760+0000 7f95e6766980 1 waiting to reach incremental sync.. 2024-03-04T17:50:58.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:50:58.760+0000 7f95e6766980 1 waiting to reach incremental sync.. 2024-03-04T17:51:03.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:03.760+0000 7f95e6766980 1 waiting to reach incremental sync.. 2024-03-04T17:51:08.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:08.764+0000 7f95e6766980 1 bucket sync caught up with source: 2024-03-04T17:51:08.761 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000003.27.10, , , , , ] 2024-03-04T17:51:08.761 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000002.25.11, , , , , ] 2024-03-04T17:51:08.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:08.764+0000 7f95e6766980 0 bucket checkpoint complete 2024-03-04T17:51:08.804 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-22 2024-03-04T17:51:08.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:51:08.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:51:08.805 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:51:08.895 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:08.895 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:08.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:08.960+0000 7fc405355980 1 waiting to reach incremental sync.. 2024-03-04T17:51:13.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:13.963+0000 7fc405355980 1 bucket sync caught up with source: 2024-03-04T17:51:13.961 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000003.27.10, , , , , ] 2024-03-04T17:51:13.961 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000003.27.10, , , , , ] 2024-03-04T17:51:13.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:13.963+0000 7fc405355980 0 bucket checkpoint complete 2024-03-04T17:51:14.006 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-22 2024-03-04T17:51:14.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:51:14.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:51:14.007 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:51:14.055 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:14.055 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:14.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:14.119+0000 7fbabd953980 1 waiting for incremental sync to catch up: 2024-03-04T17:51:14.115 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000002.51.11, , , , , ] 2024-03-04T17:51:14.115 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000003.53.10, , , , , ] 2024-03-04T17:51:19.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:19.119+0000 7fbabd953980 1 waiting for incremental sync to catch up: 2024-03-04T17:51:19.118 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000002.51.11, , , , , ] 2024-03-04T17:51:19.118 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000003.53.10, , , , , ] 2024-03-04T17:51:24.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:24.123+0000 7fbabd953980 1 waiting for incremental sync to catch up: 2024-03-04T17:51:24.120 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000002.51.11, , , , , ] 2024-03-04T17:51:24.120 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000003.53.10, , , , , ] 2024-03-04T17:51:29.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:29.127+0000 7fbabd953980 1 bucket sync caught up with source: 2024-03-04T17:51:29.123 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000003.53.10, , , , , ] 2024-03-04T17:51:29.123 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000003.53.10, , , , , ] 2024-03-04T17:51:29.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:29.127+0000 7fbabd953980 0 bucket checkpoint complete 2024-03-04T17:51:29.160 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-22 2024-03-04T17:51:29.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:51:29.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:51:29.160 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:51:29.210 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:29.211 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:51:29.278+0000 7ff9eebc5980 1 bucket sync caught up with source: 2024-03-04T17:51:29.273 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000003.53.10, , , , , ] 2024-03-04T17:51:29.273 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000003.53.10, , , , , ] 2024-03-04T17:51:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:51:29.278+0000 7ff9eebc5980 0 bucket checkpoint complete 2024-03-04T17:51:29.295 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-22 zones={test-zone2, test-zone1} 2024-03-04T17:51:29.295 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.295 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.295 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.296 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:29.296 DEBUG:boto:Data: 2024-03-04T17:51:29.296 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.296 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.296 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.296 DEBUG:boto:Params: {} 2024-03-04T17:51:29.296 DEBUG:boto:Token: None 2024-03-04T17:51:29.297 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.297 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sUF+p9DyDIr6sc0nt6GYaUpc1h0= 2024-03-04T17:51:29.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sUF+p9DyDIr6sc0nt6GYaUpc1h0=', 'Content-Length': '0'} 2024-03-04T17:51:29.301 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006baa677bd2381486-0065e60a21-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.301 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.301 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.301 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.302 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:29.302 DEBUG:boto:Data: 2024-03-04T17:51:29.302 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.302 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.302 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.302 DEBUG:boto:Params: {} 2024-03-04T17:51:29.302 DEBUG:boto:Token: None 2024-03-04T17:51:29.302 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.302 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sUF+p9DyDIr6sc0nt6GYaUpc1h0= 2024-03-04T17:51:29.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sUF+p9DyDIr6sc0nt6GYaUpc1h0=', 'Content-Length': '0'} 2024-03-04T17:51:29.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000324cf530ebb7aa0e-0065e60a21-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.307 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:51:29.307 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.307 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.307 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.307 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.307 DEBUG:boto:Method: GET 2024-03-04T17:51:29.307 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.308 DEBUG:boto:Data: 2024-03-04T17:51:29.308 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.308 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.308 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.308 DEBUG:boto:Params: {} 2024-03-04T17:51:29.308 DEBUG:boto:Token: None 2024-03-04T17:51:29.308 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/?versions 2024-03-04T17:51:29.308 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI= 2024-03-04T17:51:29.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI=', 'Content-Length': '0'} 2024-03-04T17:51:29.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038f00b8fcb4ec553-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.311 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.312 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:51:29.312 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:51:29.312 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.312 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.313 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.313 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.313 DEBUG:boto:Method: GET 2024-03-04T17:51:29.313 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.313 DEBUG:boto:Data: 2024-03-04T17:51:29.313 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.313 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.313 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.314 DEBUG:boto:Params: {} 2024-03-04T17:51:29.314 DEBUG:boto:Token: None 2024-03-04T17:51:29.314 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/?versions 2024-03-04T17:51:29.314 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI= 2024-03-04T17:51:29.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI=', 'Content-Length': '0'} 2024-03-04T17:51:29.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093b92e8af65af83f-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.360 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.361 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:51:29.361 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.361 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.361 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.361 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.361 DEBUG:boto:Method: GET 2024-03-04T17:51:29.361 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.361 DEBUG:boto:Data: 2024-03-04T17:51:29.361 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.362 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.362 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.362 DEBUG:boto:Params: {} 2024-03-04T17:51:29.362 DEBUG:boto:Token: None 2024-03-04T17:51:29.362 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/?versions 2024-03-04T17:51:29.362 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI= 2024-03-04T17:51:29.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI=', 'Content-Length': '0'} 2024-03-04T17:51:29.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009bf93bbd9fa33fe-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.366 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.366 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.367 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.367 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.367 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.367 DEBUG:boto:Method: GET 2024-03-04T17:51:29.367 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.367 DEBUG:boto:Data: 2024-03-04T17:51:29.367 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.367 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.367 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.367 DEBUG:boto:Params: {} 2024-03-04T17:51:29.368 DEBUG:boto:Token: None 2024-03-04T17:51:29.368 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/?versions 2024-03-04T17:51:29.368 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI= 2024-03-04T17:51:29.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI=', 'Content-Length': '0'} 2024-03-04T17:51:29.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003415e279c1389dfd-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.373 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:51:29.373 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.373 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.373 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.373 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.373 DEBUG:boto:Method: GET 2024-03-04T17:51:29.374 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.374 DEBUG:boto:Data: 2024-03-04T17:51:29.374 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.374 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.374 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.374 DEBUG:boto:Params: {} 2024-03-04T17:51:29.374 DEBUG:boto:Token: None 2024-03-04T17:51:29.374 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.374 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc47a6e7b7221771-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.377 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.377 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.378 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.378 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.378 DEBUG:boto:Method: GET 2024-03-04T17:51:29.378 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.378 DEBUG:boto:Data: 2024-03-04T17:51:29.378 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.379 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.379 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.379 DEBUG:boto:Params: {} 2024-03-04T17:51:29.379 DEBUG:boto:Token: None 2024-03-04T17:51:29.379 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.379 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077f505fd8ee63710-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.382 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.382 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.383 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.383 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.383 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.383 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.383 DEBUG:boto:Data: 2024-03-04T17:51:29.383 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.383 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.383 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.383 DEBUG:boto:Params: {} 2024-03-04T17:51:29.383 DEBUG:boto:Token: None 2024-03-04T17:51:29.384 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.384 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vh5BNx+liyojrp/buThf7u0V2GU= 2024-03-04T17:51:29.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vh5BNx+liyojrp/buThf7u0V2GU=', 'Content-Length': '0'} 2024-03-04T17:51:29.387 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d4e00ac62e4a1bb9-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.387 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.387 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.387 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.387 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.387 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.387 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.388 DEBUG:boto:Data: 2024-03-04T17:51:29.388 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.388 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.388 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.388 DEBUG:boto:Params: {} 2024-03-04T17:51:29.388 DEBUG:boto:Token: None 2024-03-04T17:51:29.388 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.388 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vh5BNx+liyojrp/buThf7u0V2GU= 2024-03-04T17:51:29.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vh5BNx+liyojrp/buThf7u0V2GU=', 'Content-Length': '0'} 2024-03-04T17:51:29.391 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092de21a39f6ec5f3-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:51:29.391 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.391 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.391 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.391 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.392 DEBUG:boto:Method: GET 2024-03-04T17:51:29.392 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.392 DEBUG:boto:Data: 2024-03-04T17:51:29.392 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.392 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.392 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.392 DEBUG:boto:Params: {} 2024-03-04T17:51:29.392 DEBUG:boto:Token: None 2024-03-04T17:51:29.392 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.393 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.395 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e86351467f505262-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.395 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.396 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.396 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.396 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.396 DEBUG:boto:Method: GET 2024-03-04T17:51:29.396 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.396 DEBUG:boto:Data: 2024-03-04T17:51:29.396 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.396 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.396 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.396 DEBUG:boto:Params: {} 2024-03-04T17:51:29.397 DEBUG:boto:Token: None 2024-03-04T17:51:29.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.397 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.399 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af8ba10431c45868-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.400 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.400 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.400 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.400 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.400 DEBUG:boto:Method: GET 2024-03-04T17:51:29.400 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.400 DEBUG:boto:Data: 2024-03-04T17:51:29.400 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.400 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.400 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.401 DEBUG:boto:Params: {} 2024-03-04T17:51:29.401 DEBUG:boto:Token: None 2024-03-04T17:51:29.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.401 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM= 2024-03-04T17:51:29.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM=', 'Content-Length': '0'} 2024-03-04T17:51:29.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb692bdb9ddb6115-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.448 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.449 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.449 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.449 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.449 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.450 DEBUG:boto:Method: GET 2024-03-04T17:51:29.450 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.450 DEBUG:boto:Data: 2024-03-04T17:51:29.450 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.450 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.451 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.451 DEBUG:boto:Params: {} 2024-03-04T17:51:29.451 DEBUG:boto:Token: None 2024-03-04T17:51:29.451 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.452 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM= 2024-03-04T17:51:29.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM=', 'Content-Length': '0'} 2024-03-04T17:51:29.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f73a81a6e745bf00-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.457 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.457 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.458 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.458 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.458 DEBUG:boto:Path: /ppfahn-22/dummy 2024-03-04T17:51:29.458 DEBUG:boto:Data: 2024-03-04T17:51:29.458 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.458 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.458 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.458 DEBUG:boto:Params: {} 2024-03-04T17:51:29.458 DEBUG:boto:Token: None 2024-03-04T17:51:29.458 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy 2024-03-04T17:51:29.459 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f2qs5zU0evDR6zOXl8vfkLN5cXU= 2024-03-04T17:51:29.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f2qs5zU0evDR6zOXl8vfkLN5cXU=', 'Content-Length': '0'} 2024-03-04T17:51:29.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af675f7f99c29a76-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.463 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.463 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.463 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.463 DEBUG:boto:Path: /ppfahn-22/dummy 2024-03-04T17:51:29.463 DEBUG:boto:Data: 2024-03-04T17:51:29.463 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.463 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.463 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.463 DEBUG:boto:Params: {} 2024-03-04T17:51:29.464 DEBUG:boto:Token: None 2024-03-04T17:51:29.464 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy 2024-03-04T17:51:29.464 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f2qs5zU0evDR6zOXl8vfkLN5cXU= 2024-03-04T17:51:29.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f2qs5zU0evDR6zOXl8vfkLN5cXU=', 'Content-Length': '0'} 2024-03-04T17:51:29.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d75c199461ce955-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:51:29.468 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.468 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.468 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.468 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.468 DEBUG:boto:Method: GET 2024-03-04T17:51:29.468 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.468 DEBUG:boto:Data: 2024-03-04T17:51:29.469 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.469 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.469 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.469 DEBUG:boto:Params: {} 2024-03-04T17:51:29.469 DEBUG:boto:Token: None 2024-03-04T17:51:29.469 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.469 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004780dffc9de0446d-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.472 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.472 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.472 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.473 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.473 DEBUG:boto:Method: GET 2024-03-04T17:51:29.473 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.473 DEBUG:boto:Data: 2024-03-04T17:51:29.473 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.473 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.473 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.473 DEBUG:boto:Params: {} 2024-03-04T17:51:29.473 DEBUG:boto:Token: None 2024-03-04T17:51:29.473 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.474 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef5849addd3d1df0-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.477 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.477 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.477 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.477 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.477 DEBUG:boto:Method: GET 2024-03-04T17:51:29.477 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.477 DEBUG:boto:Data: 2024-03-04T17:51:29.477 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.477 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.477 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.478 DEBUG:boto:Params: {} 2024-03-04T17:51:29.478 DEBUG:boto:Token: None 2024-03-04T17:51:29.478 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.478 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM= 2024-03-04T17:51:29.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM=', 'Content-Length': '0'} 2024-03-04T17:51:29.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bc11cdf179977b2-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.528 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.528 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.529 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.529 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.529 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.529 DEBUG:boto:Method: GET 2024-03-04T17:51:29.529 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.529 DEBUG:boto:Data: 2024-03-04T17:51:29.529 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.529 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.529 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.529 DEBUG:boto:Params: {} 2024-03-04T17:51:29.529 DEBUG:boto:Token: None 2024-03-04T17:51:29.530 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.530 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM= 2024-03-04T17:51:29.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM=', 'Content-Length': '0'} 2024-03-04T17:51:29.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009efc6af79fb6424f-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.535 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-22 zones={test-zone2, test-zone1} 2024-03-04T17:51:29.535 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-22 zones={test-zone3, test-zone1} 2024-03-04T17:51:29.535 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.535 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.535 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.535 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:29.536 DEBUG:boto:Data: 2024-03-04T17:51:29.536 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.536 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.536 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.536 DEBUG:boto:Params: {} 2024-03-04T17:51:29.536 DEBUG:boto:Token: None 2024-03-04T17:51:29.536 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.537 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sUF+p9DyDIr6sc0nt6GYaUpc1h0= 2024-03-04T17:51:29.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sUF+p9DyDIr6sc0nt6GYaUpc1h0=', 'Content-Length': '0'} 2024-03-04T17:51:29.541 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d9f40740e05a5ce-0065e60a21-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.541 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.541 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.542 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.542 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:29.542 DEBUG:boto:Data: 2024-03-04T17:51:29.542 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.542 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.542 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.542 DEBUG:boto:Params: {} 2024-03-04T17:51:29.542 DEBUG:boto:Token: None 2024-03-04T17:51:29.542 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.542 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sUF+p9DyDIr6sc0nt6GYaUpc1h0= 2024-03-04T17:51:29.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sUF+p9DyDIr6sc0nt6GYaUpc1h0=', 'Content-Length': '0'} 2024-03-04T17:51:29.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b64ae0cf02b5481-0065e60a21-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.547 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:51:29.547 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.547 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.547 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.547 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.547 DEBUG:boto:Method: GET 2024-03-04T17:51:29.547 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.547 DEBUG:boto:Data: 2024-03-04T17:51:29.547 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.547 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.548 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.548 DEBUG:boto:Params: {} 2024-03-04T17:51:29.548 DEBUG:boto:Token: None 2024-03-04T17:51:29.548 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/?versions 2024-03-04T17:51:29.548 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI= 2024-03-04T17:51:29.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI=', 'Content-Length': '0'} 2024-03-04T17:51:29.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d71a8218c19a62-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.553 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.554 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:51:29.554 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:51:29.554 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.554 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.554 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.554 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.555 DEBUG:boto:Method: GET 2024-03-04T17:51:29.555 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.555 DEBUG:boto:Data: 2024-03-04T17:51:29.555 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.555 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.555 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.555 DEBUG:boto:Params: {} 2024-03-04T17:51:29.555 DEBUG:boto:Token: None 2024-03-04T17:51:29.555 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/?versions 2024-03-04T17:51:29.556 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI= 2024-03-04T17:51:29.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI=', 'Content-Length': '0'} 2024-03-04T17:51:29.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffb40f490de2e02b-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.604 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.604 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:51:29.605 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.605 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.605 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.605 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.605 DEBUG:boto:Method: GET 2024-03-04T17:51:29.605 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.605 DEBUG:boto:Data: 2024-03-04T17:51:29.605 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.605 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.605 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.605 DEBUG:boto:Params: {} 2024-03-04T17:51:29.606 DEBUG:boto:Token: None 2024-03-04T17:51:29.606 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/?versions 2024-03-04T17:51:29.606 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI= 2024-03-04T17:51:29.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI=', 'Content-Length': '0'} 2024-03-04T17:51:29.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af0afa4ab74339c5-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.611 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.612 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.612 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.612 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.612 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.612 DEBUG:boto:Method: GET 2024-03-04T17:51:29.612 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.612 DEBUG:boto:Data: 2024-03-04T17:51:29.612 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.612 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.612 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.613 DEBUG:boto:Params: {} 2024-03-04T17:51:29.613 DEBUG:boto:Token: None 2024-03-04T17:51:29.613 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/?versions 2024-03-04T17:51:29.613 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI= 2024-03-04T17:51:29.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI=', 'Content-Length': '0'} 2024-03-04T17:51:29.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f22ed97cfad99faf-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.618 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:51:29.618 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.619 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.619 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.619 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.619 DEBUG:boto:Method: GET 2024-03-04T17:51:29.619 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.619 DEBUG:boto:Data: 2024-03-04T17:51:29.619 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.619 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.619 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.619 DEBUG:boto:Params: {} 2024-03-04T17:51:29.620 DEBUG:boto:Token: None 2024-03-04T17:51:29.620 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.620 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000243c1d5d3c67bc58-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.623 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.623 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.623 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.623 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.623 DEBUG:boto:Method: GET 2024-03-04T17:51:29.623 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.624 DEBUG:boto:Data: 2024-03-04T17:51:29.624 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.624 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.624 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.624 DEBUG:boto:Params: {} 2024-03-04T17:51:29.624 DEBUG:boto:Token: None 2024-03-04T17:51:29.624 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.624 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003d35bbb5b09726b1-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.628 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.628 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.628 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.628 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.628 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.628 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.628 DEBUG:boto:Data: 2024-03-04T17:51:29.628 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.629 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.629 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.629 DEBUG:boto:Params: {} 2024-03-04T17:51:29.629 DEBUG:boto:Token: None 2024-03-04T17:51:29.629 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.629 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vh5BNx+liyojrp/buThf7u0V2GU= 2024-03-04T17:51:29.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vh5BNx+liyojrp/buThf7u0V2GU=', 'Content-Length': '0'} 2024-03-04T17:51:29.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000abb66fe17dff7901-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.632 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.632 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.632 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.632 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.632 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.632 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.633 DEBUG:boto:Data: 2024-03-04T17:51:29.633 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.633 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.633 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.633 DEBUG:boto:Params: {} 2024-03-04T17:51:29.633 DEBUG:boto:Token: None 2024-03-04T17:51:29.633 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.633 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vh5BNx+liyojrp/buThf7u0V2GU= 2024-03-04T17:51:29.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vh5BNx+liyojrp/buThf7u0V2GU=', 'Content-Length': '0'} 2024-03-04T17:51:29.636 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000081e79b369162120b-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:51:29.636 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.637 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.637 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.637 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.637 DEBUG:boto:Method: GET 2024-03-04T17:51:29.637 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.637 DEBUG:boto:Data: 2024-03-04T17:51:29.637 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.637 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.637 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.637 DEBUG:boto:Params: {} 2024-03-04T17:51:29.637 DEBUG:boto:Token: None 2024-03-04T17:51:29.638 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.638 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.640 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000838af95823c66de8-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.641 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.641 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.641 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.641 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.641 DEBUG:boto:Method: GET 2024-03-04T17:51:29.641 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.641 DEBUG:boto:Data: 2024-03-04T17:51:29.641 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.641 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.642 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.642 DEBUG:boto:Params: {} 2024-03-04T17:51:29.642 DEBUG:boto:Token: None 2024-03-04T17:51:29.642 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.642 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.645 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7b38aceb9f17a94-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.645 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.645 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.645 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.645 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.645 DEBUG:boto:Method: GET 2024-03-04T17:51:29.645 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.646 DEBUG:boto:Data: 2024-03-04T17:51:29.646 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.646 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.646 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.646 DEBUG:boto:Params: {} 2024-03-04T17:51:29.646 DEBUG:boto:Token: None 2024-03-04T17:51:29.646 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.646 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM= 2024-03-04T17:51:29.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM=', 'Content-Length': '0'} 2024-03-04T17:51:29.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abc5690c2bd43ed9-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.692 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.692 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.693 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.693 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.693 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.693 DEBUG:boto:Method: GET 2024-03-04T17:51:29.693 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.694 DEBUG:boto:Data: 2024-03-04T17:51:29.694 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.694 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.694 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.694 DEBUG:boto:Params: {} 2024-03-04T17:51:29.694 DEBUG:boto:Token: None 2024-03-04T17:51:29.694 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.695 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM= 2024-03-04T17:51:29.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM=', 'Content-Length': '0'} 2024-03-04T17:51:29.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000806885e41abcaf29-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.700 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.700 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.701 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.701 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.701 DEBUG:boto:Path: /ppfahn-22/dummy 2024-03-04T17:51:29.701 DEBUG:boto:Data: 2024-03-04T17:51:29.701 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.701 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.701 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.702 DEBUG:boto:Params: {} 2024-03-04T17:51:29.702 DEBUG:boto:Token: None 2024-03-04T17:51:29.702 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy 2024-03-04T17:51:29.702 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f2qs5zU0evDR6zOXl8vfkLN5cXU= 2024-03-04T17:51:29.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f2qs5zU0evDR6zOXl8vfkLN5cXU=', 'Content-Length': '0'} 2024-03-04T17:51:29.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008884731ebd3439e3-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.706 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.706 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.707 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.707 DEBUG:boto:Path: /ppfahn-22/dummy 2024-03-04T17:51:29.707 DEBUG:boto:Data: 2024-03-04T17:51:29.707 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.707 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.707 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.707 DEBUG:boto:Params: {} 2024-03-04T17:51:29.707 DEBUG:boto:Token: None 2024-03-04T17:51:29.707 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy 2024-03-04T17:51:29.708 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f2qs5zU0evDR6zOXl8vfkLN5cXU= 2024-03-04T17:51:29.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f2qs5zU0evDR6zOXl8vfkLN5cXU=', 'Content-Length': '0'} 2024-03-04T17:51:29.712 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000acc941686ea82021-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:51:29.713 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.713 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.713 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.713 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.713 DEBUG:boto:Method: GET 2024-03-04T17:51:29.713 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.713 DEBUG:boto:Data: 2024-03-04T17:51:29.713 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.714 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.714 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.714 DEBUG:boto:Params: {} 2024-03-04T17:51:29.714 DEBUG:boto:Token: None 2024-03-04T17:51:29.714 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.714 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cd1369a036accb17-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.718 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.718 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.718 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.718 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.718 DEBUG:boto:Method: GET 2024-03-04T17:51:29.718 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.718 DEBUG:boto:Data: 2024-03-04T17:51:29.719 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.719 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.719 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.719 DEBUG:boto:Params: {} 2024-03-04T17:51:29.719 DEBUG:boto:Token: None 2024-03-04T17:51:29.719 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.719 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004fc01f80dda976b0-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.722 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.723 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.723 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.723 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.723 DEBUG:boto:Method: GET 2024-03-04T17:51:29.723 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.723 DEBUG:boto:Data: 2024-03-04T17:51:29.723 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.723 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.723 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.723 DEBUG:boto:Params: {} 2024-03-04T17:51:29.724 DEBUG:boto:Token: None 2024-03-04T17:51:29.724 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.724 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM= 2024-03-04T17:51:29.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM=', 'Content-Length': '0'} 2024-03-04T17:51:29.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ad4c153206df9cf-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.772 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.772 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.772 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.773 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.773 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.773 DEBUG:boto:Method: GET 2024-03-04T17:51:29.773 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.773 DEBUG:boto:Data: 2024-03-04T17:51:29.773 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.773 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:29.773 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.773 DEBUG:boto:Params: {} 2024-03-04T17:51:29.773 DEBUG:boto:Token: None 2024-03-04T17:51:29.774 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.774 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM= 2024-03-04T17:51:29.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM=', 'Content-Length': '0'} 2024-03-04T17:51:29.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7481f08e2bd5cca-0065e60a21-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.779 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.779 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-22 zones={test-zone3, test-zone1} 2024-03-04T17:51:29.779 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-22 zones={test-zone3, test-zone2} 2024-03-04T17:51:29.779 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.779 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.780 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.780 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:29.780 DEBUG:boto:Data: 2024-03-04T17:51:29.780 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.780 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.780 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.780 DEBUG:boto:Params: {} 2024-03-04T17:51:29.780 DEBUG:boto:Token: None 2024-03-04T17:51:29.780 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.780 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sUF+p9DyDIr6sc0nt6GYaUpc1h0= 2024-03-04T17:51:29.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sUF+p9DyDIr6sc0nt6GYaUpc1h0=', 'Content-Length': '0'} 2024-03-04T17:51:29.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e30e4c777346ba6e-0065e60a21-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.785 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.785 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.785 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.785 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:29.785 DEBUG:boto:Data: 2024-03-04T17:51:29.785 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.786 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.786 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.786 DEBUG:boto:Params: {} 2024-03-04T17:51:29.786 DEBUG:boto:Token: None 2024-03-04T17:51:29.786 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.786 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sUF+p9DyDIr6sc0nt6GYaUpc1h0= 2024-03-04T17:51:29.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sUF+p9DyDIr6sc0nt6GYaUpc1h0=', 'Content-Length': '0'} 2024-03-04T17:51:29.790 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ac61f7a3f623d07-0065e60a21-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.790 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:51:29.790 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.790 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.790 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.791 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.791 DEBUG:boto:Method: GET 2024-03-04T17:51:29.791 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.791 DEBUG:boto:Data: 2024-03-04T17:51:29.791 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.791 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.791 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.791 DEBUG:boto:Params: {} 2024-03-04T17:51:29.791 DEBUG:boto:Token: None 2024-03-04T17:51:29.791 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/?versions 2024-03-04T17:51:29.792 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI= 2024-03-04T17:51:29.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI=', 'Content-Length': '0'} 2024-03-04T17:51:29.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfd048cac77861be-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.796 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.797 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:51:29.797 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:51:29.797 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.797 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.797 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.797 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.797 DEBUG:boto:Method: GET 2024-03-04T17:51:29.797 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.797 DEBUG:boto:Data: 2024-03-04T17:51:29.798 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.798 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.798 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.798 DEBUG:boto:Params: {} 2024-03-04T17:51:29.798 DEBUG:boto:Token: None 2024-03-04T17:51:29.798 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/?versions 2024-03-04T17:51:29.798 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI= 2024-03-04T17:51:29.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI=', 'Content-Length': '0'} 2024-03-04T17:51:29.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007687f29e5474ff24-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.803 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.803 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:51:29.803 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.803 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.803 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.803 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.804 DEBUG:boto:Method: GET 2024-03-04T17:51:29.804 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.804 DEBUG:boto:Data: 2024-03-04T17:51:29.804 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.804 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.804 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.804 DEBUG:boto:Params: {} 2024-03-04T17:51:29.804 DEBUG:boto:Token: None 2024-03-04T17:51:29.804 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/?versions 2024-03-04T17:51:29.805 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI= 2024-03-04T17:51:29.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI=', 'Content-Length': '0'} 2024-03-04T17:51:29.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039b7981ea1ea9476-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.852 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.852 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.853 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.853 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.853 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.853 DEBUG:boto:Method: GET 2024-03-04T17:51:29.853 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:51:29.853 DEBUG:boto:Data: 2024-03-04T17:51:29.853 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.853 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.853 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.853 DEBUG:boto:Params: {} 2024-03-04T17:51:29.853 DEBUG:boto:Token: None 2024-03-04T17:51:29.854 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/?versions 2024-03-04T17:51:29.854 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI= 2024-03-04T17:51:29.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TSdN9Mv50a57xHy8mZCS6ON1WYI=', 'Content-Length': '0'} 2024-03-04T17:51:29.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ee83ef3a75b2b03-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.859 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:51:29.859 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.859 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.860 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.860 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.860 DEBUG:boto:Method: GET 2024-03-04T17:51:29.860 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.860 DEBUG:boto:Data: 2024-03-04T17:51:29.860 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.860 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.860 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.860 DEBUG:boto:Params: {} 2024-03-04T17:51:29.860 DEBUG:boto:Token: None 2024-03-04T17:51:29.861 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.861 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002cc141e954983b33-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.864 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.864 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.864 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.864 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.864 DEBUG:boto:Method: GET 2024-03-04T17:51:29.864 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.865 DEBUG:boto:Data: 2024-03-04T17:51:29.865 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.865 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.865 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.865 DEBUG:boto:Params: {} 2024-03-04T17:51:29.865 DEBUG:boto:Token: None 2024-03-04T17:51:29.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.865 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.868 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c95e86fd946ceef-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.868 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.868 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.869 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.869 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.869 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.869 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.869 DEBUG:boto:Data: 2024-03-04T17:51:29.869 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.869 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.869 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.869 DEBUG:boto:Params: {} 2024-03-04T17:51:29.869 DEBUG:boto:Token: None 2024-03-04T17:51:29.869 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.870 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vh5BNx+liyojrp/buThf7u0V2GU= 2024-03-04T17:51:29.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vh5BNx+liyojrp/buThf7u0V2GU=', 'Content-Length': '0'} 2024-03-04T17:51:29.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fd48cf0b790291c3-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.872 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.873 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.873 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.873 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.873 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.873 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.873 DEBUG:boto:Data: 2024-03-04T17:51:29.873 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.873 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.873 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.873 DEBUG:boto:Params: {} 2024-03-04T17:51:29.874 DEBUG:boto:Token: None 2024-03-04T17:51:29.874 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.874 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vh5BNx+liyojrp/buThf7u0V2GU= 2024-03-04T17:51:29.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vh5BNx+liyojrp/buThf7u0V2GU=', 'Content-Length': '0'} 2024-03-04T17:51:29.877 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f33ce8107a75e2f-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:51:29.877 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.877 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.877 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.877 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.877 DEBUG:boto:Method: GET 2024-03-04T17:51:29.877 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.878 DEBUG:boto:Data: 2024-03-04T17:51:29.878 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.878 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.878 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.878 DEBUG:boto:Params: {} 2024-03-04T17:51:29.878 DEBUG:boto:Token: None 2024-03-04T17:51:29.878 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.878 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000844dc4f70a47a7a6-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.882 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.882 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.882 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.882 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.882 DEBUG:boto:Method: GET 2024-03-04T17:51:29.883 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.883 DEBUG:boto:Data: 2024-03-04T17:51:29.883 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.883 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.883 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.883 DEBUG:boto:Params: {} 2024-03-04T17:51:29.883 DEBUG:boto:Token: None 2024-03-04T17:51:29.883 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.883 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.887 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000064645ee2e1930430-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.887 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.887 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.887 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.888 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.888 DEBUG:boto:Method: GET 2024-03-04T17:51:29.888 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.888 DEBUG:boto:Data: 2024-03-04T17:51:29.888 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.888 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.888 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.888 DEBUG:boto:Params: {} 2024-03-04T17:51:29.888 DEBUG:boto:Token: None 2024-03-04T17:51:29.889 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.889 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM= 2024-03-04T17:51:29.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM=', 'Content-Length': '0'} 2024-03-04T17:51:29.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035e435ccdb802ed5-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.936 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.936 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.937 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.937 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.937 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.937 DEBUG:boto:Method: GET 2024-03-04T17:51:29.937 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.937 DEBUG:boto:Data: 2024-03-04T17:51:29.937 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.937 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.937 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.937 DEBUG:boto:Params: {} 2024-03-04T17:51:29.938 DEBUG:boto:Token: None 2024-03-04T17:51:29.938 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.938 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM= 2024-03-04T17:51:29.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM=', 'Content-Length': '0'} 2024-03-04T17:51:29.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eca3dc7d413577ff-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.943 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:29.943 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.943 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.943 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.943 DEBUG:boto:Path: /ppfahn-22/dummy 2024-03-04T17:51:29.943 DEBUG:boto:Data: 2024-03-04T17:51:29.944 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.944 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.944 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.944 DEBUG:boto:Params: {} 2024-03-04T17:51:29.944 DEBUG:boto:Token: None 2024-03-04T17:51:29.944 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy 2024-03-04T17:51:29.944 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f2qs5zU0evDR6zOXl8vfkLN5cXU= 2024-03-04T17:51:29.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f2qs5zU0evDR6zOXl8vfkLN5cXU=', 'Content-Length': '0'} 2024-03-04T17:51:29.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f315c793696ff69d-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.948 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.948 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.949 DEBUG:boto:Method: HEAD 2024-03-04T17:51:29.949 DEBUG:boto:Path: /ppfahn-22/dummy 2024-03-04T17:51:29.949 DEBUG:boto:Data: 2024-03-04T17:51:29.949 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.949 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.949 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.949 DEBUG:boto:Params: {} 2024-03-04T17:51:29.949 DEBUG:boto:Token: None 2024-03-04T17:51:29.949 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy 2024-03-04T17:51:29.949 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f2qs5zU0evDR6zOXl8vfkLN5cXU= 2024-03-04T17:51:29.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f2qs5zU0evDR6zOXl8vfkLN5cXU=', 'Content-Length': '0'} 2024-03-04T17:51:29.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a65e6a8a67d26360-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:51:29.954 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.954 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.954 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.954 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.954 DEBUG:boto:Method: GET 2024-03-04T17:51:29.954 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.954 DEBUG:boto:Data: 2024-03-04T17:51:29.954 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.955 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.955 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.955 DEBUG:boto:Params: {} 2024-03-04T17:51:29.955 DEBUG:boto:Token: None 2024-03-04T17:51:29.955 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.955 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ceb1275de65ddfac-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.958 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:51:29.959 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:51:29.959 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.959 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.959 DEBUG:boto:Method: GET 2024-03-04T17:51:29.959 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.959 DEBUG:boto:Data: 2024-03-04T17:51:29.959 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.959 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:29.959 DEBUG:boto:Port: 8000 2024-03-04T17:51:29.959 DEBUG:boto:Params: {} 2024-03-04T17:51:29.960 DEBUG:boto:Token: None 2024-03-04T17:51:29.960 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:51:29.960 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc= 2024-03-04T17:51:29.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y44pnQCbtfUMryffAZn7UKSF0Gc=', 'Content-Length': '0'} 2024-03-04T17:51:29.963 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d424678ff38809b7-0065e60a21-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:29.963 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:29.963 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:29.963 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.963 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.963 DEBUG:boto:Method: GET 2024-03-04T17:51:29.964 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:51:29.964 DEBUG:boto:Data: 2024-03-04T17:51:29.964 DEBUG:boto:Headers: {} 2024-03-04T17:51:29.964 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:29.964 DEBUG:boto:Port: 8001 2024-03-04T17:51:29.964 DEBUG:boto:Params: {} 2024-03-04T17:51:29.964 DEBUG:boto:Token: None 2024-03-04T17:51:29.964 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:29 GMT /ppfahn-22/ 2024-03-04T17:51:29.964 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM= 2024-03-04T17:51:29.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GTeBURBsWLhiS9XM63F7kIznVDM=', 'Content-Length': '0'} 2024-03-04T17:51:29.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d05bdc373c4967d-0065e60a21-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.012 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:30.012 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.013 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.013 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:30.013 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:51:30.013 DEBUG:boto:Method: GET 2024-03-04T17:51:30.013 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:51:30.013 DEBUG:boto:Data: 2024-03-04T17:51:30.013 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.013 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.013 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.013 DEBUG:boto:Params: {} 2024-03-04T17:51:30.014 DEBUG:boto:Token: None 2024-03-04T17:51:30.014 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.014 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dKmPZzLbZsHPK4qd6BNCTsDk/8g= 2024-03-04T17:51:30.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dKmPZzLbZsHPK4qd6BNCTsDk/8g=', 'Content-Length': '0'} 2024-03-04T17:51:30.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007388a941d41f71d4-0065e60a22-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.019 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:51:30.019 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-22 zones={test-zone3, test-zone2} 2024-03-04T17:51:30.020 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.020 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.020 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.020 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.020 DEBUG:boto:Data: 2024-03-04T17:51:30.020 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.020 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.020 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.020 DEBUG:boto:Params: {} 2024-03-04T17:51:30.021 DEBUG:boto:Token: None 2024-03-04T17:51:30.021 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.021 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.025 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b8f5f4655340264-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.026 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.026 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.026 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.026 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.026 DEBUG:boto:Data: 2024-03-04T17:51:30.026 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.026 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.026 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.026 DEBUG:boto:Params: {} 2024-03-04T17:51:30.026 DEBUG:boto:Token: None 2024-03-04T17:51:30.027 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.027 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-amz-request-id', 'tx0000095c97e8de803a43b-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.052 DEBUG:rgw_multi.tests:zone=test-zone1 version=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:51:30.052 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.052 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.053 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.053 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.053 DEBUG:boto:Data: 2024-03-04T17:51:30.053 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.053 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.053 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.053 DEBUG:boto:Params: {} 2024-03-04T17:51:30.053 DEBUG:boto:Token: None 2024-03-04T17:51:30.053 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.054 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.058 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002b807c4f52c5262-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.059 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.059 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.059 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.059 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.059 DEBUG:boto:Data: 2024-03-04T17:51:30.059 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.059 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.059 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.060 DEBUG:boto:Params: {} 2024-03-04T17:51:30.060 DEBUG:boto:Token: None 2024-03-04T17:51:30.060 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.060 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-amz-request-id', 'tx00000b37f71fd1a2e2f70-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.108 DEBUG:rgw_multi.tests:zone=test-zone2 version=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:51:30.108 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.109 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.109 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.109 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.109 DEBUG:boto:Data: 2024-03-04T17:51:30.109 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.109 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.109 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.109 DEBUG:boto:Params: {} 2024-03-04T17:51:30.109 DEBUG:boto:Token: None 2024-03-04T17:51:30.109 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.110 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ac535ff47a4ce78-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.115 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.115 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.115 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.115 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.116 DEBUG:boto:Data: 2024-03-04T17:51:30.116 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.116 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.116 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.116 DEBUG:boto:Params: {} 2024-03-04T17:51:30.116 DEBUG:boto:Token: None 2024-03-04T17:51:30.116 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.116 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.141 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-amz-request-id', 'tx0000057c348aecca5f034-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.141 DEBUG:rgw_multi.tests:zone=test-zone3 version=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:51:30.141 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.141 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.141 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.142 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.142 DEBUG:boto:Data: 2024-03-04T17:51:30.142 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.142 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.142 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.142 DEBUG:boto:Params: {} 2024-03-04T17:51:30.142 DEBUG:boto:Token: None 2024-03-04T17:51:30.142 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.142 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.147 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dd07c3eadda4292-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.147 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.147 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.147 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.147 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.147 DEBUG:boto:Data: 2024-03-04T17:51:30.147 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.147 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.148 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.148 DEBUG:boto:Params: {} 2024-03-04T17:51:30.148 DEBUG:boto:Token: None 2024-03-04T17:51:30.148 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.148 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.192 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-amz-request-id', 'tx000005887889b721670da-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.192 DEBUG:rgw_multi.tests:zone=test-zone1 version=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:51:30.192 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.192 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.193 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.193 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.193 DEBUG:boto:Data: 2024-03-04T17:51:30.193 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.193 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.193 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.193 DEBUG:boto:Params: {} 2024-03-04T17:51:30.193 DEBUG:boto:Token: None 2024-03-04T17:51:30.193 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.193 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028d9b397b5d51d7d-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.198 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.198 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.198 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.198 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.198 DEBUG:boto:Data: 2024-03-04T17:51:30.199 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.199 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.199 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.199 DEBUG:boto:Params: {} 2024-03-04T17:51:30.199 DEBUG:boto:Token: None 2024-03-04T17:51:30.199 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.199 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.222 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-amz-request-id', 'tx000002204492706acc076-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.222 DEBUG:rgw_multi.tests:zone=test-zone2 version=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:51:30.222 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.222 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.222 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.223 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.223 DEBUG:boto:Data: 2024-03-04T17:51:30.223 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.223 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.223 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.223 DEBUG:boto:Params: {} 2024-03-04T17:51:30.223 DEBUG:boto:Token: None 2024-03-04T17:51:30.223 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.223 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf30c30337e5fad0-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.228 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.229 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.229 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.229 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.229 DEBUG:boto:Data: 2024-03-04T17:51:30.229 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.229 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.229 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.229 DEBUG:boto:Params: {} 2024-03-04T17:51:30.229 DEBUG:boto:Token: None 2024-03-04T17:51:30.230 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.230 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.251 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-amz-request-id', 'tx00000939af4f17624c541-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.252 DEBUG:rgw_multi.tests:zone=test-zone3 version=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:51:30.252 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.252 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.252 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.252 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.252 DEBUG:boto:Data: 2024-03-04T17:51:30.252 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.252 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.252 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.252 DEBUG:boto:Params: {} 2024-03-04T17:51:30.253 DEBUG:boto:Token: None 2024-03-04T17:51:30.253 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.253 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.257 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005741abded3f02b8f-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.258 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.258 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.258 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.258 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.258 DEBUG:boto:Data: 2024-03-04T17:51:30.258 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.258 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.258 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.258 DEBUG:boto:Params: {} 2024-03-04T17:51:30.258 DEBUG:boto:Token: None 2024-03-04T17:51:30.259 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.259 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.284 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-amz-request-id', 'tx00000761c82fdbb4798d1-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.285 DEBUG:rgw_multi.tests:zone=test-zone1 version=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:51:30.285 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.285 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.285 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.285 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.285 DEBUG:boto:Data: 2024-03-04T17:51:30.285 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.285 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.285 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.286 DEBUG:boto:Params: {} 2024-03-04T17:51:30.286 DEBUG:boto:Token: None 2024-03-04T17:51:30.286 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.286 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b7379fa46ce38c8-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.292 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.292 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.292 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.292 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.293 DEBUG:boto:Data: 2024-03-04T17:51:30.293 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.293 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.293 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.293 DEBUG:boto:Params: {} 2024-03-04T17:51:30.293 DEBUG:boto:Token: None 2024-03-04T17:51:30.293 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.293 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-amz-request-id', 'tx000008dc441d6e86e6a1b-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.315 DEBUG:rgw_multi.tests:zone=test-zone2 version=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:51:30.315 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.315 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.316 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.316 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.316 DEBUG:boto:Data: 2024-03-04T17:51:30.316 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.316 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.316 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.316 DEBUG:boto:Params: {} 2024-03-04T17:51:30.316 DEBUG:boto:Token: None 2024-03-04T17:51:30.316 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.316 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004413ad703f71becc-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.324 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.324 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.325 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.325 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.325 DEBUG:boto:Data: 2024-03-04T17:51:30.325 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.325 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.325 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.325 DEBUG:boto:Params: {} 2024-03-04T17:51:30.325 DEBUG:boto:Token: None 2024-03-04T17:51:30.325 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.326 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-amz-request-id', 'tx000006f0d2e365876f968-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.369 DEBUG:rgw_multi.tests:zone=test-zone3 version=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:51:30.369 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.369 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.369 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.369 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.369 DEBUG:boto:Data: 2024-03-04T17:51:30.369 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.370 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.370 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.370 DEBUG:boto:Params: {} 2024-03-04T17:51:30.370 DEBUG:boto:Token: None 2024-03-04T17:51:30.370 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.370 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d62cb75ccac5040f-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.375 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.375 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.375 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.376 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.376 DEBUG:boto:Data: 2024-03-04T17:51:30.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.376 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.376 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.376 DEBUG:boto:Params: {} 2024-03-04T17:51:30.376 DEBUG:boto:Token: None 2024-03-04T17:51:30.377 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.377 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-amz-request-id', 'tx00000005e166bc0abbf99-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.399 DEBUG:rgw_multi.tests:zone=test-zone1 version=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:51:30.399 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.399 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.399 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.399 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.399 DEBUG:boto:Data: 2024-03-04T17:51:30.400 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.400 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.400 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.400 DEBUG:boto:Params: {} 2024-03-04T17:51:30.400 DEBUG:boto:Token: None 2024-03-04T17:51:30.400 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.400 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8c8842c259c1728-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.405 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.405 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.405 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.406 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.406 DEBUG:boto:Data: 2024-03-04T17:51:30.406 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.406 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.406 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.406 DEBUG:boto:Params: {} 2024-03-04T17:51:30.406 DEBUG:boto:Token: None 2024-03-04T17:51:30.407 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.407 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-amz-request-id', 'tx00000e8ac0963ca8c620e-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.452 DEBUG:rgw_multi.tests:zone=test-zone2 version=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:51:30.452 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.452 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.453 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.453 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.453 DEBUG:boto:Data: 2024-03-04T17:51:30.453 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.453 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.453 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.453 DEBUG:boto:Params: {} 2024-03-04T17:51:30.453 DEBUG:boto:Token: None 2024-03-04T17:51:30.453 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.454 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d68d29c0413047f-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.459 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.459 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.459 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.459 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.459 DEBUG:boto:Data: 2024-03-04T17:51:30.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.459 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.459 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.459 DEBUG:boto:Params: {} 2024-03-04T17:51:30.459 DEBUG:boto:Token: None 2024-03-04T17:51:30.460 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.460 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.483 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-amz-request-id', 'tx0000077c2b99d4347031d-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.483 DEBUG:rgw_multi.tests:zone=test-zone3 version=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:51:30.483 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.484 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.484 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.484 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.484 DEBUG:boto:Data: 2024-03-04T17:51:30.484 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.484 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.484 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.484 DEBUG:boto:Params: {} 2024-03-04T17:51:30.484 DEBUG:boto:Token: None 2024-03-04T17:51:30.485 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.485 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a6fca81f8f1be67-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.489 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.489 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.489 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.489 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.490 DEBUG:boto:Data: 2024-03-04T17:51:30.490 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.490 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.490 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.490 DEBUG:boto:Params: {} 2024-03-04T17:51:30.490 DEBUG:boto:Token: None 2024-03-04T17:51:30.490 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.490 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-amz-request-id', 'tx00000f208fd10901c62ab-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.536 DEBUG:rgw_multi.tests:zone=test-zone1 version=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:51:30.536 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.537 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.537 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.537 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.537 DEBUG:boto:Data: 2024-03-04T17:51:30.537 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.537 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.537 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.537 DEBUG:boto:Params: {} 2024-03-04T17:51:30.537 DEBUG:boto:Token: None 2024-03-04T17:51:30.537 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.538 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd20f8f64af18e10-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.542 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.542 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.542 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.542 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.542 DEBUG:boto:Data: 2024-03-04T17:51:30.542 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.542 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.543 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.543 DEBUG:boto:Params: {} 2024-03-04T17:51:30.543 DEBUG:boto:Token: None 2024-03-04T17:51:30.543 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.543 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.565 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-amz-request-id', 'tx00000e4af2cee3ea48380-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.565 DEBUG:rgw_multi.tests:zone=test-zone2 version=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:51:30.565 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.565 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.565 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.565 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.565 DEBUG:boto:Data: 2024-03-04T17:51:30.566 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.566 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.566 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.566 DEBUG:boto:Params: {} 2024-03-04T17:51:30.566 DEBUG:boto:Token: None 2024-03-04T17:51:30.566 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.566 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bb5896047c72a00-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.571 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.571 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.571 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.571 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.571 DEBUG:boto:Data: 2024-03-04T17:51:30.571 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.571 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.571 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.571 DEBUG:boto:Params: {} 2024-03-04T17:51:30.572 DEBUG:boto:Token: None 2024-03-04T17:51:30.572 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.572 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-amz-request-id', 'tx000007161cd3f524e8f66-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.595 DEBUG:rgw_multi.tests:zone=test-zone3 version=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:51:30.595 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.595 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.595 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.595 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.595 DEBUG:boto:Data: 2024-03-04T17:51:30.595 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.595 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.595 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.595 DEBUG:boto:Params: {} 2024-03-04T17:51:30.596 DEBUG:boto:Token: None 2024-03-04T17:51:30.596 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.596 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.600 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e90572f42569d7af-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.600 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.600 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.600 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.601 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.601 DEBUG:boto:Data: 2024-03-04T17:51:30.601 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.601 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.601 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.601 DEBUG:boto:Params: {} 2024-03-04T17:51:30.601 DEBUG:boto:Token: None 2024-03-04T17:51:30.601 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.602 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-amz-request-id', 'tx0000019fd2cac910b8bab-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.623 DEBUG:rgw_multi.tests:zone=test-zone1 version=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:51:30.623 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.623 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.624 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.624 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.624 DEBUG:boto:Data: 2024-03-04T17:51:30.624 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.624 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.624 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.624 DEBUG:boto:Params: {} 2024-03-04T17:51:30.624 DEBUG:boto:Token: None 2024-03-04T17:51:30.624 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.625 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.629 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095746caf47479e13-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.629 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.629 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.629 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.629 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.629 DEBUG:boto:Data: 2024-03-04T17:51:30.630 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.630 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.630 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.630 DEBUG:boto:Params: {} 2024-03-04T17:51:30.630 DEBUG:boto:Token: None 2024-03-04T17:51:30.630 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.630 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-amz-request-id', 'tx00000c41224d1f619744f-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.653 DEBUG:rgw_multi.tests:zone=test-zone2 version=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:51:30.653 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.653 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.653 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.654 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.654 DEBUG:boto:Data: 2024-03-04T17:51:30.654 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.654 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.654 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.654 DEBUG:boto:Params: {} 2024-03-04T17:51:30.654 DEBUG:boto:Token: None 2024-03-04T17:51:30.654 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.654 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000446ae4a5f9e856c2-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.659 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.659 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.659 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.659 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.659 DEBUG:boto:Data: 2024-03-04T17:51:30.659 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.659 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.659 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.660 DEBUG:boto:Params: {} 2024-03-04T17:51:30.660 DEBUG:boto:Token: None 2024-03-04T17:51:30.660 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.660 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.704 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-amz-request-id', 'tx00000df8bf9f1bf608bbe-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.704 DEBUG:rgw_multi.tests:zone=test-zone3 version=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:51:30.705 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.705 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.705 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.705 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.705 DEBUG:boto:Data: 2024-03-04T17:51:30.705 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.705 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.705 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.705 DEBUG:boto:Params: {} 2024-03-04T17:51:30.706 DEBUG:boto:Token: None 2024-03-04T17:51:30.706 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.706 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000418b440936138f6d-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.711 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.711 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.711 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.711 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.711 DEBUG:boto:Data: 2024-03-04T17:51:30.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.711 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.711 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.711 DEBUG:boto:Params: {} 2024-03-04T17:51:30.711 DEBUG:boto:Token: None 2024-03-04T17:51:30.711 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.712 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-amz-request-id', 'tx00000e27ebeca54483ed9-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.733 DEBUG:rgw_multi.tests:zone=test-zone1 version=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:51:30.734 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.734 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.734 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.734 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.734 DEBUG:boto:Data: 2024-03-04T17:51:30.734 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.734 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.734 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.734 DEBUG:boto:Params: {} 2024-03-04T17:51:30.734 DEBUG:boto:Token: None 2024-03-04T17:51:30.735 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.735 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037848ce0606856a3-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.740 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.740 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.740 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.740 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.740 DEBUG:boto:Data: 2024-03-04T17:51:30.740 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.740 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.740 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.740 DEBUG:boto:Params: {} 2024-03-04T17:51:30.740 DEBUG:boto:Token: None 2024-03-04T17:51:30.741 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.741 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.784 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-amz-request-id', 'tx00000e937a9006595dd00-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.784 DEBUG:rgw_multi.tests:zone=test-zone2 version=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:51:30.785 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.785 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.785 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.785 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.785 DEBUG:boto:Data: 2024-03-04T17:51:30.785 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.785 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.785 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.785 DEBUG:boto:Params: {} 2024-03-04T17:51:30.785 DEBUG:boto:Token: None 2024-03-04T17:51:30.786 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.786 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fb930608f4a11a4-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.791 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.791 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.791 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.791 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.791 DEBUG:boto:Data: 2024-03-04T17:51:30.792 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.792 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.792 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.792 DEBUG:boto:Params: {} 2024-03-04T17:51:30.792 DEBUG:boto:Token: None 2024-03-04T17:51:30.792 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.792 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-amz-request-id', 'tx00000b2e1fb5758f33278-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.814 DEBUG:rgw_multi.tests:zone=test-zone3 version=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:51:30.814 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.814 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.815 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.815 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.815 DEBUG:boto:Data: 2024-03-04T17:51:30.815 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.815 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.815 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.815 DEBUG:boto:Params: {} 2024-03-04T17:51:30.815 DEBUG:boto:Token: None 2024-03-04T17:51:30.815 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.816 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c8ef615c94177d6-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.820 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.820 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.820 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.820 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.820 DEBUG:boto:Data: 2024-03-04T17:51:30.821 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.821 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.821 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.821 DEBUG:boto:Params: {} 2024-03-04T17:51:30.821 DEBUG:boto:Token: None 2024-03-04T17:51:30.821 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.821 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-amz-request-id', 'tx000005f2228ba7ab89d96-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.843 DEBUG:rgw_multi.tests:zone=test-zone1 version=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:51:30.843 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.843 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.843 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.843 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.844 DEBUG:boto:Data: 2024-03-04T17:51:30.844 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.844 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.844 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.844 DEBUG:boto:Params: {} 2024-03-04T17:51:30.844 DEBUG:boto:Token: None 2024-03-04T17:51:30.844 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.844 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.848 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9082bd18acb99a1-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.849 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.849 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.849 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.849 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.849 DEBUG:boto:Data: 2024-03-04T17:51:30.849 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.849 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.849 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.850 DEBUG:boto:Params: {} 2024-03-04T17:51:30.850 DEBUG:boto:Token: None 2024-03-04T17:51:30.850 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.850 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-amz-request-id', 'tx00000624fc7efbe6106d0-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.871 DEBUG:rgw_multi.tests:zone=test-zone2 version=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:51:30.871 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.871 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.871 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.871 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.871 DEBUG:boto:Data: 2024-03-04T17:51:30.871 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.871 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.871 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.871 DEBUG:boto:Params: {} 2024-03-04T17:51:30.872 DEBUG:boto:Token: None 2024-03-04T17:51:30.872 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.872 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ecf207d25816718-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.877 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.877 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.877 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.877 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.877 DEBUG:boto:Data: 2024-03-04T17:51:30.877 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.877 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.877 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.877 DEBUG:boto:Params: {} 2024-03-04T17:51:30.877 DEBUG:boto:Token: None 2024-03-04T17:51:30.878 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.878 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.900 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-amz-request-id', 'tx000000b0b39156f74cdb1-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.900 DEBUG:rgw_multi.tests:zone=test-zone3 version=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:51:30.900 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.900 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.901 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.901 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.901 DEBUG:boto:Data: 2024-03-04T17:51:30.901 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.901 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.901 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.901 DEBUG:boto:Params: {} 2024-03-04T17:51:30.901 DEBUG:boto:Token: None 2024-03-04T17:51:30.901 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.902 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072042eec993a530e-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.906 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.906 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.906 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.907 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.907 DEBUG:boto:Data: 2024-03-04T17:51:30.907 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.907 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.907 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.907 DEBUG:boto:Params: {} 2024-03-04T17:51:30.907 DEBUG:boto:Token: None 2024-03-04T17:51:30.907 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.907 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.929 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-amz-request-id', 'tx000000bac7888a188fe94-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.929 DEBUG:rgw_multi.tests:zone=test-zone1 version=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:51:30.929 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.930 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.930 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.930 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.930 DEBUG:boto:Data: 2024-03-04T17:51:30.930 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.930 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.930 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.930 DEBUG:boto:Params: {} 2024-03-04T17:51:30.930 DEBUG:boto:Token: None 2024-03-04T17:51:30.931 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.931 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.935 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df810f2da616630b-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.936 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.936 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.936 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.936 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.936 DEBUG:boto:Data: 2024-03-04T17:51:30.936 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.936 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:30.937 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.937 DEBUG:boto:Params: {} 2024-03-04T17:51:30.937 DEBUG:boto:Token: None 2024-03-04T17:51:30.937 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.937 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-amz-request-id', 'tx0000074b4bff33a7ad6d7-0065e60a22-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.958 DEBUG:rgw_multi.tests:zone=test-zone2 version=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:51:30.958 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.958 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.958 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.958 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.959 DEBUG:boto:Data: 2024-03-04T17:51:30.959 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.959 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.959 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.959 DEBUG:boto:Params: {} 2024-03-04T17:51:30.959 DEBUG:boto:Token: None 2024-03-04T17:51:30.959 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.959 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.963 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c8384f7ab0fc6ff-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.964 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.964 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.964 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.964 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.964 DEBUG:boto:Data: 2024-03-04T17:51:30.964 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.965 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:30.965 DEBUG:boto:Port: 8001 2024-03-04T17:51:30.965 DEBUG:boto:Params: {} 2024-03-04T17:51:30.965 DEBUG:boto:Token: None 2024-03-04T17:51:30.965 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.965 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:30.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-amz-request-id', 'tx00000f459b34408ad6a6e-0065e60a22-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.987 DEBUG:rgw_multi.tests:zone=test-zone3 version=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:51:30.987 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:30.987 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:30.987 DEBUG:boto:Method: HEAD 2024-03-04T17:51:30.987 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:30.987 DEBUG:boto:Data: 2024-03-04T17:51:30.988 DEBUG:boto:Headers: {} 2024-03-04T17:51:30.988 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.988 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.988 DEBUG:boto:Params: {} 2024-03-04T17:51:30.988 DEBUG:boto:Token: None 2024-03-04T17:51:30.988 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/ 2024-03-04T17:51:30.988 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo= 2024-03-04T17:51:30.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVDy7cNFyH7bj7WzgCJOOTpAvQo=', 'Content-Length': '0'} 2024-03-04T17:51:30.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000436d0f244c481ec-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:30.994 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:30.994 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:30.994 DEBUG:boto:Method: PUT 2024-03-04T17:51:30.994 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:30.994 DEBUG:boto:Data: 2024-03-04T17:51:30.995 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:30.995 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:51:30.995 DEBUG:boto:Port: 8000 2024-03-04T17:51:30.995 DEBUG:boto:Params: {} 2024-03-04T17:51:30.995 DEBUG:boto:Token: None 2024-03-04T17:51:30.996 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:30 GMT /ppfahn-22/obj 2024-03-04T17:51:30.996 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc= 2024-03-04T17:51:30.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yAKB+opm8TQYjsIcuN9UXTQJGjc='} 2024-03-04T17:51:31.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-amz-request-id', 'tx000005baa287091e2e849-0065e60a22-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:31.040 DEBUG:rgw_multi.tests:zone=test-zone1 version=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:51:31.041 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:31.041 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:31.041 DEBUG:boto:Method: HEAD 2024-03-04T17:51:31.041 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:31.041 DEBUG:boto:Data: 2024-03-04T17:51:31.041 DEBUG:boto:Headers: {} 2024-03-04T17:51:31.041 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:31.041 DEBUG:boto:Port: 8000 2024-03-04T17:51:31.041 DEBUG:boto:Params: {} 2024-03-04T17:51:31.041 DEBUG:boto:Token: None 2024-03-04T17:51:31.042 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:31 GMT /ppfahn-22/ 2024-03-04T17:51:31.042 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H6zjTQjWRwQspBu85pCB/pjUnCs= 2024-03-04T17:51:31.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H6zjTQjWRwQspBu85pCB/pjUnCs=', 'Content-Length': '0'} 2024-03-04T17:51:31.046 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a797a3a30ab5c58-0065e60a23-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:31.047 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:31.047 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:31.047 DEBUG:boto:Method: PUT 2024-03-04T17:51:31.047 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:31.047 DEBUG:boto:Data: 2024-03-04T17:51:31.047 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:31.047 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:51:31.048 DEBUG:boto:Port: 8000 2024-03-04T17:51:31.048 DEBUG:boto:Params: {} 2024-03-04T17:51:31.048 DEBUG:boto:Token: None 2024-03-04T17:51:31.048 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:31 GMT /ppfahn-22/obj 2024-03-04T17:51:31.048 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UIpS72G9L860dzyJKjOVSws89V0= 2024-03-04T17:51:31.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UIpS72G9L860dzyJKjOVSws89V0='} 2024-03-04T17:51:31.092 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-amz-request-id', 'tx00000530a2ea636d20b67-0065e60a23-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:31.092 DEBUG:rgw_multi.tests:zone=test-zone2 version=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:51:31.092 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:51:31.093 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:51:31.093 DEBUG:boto:Method: HEAD 2024-03-04T17:51:31.093 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:51:31.093 DEBUG:boto:Data: 2024-03-04T17:51:31.093 DEBUG:boto:Headers: {} 2024-03-04T17:51:31.093 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:31.093 DEBUG:boto:Port: 8001 2024-03-04T17:51:31.093 DEBUG:boto:Params: {} 2024-03-04T17:51:31.093 DEBUG:boto:Token: None 2024-03-04T17:51:31.093 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:51:31 GMT /ppfahn-22/ 2024-03-04T17:51:31.094 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H6zjTQjWRwQspBu85pCB/pjUnCs= 2024-03-04T17:51:31.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:51:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H6zjTQjWRwQspBu85pCB/pjUnCs=', 'Content-Length': '0'} 2024-03-04T17:51:31.098 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010743bea3c42622a-0065e60a23-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:31.098 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:51:31.098 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:51:31.099 DEBUG:boto:Method: PUT 2024-03-04T17:51:31.099 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:51:31.099 DEBUG:boto:Data: 2024-03-04T17:51:31.099 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T17:51:31.099 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:51:31.099 DEBUG:boto:Port: 8001 2024-03-04T17:51:31.099 DEBUG:boto:Params: {} 2024-03-04T17:51:31.099 DEBUG:boto:Token: None 2024-03-04T17:51:31.099 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 17:51:31 GMT /ppfahn-22/obj 2024-03-04T17:51:31.099 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UIpS72G9L860dzyJKjOVSws89V0= 2024-03-04T17:51:31.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:51:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UIpS72G9L860dzyJKjOVSws89V0='} 2024-03-04T17:51:31.144 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-amz-request-id', 'tx00000a2bab71e15affcc7-0065e60a23-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:51:31.145 DEBUG:rgw_multi.tests:zone=test-zone3 version=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:51:31.145 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-22 2024-03-04T17:51:31.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:51:31.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:51:31.145 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:51:31.195 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:31.196 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:31.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:51:31.262+0000 7ff524949980 1 waiting for incremental sync to catch up: 2024-03-04T17:51:31.258 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000004.41.10, , , , , ] 2024-03-04T17:51:31.258 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000005.43.10, , , 00000000020.48.12, , ] 2024-03-04T17:51:36.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:51:36.262+0000 7ff524949980 1 waiting for incremental sync to catch up: 2024-03-04T17:51:36.261 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000004.41.10, , , , , ] 2024-03-04T17:51:36.261 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000005.43.10, , , 00000000020.48.12, , ] 2024-03-04T17:51:41.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:51:41.266+0000 7ff524949980 1 waiting for incremental sync to catch up: 2024-03-04T17:51:41.263 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000004.41.10, , , , , ] 2024-03-04T17:51:41.263 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000005.43.10, , , 00000000020.48.12, , ] 2024-03-04T17:51:46.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:51:46.270+0000 7ff524949980 1 waiting for incremental sync to catch up: 2024-03-04T17:51:46.265 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000004.41.10, , , 00000000036.94.6, , ] 2024-03-04T17:51:46.265 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000005.43.10, , , 00000000020.48.12, , ] 2024-03-04T17:51:51.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:51:51.270+0000 7ff524949980 1 waiting for incremental sync to catch up: 2024-03-04T17:51:51.268 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000004.41.10, , , 00000000036.94.6, , ] 2024-03-04T17:51:51.268 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000005.43.10, , , 00000000020.48.12, , ] 2024-03-04T17:51:56.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:51:56.278+0000 7ff524949980 1 bucket sync caught up with source: 2024-03-04T17:51:56.273 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000005.43.10, , , 00000000036.94.6, , ] 2024-03-04T17:51:56.273 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000005.43.10, , , 00000000020.48.12, , ] 2024-03-04T17:51:56.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:51:56.278+0000 7ff524949980 0 bucket checkpoint complete 2024-03-04T17:51:56.309 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-22 2024-03-04T17:51:56.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:51:56.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:51:56.310 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:51:56.379 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:56.379 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:56.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:56.438+0000 7f073d366980 1 bucket sync caught up with source: 2024-03-04T17:51:56.438 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000005.43.10, , , 00000000042.110.12, , ] 2024-03-04T17:51:56.438 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000005.43.10, , , 00000000042.110.12, , ] 2024-03-04T17:51:56.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:56.438+0000 7f073d366980 0 bucket checkpoint complete 2024-03-04T17:51:56.459 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-22 2024-03-04T17:51:56.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:51:56.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:51:56.460 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:51:56.500 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:56.500 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:56.559 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:51:56.562+0000 7f99c58f1980 1 waiting for incremental sync to catch up: 2024-03-04T17:51:56.560 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000020.52.12, , ] 2024-03-04T17:51:56.560 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000037.101.6, , ] 2024-03-04T17:52:01.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:01.566+0000 7f99c58f1980 1 waiting for incremental sync to catch up: 2024-03-04T17:52:01.562 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000020.52.12, , ] 2024-03-04T17:52:01.562 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000037.101.6, , ] 2024-03-04T17:52:06.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:06.566+0000 7f99c58f1980 1 waiting for incremental sync to catch up: 2024-03-04T17:52:06.565 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000022.58.6, , ] 2024-03-04T17:52:06.565 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000037.101.6, , ] 2024-03-04T17:52:11.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:11.570+0000 7f99c58f1980 1 waiting for incremental sync to catch up: 2024-03-04T17:52:11.567 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000022.58.6, , ] 2024-03-04T17:52:11.567 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000037.101.6, , ] 2024-03-04T17:52:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:16.570+0000 7f99c58f1980 1 waiting for incremental sync to catch up: 2024-03-04T17:52:16.570 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000022.58.6, , ] 2024-03-04T17:52:16.570 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000037.101.6, , ] 2024-03-04T17:52:21.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:21.574+0000 7f99c58f1980 1 waiting for incremental sync to catch up: 2024-03-04T17:52:21.572 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000022.58.6, , ] 2024-03-04T17:52:21.572 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000037.101.6, , ] 2024-03-04T17:52:26.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:26.578+0000 7f99c58f1980 1 waiting for incremental sync to catch up: 2024-03-04T17:52:26.575 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000022.58.6, , ] 2024-03-04T17:52:26.575 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000037.101.6, , ] 2024-03-04T17:52:26.765 INFO:tasks.rgw.c2.client.0.smithi153.stdout:2024-03-04T17:52:26.770+0000 7fa53962a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:52:31.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:31.578+0000 7f99c58f1980 1 waiting for incremental sync to catch up: 2024-03-04T17:52:31.577 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000022.58.6, , ] 2024-03-04T17:52:31.577 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000037.101.6, , ] 2024-03-04T17:52:36.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:36.582+0000 7f99c58f1980 1 waiting for incremental sync to catch up: 2024-03-04T17:52:36.579 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000022.58.6, , ] 2024-03-04T17:52:36.580 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000037.101.6, , ] 2024-03-04T17:52:41.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:41.581+0000 7f99c58f1980 1 waiting for incremental sync to catch up: 2024-03-04T17:52:41.582 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000022.58.6, , ] 2024-03-04T17:52:41.582 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000037.101.6, , ] 2024-03-04T17:52:46.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:46.585+0000 7f99c58f1980 1 bucket sync caught up with source: 2024-03-04T17:52:46.584 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000044.118.10, , ] 2024-03-04T17:52:46.584 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000037.101.6, , ] 2024-03-04T17:52:46.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:46.585+0000 7f99c58f1980 0 bucket checkpoint complete 2024-03-04T17:52:46.627 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-22 2024-03-04T17:52:46.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:52:46.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:52:46.628 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:52:46.720 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:46.721 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:46.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:46.781+0000 7f1e859f4980 1 waiting for incremental sync to catch up: 2024-03-04T17:52:46.782 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000031.85.6, , ] 2024-03-04T17:52:46.782 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000044.118.10, , ] 2024-03-04T17:52:51.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:51.785+0000 7f1e859f4980 1 bucket sync caught up with source: 2024-03-04T17:52:51.784 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000044.118.10, , ] 2024-03-04T17:52:51.784 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000044.118.10, , ] 2024-03-04T17:52:51.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:51.785+0000 7f1e859f4980 0 bucket checkpoint complete 2024-03-04T17:52:51.822 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-22 2024-03-04T17:52:51.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:52:51.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:52:51.823 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:52:51.872 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:51.872 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:51.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:51.933+0000 7f9863ddf980 1 waiting for incremental sync to catch up: 2024-03-04T17:52:51.932 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.55.10, , , 00000000023.66.6, , ] 2024-03-04T17:52:51.932 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.55.10, , , 00000000044.123.10, , ] 2024-03-04T17:52:56.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:52:56.937+0000 7f9863ddf980 1 waiting for incremental sync to catch up: 2024-03-04T17:52:56.934 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.55.10, , , 00000000023.66.6, , ] 2024-03-04T17:52:56.935 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.55.10, , , 00000000044.123.10, , ] 2024-03-04T17:53:01.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:53:01.937+0000 7f9863ddf980 1 waiting for incremental sync to catch up: 2024-03-04T17:53:01.937 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.55.10, , , 00000000023.66.6, , ] 2024-03-04T17:53:01.937 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.55.10, , , 00000000044.123.10, , ] 2024-03-04T17:53:06.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:53:06.941+0000 7f9863ddf980 1 bucket sync caught up with source: 2024-03-04T17:53:06.940 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , 00000000004.55.10, , , 00000000044.123.10, , ] 2024-03-04T17:53:06.940 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , 00000000004.55.10, , , 00000000044.123.10, , ] 2024-03-04T17:53:06.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:53:06.941+0000 7f9863ddf980 0 bucket checkpoint complete 2024-03-04T17:53:06.983 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-22 2024-03-04T17:53:06.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:53:06.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:53:06.984 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:53:07.033 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:07.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:53:07.101+0000 7fc6241c8980 1 waiting for incremental sync to catch up: 2024-03-04T17:53:07.097 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000004.55.10, , , 00000000039.109.6, , ] 2024-03-04T17:53:07.097 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000004.55.10, , , 00000000044.123.10, , ] 2024-03-04T17:53:12.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:53:12.101+0000 7fc6241c8980 1 waiting for incremental sync to catch up: 2024-03-04T17:53:12.099 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000004.55.10, , , 00000000039.109.6, , ] 2024-03-04T17:53:12.100 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000004.55.10, , , 00000000044.123.10, , ] 2024-03-04T17:53:17.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:53:17.105+0000 7fc6241c8980 1 waiting for incremental sync to catch up: 2024-03-04T17:53:17.102 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000004.55.10, , , 00000000039.109.6, , ] 2024-03-04T17:53:17.102 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000004.55.10, , , 00000000044.123.10, , ] 2024-03-04T17:53:22.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:53:22.109+0000 7fc6241c8980 1 waiting for incremental sync to catch up: 2024-03-04T17:53:22.104 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000004.55.10, , , 00000000039.109.6, , ] 2024-03-04T17:53:22.104 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000004.55.10, , , 00000000044.123.10, , ] 2024-03-04T17:53:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:53:27.109+0000 7fc6241c8980 1 bucket sync caught up with source: 2024-03-04T17:53:27.106 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , 00000000004.55.10, , , 00000000044.123.10, , ] 2024-03-04T17:53:27.106 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , 00000000004.55.10, , , 00000000044.123.10, , ] 2024-03-04T17:53:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:53:27.109+0000 7fc6241c8980 0 bucket checkpoint complete 2024-03-04T17:53:27.146 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-22 zones={test-zone2, test-zone1} 2024-03-04T17:53:27.146 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.147 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.147 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.147 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:53:27.147 DEBUG:boto:Data: 2024-03-04T17:53:27.147 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.147 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.147 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.147 DEBUG:boto:Params: {} 2024-03-04T17:53:27.148 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:53:27.148 DEBUG:boto:Token: None 2024-03-04T17:53:27.148 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.148 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PCBJ6F8Jr0aRazmpangDSa2c2HI= 2024-03-04T17:53:27.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PCBJ6F8Jr0aRazmpangDSa2c2HI=', 'Content-Length': '0'} 2024-03-04T17:53:27.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7491a7b43b15155-0065e60a97-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.154 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.154 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.155 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.155 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:53:27.155 DEBUG:boto:Data: 2024-03-04T17:53:27.155 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.155 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.155 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.155 DEBUG:boto:Params: {} 2024-03-04T17:53:27.155 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:53:27.155 DEBUG:boto:Token: None 2024-03-04T17:53:27.155 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.156 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PCBJ6F8Jr0aRazmpangDSa2c2HI= 2024-03-04T17:53:27.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PCBJ6F8Jr0aRazmpangDSa2c2HI=', 'Content-Length': '0'} 2024-03-04T17:53:27.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000110738039dc25bd-0065e60a97-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.160 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:53:27.160 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.160 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.160 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:27.160 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:27.160 DEBUG:boto:Method: GET 2024-03-04T17:53:27.160 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:27.160 DEBUG:boto:Data: 2024-03-04T17:53:27.161 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.161 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.161 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.161 DEBUG:boto:Params: {} 2024-03-04T17:53:27.161 DEBUG:boto:Token: None 2024-03-04T17:53:27.161 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/?versions 2024-03-04T17:53:27.161 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aMsPxwtKR+TLT3mC23gTjwCyTco= 2024-03-04T17:53:27.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aMsPxwtKR+TLT3mC23gTjwCyTco=', 'Content-Length': '0'} 2024-03-04T17:53:27.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012da19d38e43069f-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.166 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG6tTJPArSLLFYQGx5Z9J94Qp3btVq8wtrue2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQDkS502cdQ0sD3MSSr4AJ.nNDwrboFEfalse2024-03-04T17:51:31.005Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfexW91D920ZLyPSn0KVa8L-0Jf11s3qfalse2024-03-04T17:51:31.057Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4aDzoElev9M4kJpBvZIFgGfNSrnTu1hfalse2024-03-04T17:51:30.946Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrBR7sHkOppK-w8aON9rPlfIadTwy8LKfalse2024-03-04T17:51:30.974Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtp1YBJlmM1cmYYj2xRDAuLHN-EIcD23false2024-03-04T17:51:30.916Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxsfalse2024-03-04T17:51:30.859Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIgnYdWgPuuz09tkj9qWq7P8qzgBSCIyfalse2024-03-04T17:51:30.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZw7l4YYB37p1xa7jtLR7kDu1d5Xjl6Gfalse2024-03-04T17:51:30.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10Vjj0fS8rYkr3OIS1pBSR-3fCH6svyfalse2024-03-04T17:51:30.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXak5SqkkJ7eVzuGI.8.u99lZqCC3tNwfalse2024-03-04T17:51:30.749Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzyjvmlB6z-cVoQlri1b6gVvpnPVUCjFfalse2024-03-04T17:51:30.721Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6BBhxVS-A79T2MI6AHE.THWDO21iftffalse2024-03-04T17:51:30.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7kfalse2024-03-04T17:51:30.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji.eNnYbM.4yY91CfDbCxKIqa-Uec-fxfalse2024-03-04T17:51:30.610Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkUfalse2024-03-04T17:51:30.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje9K3WlwCsFUmkYneucDfMv9rNQO6cdpfalse2024-03-04T17:51:30.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRTfalse2024-03-04T17:51:30.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgRxtOPcYyW7cm-IBYtAu0zyYNScRyoTfalse2024-03-04T17:51:30.416Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjme6.y9cJ8bkVcZ04PNMrwIXjDQUGzXRfalse2024-03-04T17:51:30.386Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoHs8pGuS1bc-GYUbjIaqPQPwQGDukkcfalse2024-03-04T17:51:30.469Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj160IY3UeXb5qH4Rh4JB09PNigxsTa5gfalse2024-03-04T17:51:30.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9WE9mNbplYdU449Vkwh8lwrICYxU6v7false2024-03-04T17:51:30.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZJZPq8HwR1LmdjGaT37eTcVO7gvndKgfalse2024-03-04T17:51:30.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lwBSkQlMAA54TFAsh7J7MCnzo0gTfafalse2024-03-04T17:51:30.157Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq4h-uWPuCpU7FK9SfN-LHLv-vtuKwuEfalse2024-03-04T17:51:30.239Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzKZVhMV59rWXQPf05XlhVIimSaZTr40false2024-03-04T17:51:30.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAcrzzw6Ded5-7UcCqpx-w2RGP5BSbOUfalse2024-03-04T17:51:30.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdVzfZSb65W93TGSPX8waGA5tm4N9lW6false2024-03-04T17:51:30.037Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlwBZf1YBiU2Wqy250Elw-2.f6FuQF0efalse2024-03-04T17:51:30.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.168 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:53:27.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.171 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:53:27.171 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.171 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.171 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:27.172 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:27.172 DEBUG:boto:Method: GET 2024-03-04T17:53:27.172 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:27.172 DEBUG:boto:Data: 2024-03-04T17:53:27.172 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.172 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.172 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.172 DEBUG:boto:Params: {} 2024-03-04T17:53:27.172 DEBUG:boto:Token: None 2024-03-04T17:53:27.172 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/?versions 2024-03-04T17:53:27.173 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aMsPxwtKR+TLT3mC23gTjwCyTco= 2024-03-04T17:53:27.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aMsPxwtKR+TLT3mC23gTjwCyTco=', 'Content-Length': '0'} 2024-03-04T17:53:27.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014be4765dd8a2c69-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.175 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG6tTJPArSLLFYQGx5Z9J94Qp3btVq8wtrue2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQDkS502cdQ0sD3MSSr4AJ.nNDwrboFEfalse2024-03-04T17:51:31.005Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfexW91D920ZLyPSn0KVa8L-0Jf11s3qfalse2024-03-04T17:51:31.057Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4aDzoElev9M4kJpBvZIFgGfNSrnTu1hfalse2024-03-04T17:51:30.946Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrBR7sHkOppK-w8aON9rPlfIadTwy8LKfalse2024-03-04T17:51:30.974Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtp1YBJlmM1cmYYj2xRDAuLHN-EIcD23false2024-03-04T17:51:30.916Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxsfalse2024-03-04T17:51:30.859Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIgnYdWgPuuz09tkj9qWq7P8qzgBSCIyfalse2024-03-04T17:51:30.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZw7l4YYB37p1xa7jtLR7kDu1d5Xjl6Gfalse2024-03-04T17:51:30.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10Vjj0fS8rYkr3OIS1pBSR-3fCH6svyfalse2024-03-04T17:51:30.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXak5SqkkJ7eVzuGI.8.u99lZqCC3tNwfalse2024-03-04T17:51:30.749Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzyjvmlB6z-cVoQlri1b6gVvpnPVUCjFfalse2024-03-04T17:51:30.721Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6BBhxVS-A79T2MI6AHE.THWDO21iftffalse2024-03-04T17:51:30.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7kfalse2024-03-04T17:51:30.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji.eNnYbM.4yY91CfDbCxKIqa-Uec-fxfalse2024-03-04T17:51:30.610Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkUfalse2024-03-04T17:51:30.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje9K3WlwCsFUmkYneucDfMv9rNQO6cdpfalse2024-03-04T17:51:30.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRTfalse2024-03-04T17:51:30.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgRxtOPcYyW7cm-IBYtAu0zyYNScRyoTfalse2024-03-04T17:51:30.416Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjme6.y9cJ8bkVcZ04PNMrwIXjDQUGzXRfalse2024-03-04T17:51:30.386Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoHs8pGuS1bc-GYUbjIaqPQPwQGDukkcfalse2024-03-04T17:51:30.469Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj160IY3UeXb5qH4Rh4JB09PNigxsTa5gfalse2024-03-04T17:51:30.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9WE9mNbplYdU449Vkwh8lwrICYxU6v7false2024-03-04T17:51:30.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZJZPq8HwR1LmdjGaT37eTcVO7gvndKgfalse2024-03-04T17:51:30.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lwBSkQlMAA54TFAsh7J7MCnzo0gTfafalse2024-03-04T17:51:30.157Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq4h-uWPuCpU7FK9SfN-LHLv-vtuKwuEfalse2024-03-04T17:51:30.239Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzKZVhMV59rWXQPf05XlhVIimSaZTr40false2024-03-04T17:51:30.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAcrzzw6Ded5-7UcCqpx-w2RGP5BSbOUfalse2024-03-04T17:51:30.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdVzfZSb65W93TGSPX8waGA5tm4N9lW6false2024-03-04T17:51:30.037Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlwBZf1YBiU2Wqy250Elw-2.f6FuQF0efalse2024-03-04T17:51:30.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.177 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:53:27.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:27.180 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.180 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.180 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:27.180 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:27.180 DEBUG:boto:Method: GET 2024-03-04T17:53:27.181 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:27.181 DEBUG:boto:Data: 2024-03-04T17:53:27.181 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.181 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.181 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.181 DEBUG:boto:Params: {} 2024-03-04T17:53:27.181 DEBUG:boto:Token: None 2024-03-04T17:53:27.181 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/?versions 2024-03-04T17:53:27.181 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aMsPxwtKR+TLT3mC23gTjwCyTco= 2024-03-04T17:53:27.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aMsPxwtKR+TLT3mC23gTjwCyTco=', 'Content-Length': '0'} 2024-03-04T17:53:27.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002004cbbcb5a4fe20-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.188 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG6tTJPArSLLFYQGx5Z9J94Qp3btVq8wtrue2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQDkS502cdQ0sD3MSSr4AJ.nNDwrboFEfalse2024-03-04T17:51:31.005Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfexW91D920ZLyPSn0KVa8L-0Jf11s3qfalse2024-03-04T17:51:31.057Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4aDzoElev9M4kJpBvZIFgGfNSrnTu1hfalse2024-03-04T17:51:30.946Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrBR7sHkOppK-w8aON9rPlfIadTwy8LKfalse2024-03-04T17:51:30.974Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtp1YBJlmM1cmYYj2xRDAuLHN-EIcD23false2024-03-04T17:51:30.916Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxsfalse2024-03-04T17:51:30.859Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIgnYdWgPuuz09tkj9qWq7P8qzgBSCIyfalse2024-03-04T17:51:30.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZw7l4YYB37p1xa7jtLR7kDu1d5Xjl6Gfalse2024-03-04T17:51:30.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10Vjj0fS8rYkr3OIS1pBSR-3fCH6svyfalse2024-03-04T17:51:30.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXak5SqkkJ7eVzuGI.8.u99lZqCC3tNwfalse2024-03-04T17:51:30.749Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzyjvmlB6z-cVoQlri1b6gVvpnPVUCjFfalse2024-03-04T17:51:30.721Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6BBhxVS-A79T2MI6AHE.THWDO21iftffalse2024-03-04T17:51:30.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7kfalse2024-03-04T17:51:30.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji.eNnYbM.4yY91CfDbCxKIqa-Uec-fxfalse2024-03-04T17:51:30.610Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkUfalse2024-03-04T17:51:30.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje9K3WlwCsFUmkYneucDfMv9rNQO6cdpfalse2024-03-04T17:51:30.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRTfalse2024-03-04T17:51:30.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgRxtOPcYyW7cm-IBYtAu0zyYNScRyoTfalse2024-03-04T17:51:30.416Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjme6.y9cJ8bkVcZ04PNMrwIXjDQUGzXRfalse2024-03-04T17:51:30.386Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoHs8pGuS1bc-GYUbjIaqPQPwQGDukkcfalse2024-03-04T17:51:30.469Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj160IY3UeXb5qH4Rh4JB09PNigxsTa5gfalse2024-03-04T17:51:30.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9WE9mNbplYdU449Vkwh8lwrICYxU6v7false2024-03-04T17:51:30.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZJZPq8HwR1LmdjGaT37eTcVO7gvndKgfalse2024-03-04T17:51:30.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lwBSkQlMAA54TFAsh7J7MCnzo0gTfafalse2024-03-04T17:51:30.157Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq4h-uWPuCpU7FK9SfN-LHLv-vtuKwuEfalse2024-03-04T17:51:30.239Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzKZVhMV59rWXQPf05XlhVIimSaZTr40false2024-03-04T17:51:30.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAcrzzw6Ded5-7UcCqpx-w2RGP5BSbOUfalse2024-03-04T17:51:30.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdVzfZSb65W93TGSPX8waGA5tm4N9lW6false2024-03-04T17:51:30.037Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlwBZf1YBiU2Wqy250Elw-2.f6FuQF0efalse2024-03-04T17:51:30.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.190 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.190 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.190 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:27.190 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:27.190 DEBUG:boto:Method: GET 2024-03-04T17:53:27.190 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:27.190 DEBUG:boto:Data: 2024-03-04T17:53:27.190 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.190 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.191 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.191 DEBUG:boto:Params: {} 2024-03-04T17:53:27.191 DEBUG:boto:Token: None 2024-03-04T17:53:27.191 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/?versions 2024-03-04T17:53:27.191 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aMsPxwtKR+TLT3mC23gTjwCyTco= 2024-03-04T17:53:27.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aMsPxwtKR+TLT3mC23gTjwCyTco=', 'Content-Length': '0'} 2024-03-04T17:53:27.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b432256e3fa92377-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.195 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG6tTJPArSLLFYQGx5Z9J94Qp3btVq8wtrue2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQDkS502cdQ0sD3MSSr4AJ.nNDwrboFEfalse2024-03-04T17:51:31.005Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfexW91D920ZLyPSn0KVa8L-0Jf11s3qfalse2024-03-04T17:51:31.057Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4aDzoElev9M4kJpBvZIFgGfNSrnTu1hfalse2024-03-04T17:51:30.946Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrBR7sHkOppK-w8aON9rPlfIadTwy8LKfalse2024-03-04T17:51:30.974Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtp1YBJlmM1cmYYj2xRDAuLHN-EIcD23false2024-03-04T17:51:30.916Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxsfalse2024-03-04T17:51:30.859Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIgnYdWgPuuz09tkj9qWq7P8qzgBSCIyfalse2024-03-04T17:51:30.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZw7l4YYB37p1xa7jtLR7kDu1d5Xjl6Gfalse2024-03-04T17:51:30.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10Vjj0fS8rYkr3OIS1pBSR-3fCH6svyfalse2024-03-04T17:51:30.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXak5SqkkJ7eVzuGI.8.u99lZqCC3tNwfalse2024-03-04T17:51:30.749Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzyjvmlB6z-cVoQlri1b6gVvpnPVUCjFfalse2024-03-04T17:51:30.721Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6BBhxVS-A79T2MI6AHE.THWDO21iftffalse2024-03-04T17:51:30.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7kfalse2024-03-04T17:51:30.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji.eNnYbM.4yY91CfDbCxKIqa-Uec-fxfalse2024-03-04T17:51:30.610Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkUfalse2024-03-04T17:51:30.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje9K3WlwCsFUmkYneucDfMv9rNQO6cdpfalse2024-03-04T17:51:30.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRTfalse2024-03-04T17:51:30.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgRxtOPcYyW7cm-IBYtAu0zyYNScRyoTfalse2024-03-04T17:51:30.416Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjme6.y9cJ8bkVcZ04PNMrwIXjDQUGzXRfalse2024-03-04T17:51:30.386Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoHs8pGuS1bc-GYUbjIaqPQPwQGDukkcfalse2024-03-04T17:51:30.469Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj160IY3UeXb5qH4Rh4JB09PNigxsTa5gfalse2024-03-04T17:51:30.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9WE9mNbplYdU449Vkwh8lwrICYxU6v7false2024-03-04T17:51:30.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZJZPq8HwR1LmdjGaT37eTcVO7gvndKgfalse2024-03-04T17:51:30.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lwBSkQlMAA54TFAsh7J7MCnzo0gTfafalse2024-03-04T17:51:30.157Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq4h-uWPuCpU7FK9SfN-LHLv-vtuKwuEfalse2024-03-04T17:51:30.239Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzKZVhMV59rWXQPf05XlhVIimSaZTr40false2024-03-04T17:51:30.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAcrzzw6Ded5-7UcCqpx-w2RGP5BSbOUfalse2024-03-04T17:51:30.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdVzfZSb65W93TGSPX8waGA5tm4N9lW6false2024-03-04T17:51:30.037Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlwBZf1YBiU2Wqy250Elw-2.f6FuQF0efalse2024-03-04T17:51:30.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:53:27.197 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:27.197 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:27.198 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.198 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.198 DEBUG:boto:Method: GET 2024-03-04T17:53:27.198 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.198 DEBUG:boto:Data: 2024-03-04T17:53:27.198 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.198 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.198 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.198 DEBUG:boto:Params: {} 2024-03-04T17:53:27.198 DEBUG:boto:Token: None 2024-03-04T17:53:27.199 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.199 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xZyNrDv7jT5YJjCXMvxy14jveu8= 2024-03-04T17:53:27.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xZyNrDv7jT5YJjCXMvxy14jveu8=', 'Content-Length': '0'} 2024-03-04T17:53:27.201 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b75bf8ae10d9b590-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.201 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:27.202 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:27.202 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.202 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.202 DEBUG:boto:Method: GET 2024-03-04T17:53:27.202 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.202 DEBUG:boto:Data: 2024-03-04T17:53:27.202 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.202 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.202 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.202 DEBUG:boto:Params: {} 2024-03-04T17:53:27.203 DEBUG:boto:Token: None 2024-03-04T17:53:27.203 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.203 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xZyNrDv7jT5YJjCXMvxy14jveu8= 2024-03-04T17:53:27.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xZyNrDv7jT5YJjCXMvxy14jveu8=', 'Content-Length': '0'} 2024-03-04T17:53:27.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c247f003f8a0c4ed-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.206 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:27.206 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:27.206 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.206 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.206 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.206 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.206 DEBUG:boto:Data: 2024-03-04T17:53:27.207 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.207 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.207 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.207 DEBUG:boto:Params: {} 2024-03-04T17:53:27.207 DEBUG:boto:Token: None 2024-03-04T17:53:27.207 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.207 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Eo3US3TiYzIgzkhnwD+2GlKNs8s= 2024-03-04T17:53:27.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Eo3US3TiYzIgzkhnwD+2GlKNs8s=', 'Content-Length': '0'} 2024-03-04T17:53:27.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005df6c2d06f0668bf-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.210 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:27.210 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:27.210 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.210 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.210 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.210 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.211 DEBUG:boto:Data: 2024-03-04T17:53:27.211 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.211 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.211 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.211 DEBUG:boto:Params: {} 2024-03-04T17:53:27.211 DEBUG:boto:Token: None 2024-03-04T17:53:27.211 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.211 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Eo3US3TiYzIgzkhnwD+2GlKNs8s= 2024-03-04T17:53:27.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Eo3US3TiYzIgzkhnwD+2GlKNs8s=', 'Content-Length': '0'} 2024-03-04T17:53:27.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000096e649d6f65cc379-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:53:27.214 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:27.214 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:27.214 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.215 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.215 DEBUG:boto:Method: GET 2024-03-04T17:53:27.215 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.215 DEBUG:boto:Data: 2024-03-04T17:53:27.215 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.215 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.215 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.215 DEBUG:boto:Params: {} 2024-03-04T17:53:27.215 DEBUG:boto:Token: None 2024-03-04T17:53:27.215 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.216 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xZyNrDv7jT5YJjCXMvxy14jveu8= 2024-03-04T17:53:27.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xZyNrDv7jT5YJjCXMvxy14jveu8=', 'Content-Length': '0'} 2024-03-04T17:53:27.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001da729269c613a57-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.219 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:27.219 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:27.219 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.219 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.219 DEBUG:boto:Method: GET 2024-03-04T17:53:27.219 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.219 DEBUG:boto:Data: 2024-03-04T17:53:27.220 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.220 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.220 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.220 DEBUG:boto:Params: {} 2024-03-04T17:53:27.220 DEBUG:boto:Token: None 2024-03-04T17:53:27.220 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.220 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xZyNrDv7jT5YJjCXMvxy14jveu8= 2024-03-04T17:53:27.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xZyNrDv7jT5YJjCXMvxy14jveu8=', 'Content-Length': '0'} 2024-03-04T17:53:27.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bbe9b8abeb1233a8-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.224 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.224 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.224 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:27.224 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:27.224 DEBUG:boto:Method: GET 2024-03-04T17:53:27.224 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:53:27.224 DEBUG:boto:Data: 2024-03-04T17:53:27.224 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.224 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.224 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.224 DEBUG:boto:Params: {} 2024-03-04T17:53:27.225 DEBUG:boto:Token: None 2024-03-04T17:53:27.225 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.225 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc79f32062adbe01-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.272 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.273 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.273 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.273 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:27.273 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:27.273 DEBUG:boto:Method: GET 2024-03-04T17:53:27.273 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:53:27.273 DEBUG:boto:Data: 2024-03-04T17:53:27.273 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.273 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.273 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.273 DEBUG:boto:Params: {} 2024-03-04T17:53:27.274 DEBUG:boto:Token: None 2024-03-04T17:53:27.274 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.274 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005805d7666eeb1803-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.278 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.279 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:27.279 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:27.279 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.279 DEBUG:boto:Path: /ppfahn-22/dummy 2024-03-04T17:53:27.279 DEBUG:boto:Data: 2024-03-04T17:53:27.279 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.279 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.279 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.279 DEBUG:boto:Params: {} 2024-03-04T17:53:27.280 DEBUG:boto:Token: None 2024-03-04T17:53:27.280 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/dummy 2024-03-04T17:53:27.280 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:j2uQJWyrgm4TiQdy6mGgE6M++Wk= 2024-03-04T17:53:27.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:j2uQJWyrgm4TiQdy6mGgE6M++Wk=', 'Content-Length': '0'} 2024-03-04T17:53:27.284 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c6304763fea07a6-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.284 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:27.284 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:27.284 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.284 DEBUG:boto:Path: /ppfahn-22/dummy 2024-03-04T17:53:27.284 DEBUG:boto:Data: 2024-03-04T17:53:27.285 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.285 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.285 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.285 DEBUG:boto:Params: {} 2024-03-04T17:53:27.285 DEBUG:boto:Token: None 2024-03-04T17:53:27.285 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/dummy 2024-03-04T17:53:27.285 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:j2uQJWyrgm4TiQdy6mGgE6M++Wk= 2024-03-04T17:53:27.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:j2uQJWyrgm4TiQdy6mGgE6M++Wk=', 'Content-Length': '0'} 2024-03-04T17:53:27.289 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036e9247f2d1286fa-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:53:27.290 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:27.290 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:27.290 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.290 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.290 DEBUG:boto:Method: GET 2024-03-04T17:53:27.290 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.290 DEBUG:boto:Data: 2024-03-04T17:53:27.291 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.291 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.291 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.291 DEBUG:boto:Params: {} 2024-03-04T17:53:27.291 DEBUG:boto:Token: None 2024-03-04T17:53:27.291 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.291 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xZyNrDv7jT5YJjCXMvxy14jveu8= 2024-03-04T17:53:27.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xZyNrDv7jT5YJjCXMvxy14jveu8=', 'Content-Length': '0'} 2024-03-04T17:53:27.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000ebc0a947bc55d30-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.295 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:27.295 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:27.295 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.295 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.295 DEBUG:boto:Method: GET 2024-03-04T17:53:27.295 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.295 DEBUG:boto:Data: 2024-03-04T17:53:27.295 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.296 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.296 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.296 DEBUG:boto:Params: {} 2024-03-04T17:53:27.296 DEBUG:boto:Token: None 2024-03-04T17:53:27.296 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:27.296 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xZyNrDv7jT5YJjCXMvxy14jveu8= 2024-03-04T17:53:27.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xZyNrDv7jT5YJjCXMvxy14jveu8=', 'Content-Length': '0'} 2024-03-04T17:53:27.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8bc7c1bba01b34f-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.300 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.300 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.300 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:27.300 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:27.300 DEBUG:boto:Method: GET 2024-03-04T17:53:27.300 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:53:27.300 DEBUG:boto:Data: 2024-03-04T17:53:27.300 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.300 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.301 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.301 DEBUG:boto:Params: {} 2024-03-04T17:53:27.301 DEBUG:boto:Token: None 2024-03-04T17:53:27.301 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.301 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5247cf72881df1a-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.306 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.306 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.306 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.306 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:27.306 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:27.306 DEBUG:boto:Method: GET 2024-03-04T17:53:27.306 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:53:27.307 DEBUG:boto:Data: 2024-03-04T17:53:27.307 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.307 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.307 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.307 DEBUG:boto:Params: {} 2024-03-04T17:53:27.307 DEBUG:boto:Token: None 2024-03-04T17:53:27.307 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.307 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7b38b6dfa42616a-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.352 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:27.352 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.353 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.353 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.353 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.353 DEBUG:boto:Method: GET 2024-03-04T17:53:27.353 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.353 DEBUG:boto:Data: 2024-03-04T17:53:27.353 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.353 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.353 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.353 DEBUG:boto:Params: {} 2024-03-04T17:53:27.353 DEBUG:boto:Token: None 2024-03-04T17:53:27.354 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.354 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto= 2024-03-04T17:53:27.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto=', 'Content-Length': '0'} 2024-03-04T17:53:27.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b0c2433c809d2e5-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.357 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.358 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.358 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.358 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.358 DEBUG:boto:Method: GET 2024-03-04T17:53:27.358 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.358 DEBUG:boto:Data: 2024-03-04T17:53:27.358 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.358 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.358 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.359 DEBUG:boto:Params: {} 2024-03-04T17:53:27.359 DEBUG:boto:Token: None 2024-03-04T17:53:27.359 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.359 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto= 2024-03-04T17:53:27.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto=', 'Content-Length': '0'} 2024-03-04T17:53:27.363 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eafed679556deecd-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.363 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.363 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.364 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.364 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.364 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.364 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.364 DEBUG:boto:Data: 2024-03-04T17:53:27.364 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.364 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.364 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.364 DEBUG:boto:Params: {} 2024-03-04T17:53:27.364 DEBUG:boto:Token: None 2024-03-04T17:53:27.365 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.365 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e0e82gYnvUa+rRs3hnsC3HgakJY= 2024-03-04T17:53:27.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e0e82gYnvUa+rRs3hnsC3HgakJY=', 'Content-Length': '0'} 2024-03-04T17:53:27.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f560fcc48f656c21-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.368 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.368 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.368 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.368 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.369 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.369 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.369 DEBUG:boto:Data: 2024-03-04T17:53:27.369 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.369 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.369 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.369 DEBUG:boto:Params: {} 2024-03-04T17:53:27.369 DEBUG:boto:Token: None 2024-03-04T17:53:27.369 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.370 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:e0e82gYnvUa+rRs3hnsC3HgakJY= 2024-03-04T17:53:27.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:e0e82gYnvUa+rRs3hnsC3HgakJY=', 'Content-Length': '0'} 2024-03-04T17:53:27.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a108362fc040d30-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:27.373 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.373 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.373 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.373 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.373 DEBUG:boto:Method: GET 2024-03-04T17:53:27.373 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.374 DEBUG:boto:Data: 2024-03-04T17:53:27.374 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.374 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.374 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.374 DEBUG:boto:Params: {} 2024-03-04T17:53:27.374 DEBUG:boto:Token: None 2024-03-04T17:53:27.374 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.374 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto= 2024-03-04T17:53:27.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto=', 'Content-Length': '0'} 2024-03-04T17:53:27.377 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b6248c195a666f2-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.424 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.424 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.424 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.424 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.424 DEBUG:boto:Method: GET 2024-03-04T17:53:27.424 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.425 DEBUG:boto:Data: 2024-03-04T17:53:27.425 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.425 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.425 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.425 DEBUG:boto:Params: {} 2024-03-04T17:53:27.425 DEBUG:boto:Token: None 2024-03-04T17:53:27.425 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.425 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto= 2024-03-04T17:53:27.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto=', 'Content-Length': '0'} 2024-03-04T17:53:27.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efe1a636cf4bf917-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.476 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.476 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.476 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.476 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.476 DEBUG:boto:Method: GET 2024-03-04T17:53:27.477 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:27.477 DEBUG:boto:Data: 2024-03-04T17:53:27.477 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.477 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.477 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.477 DEBUG:boto:Params: {} 2024-03-04T17:53:27.477 DEBUG:boto:Token: None 2024-03-04T17:53:27.477 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.477 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6de3fa096d04ea8-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.482 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.483 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.483 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.483 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.483 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.483 DEBUG:boto:Method: GET 2024-03-04T17:53:27.483 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:27.483 DEBUG:boto:Data: 2024-03-04T17:53:27.483 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.483 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.484 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.484 DEBUG:boto:Params: {} 2024-03-04T17:53:27.484 DEBUG:boto:Token: None 2024-03-04T17:53:27.484 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.484 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000842f261f56253516-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.532 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.532 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.532 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.532 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.533 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:27.533 DEBUG:boto:Data: 2024-03-04T17:53:27.533 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.533 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.533 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.533 DEBUG:boto:Params: {} 2024-03-04T17:53:27.533 DEBUG:boto:Token: None 2024-03-04T17:53:27.533 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj 2024-03-04T17:53:27.533 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lGGG2QYD50t4lSsvH6oK77uX+rA= 2024-03-04T17:53:27.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lGGG2QYD50t4lSsvH6oK77uX+rA=', 'Content-Length': '0'} 2024-03-04T17:53:27.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6de3fb216d94837-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.540 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.541 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.541 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.541 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:27.541 DEBUG:boto:Data: 2024-03-04T17:53:27.541 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.541 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.541 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.541 DEBUG:boto:Params: {} 2024-03-04T17:53:27.541 DEBUG:boto:Token: None 2024-03-04T17:53:27.541 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj 2024-03-04T17:53:27.541 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lGGG2QYD50t4lSsvH6oK77uX+rA= 2024-03-04T17:53:27.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lGGG2QYD50t4lSsvH6oK77uX+rA=', 'Content-Length': '0'} 2024-03-04T17:53:27.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000636e8efe5122bc-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:27.548 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.548 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.548 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.548 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.548 DEBUG:boto:Method: GET 2024-03-04T17:53:27.549 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.549 DEBUG:boto:Data: 2024-03-04T17:53:27.549 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.549 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.549 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.549 DEBUG:boto:Params: {} 2024-03-04T17:53:27.549 DEBUG:boto:Token: None 2024-03-04T17:53:27.550 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.550 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto= 2024-03-04T17:53:27.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto=', 'Content-Length': '0'} 2024-03-04T17:53:27.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0698aefe09d6975-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.553 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.553 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.553 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.553 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.553 DEBUG:boto:Method: GET 2024-03-04T17:53:27.553 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.554 DEBUG:boto:Data: 2024-03-04T17:53:27.554 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.554 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.554 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.554 DEBUG:boto:Params: {} 2024-03-04T17:53:27.554 DEBUG:boto:Token: None 2024-03-04T17:53:27.554 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.554 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto= 2024-03-04T17:53:27.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto=', 'Content-Length': '0'} 2024-03-04T17:53:27.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e2bdb8f2a094d2c-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.558 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.558 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.558 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.558 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.558 DEBUG:boto:Method: GET 2024-03-04T17:53:27.558 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:27.558 DEBUG:boto:Data: 2024-03-04T17:53:27.558 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.559 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.559 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.559 DEBUG:boto:Params: {} 2024-03-04T17:53:27.559 DEBUG:boto:Token: None 2024-03-04T17:53:27.559 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.559 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008148c21c9a5f01e4-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.564 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.565 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.565 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.565 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.565 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.565 DEBUG:boto:Method: GET 2024-03-04T17:53:27.565 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:27.565 DEBUG:boto:Data: 2024-03-04T17:53:27.565 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.565 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.565 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.565 DEBUG:boto:Params: {} 2024-03-04T17:53:27.566 DEBUG:boto:Token: None 2024-03-04T17:53:27.566 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.566 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ffda75aebf749f3-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.616 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:27.617 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.617 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.617 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.617 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.617 DEBUG:boto:Method: GET 2024-03-04T17:53:27.617 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.617 DEBUG:boto:Data: 2024-03-04T17:53:27.617 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.617 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.618 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.618 DEBUG:boto:Params: {} 2024-03-04T17:53:27.618 DEBUG:boto:Token: None 2024-03-04T17:53:27.618 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.618 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/REFVQoqxVHmFXvE4csJtxWG2xo= 2024-03-04T17:53:27.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/REFVQoqxVHmFXvE4csJtxWG2xo=', 'Content-Length': '0'} 2024-03-04T17:53:27.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c784b27d13b4ec46-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.621 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.622 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.622 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.622 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.622 DEBUG:boto:Method: GET 2024-03-04T17:53:27.622 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.622 DEBUG:boto:Data: 2024-03-04T17:53:27.622 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.622 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.622 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.622 DEBUG:boto:Params: {} 2024-03-04T17:53:27.622 DEBUG:boto:Token: None 2024-03-04T17:53:27.623 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.623 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/REFVQoqxVHmFXvE4csJtxWG2xo= 2024-03-04T17:53:27.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/REFVQoqxVHmFXvE4csJtxWG2xo=', 'Content-Length': '0'} 2024-03-04T17:53:27.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abe7a54a3a8d62e8-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.626 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.626 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.626 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.626 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.626 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.627 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.627 DEBUG:boto:Data: 2024-03-04T17:53:27.627 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.627 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.627 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.627 DEBUG:boto:Params: {} 2024-03-04T17:53:27.627 DEBUG:boto:Token: None 2024-03-04T17:53:27.627 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.627 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:az/T0lVcqCnBKl8TUkWF/O0cqTI= 2024-03-04T17:53:27.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:az/T0lVcqCnBKl8TUkWF/O0cqTI=', 'Content-Length': '0'} 2024-03-04T17:53:27.631 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000532ccf2278f3e362-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.631 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.631 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.631 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.631 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.631 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.631 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.631 DEBUG:boto:Data: 2024-03-04T17:53:27.632 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.632 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.632 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.632 DEBUG:boto:Params: {} 2024-03-04T17:53:27.632 DEBUG:boto:Token: None 2024-03-04T17:53:27.632 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.632 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:az/T0lVcqCnBKl8TUkWF/O0cqTI= 2024-03-04T17:53:27.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:az/T0lVcqCnBKl8TUkWF/O0cqTI=', 'Content-Length': '0'} 2024-03-04T17:53:27.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0ffa0b6d925912d-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:27.636 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.636 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.636 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.636 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.636 DEBUG:boto:Method: GET 2024-03-04T17:53:27.636 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.636 DEBUG:boto:Data: 2024-03-04T17:53:27.636 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.636 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.636 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.637 DEBUG:boto:Params: {} 2024-03-04T17:53:27.637 DEBUG:boto:Token: None 2024-03-04T17:53:27.637 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.637 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/REFVQoqxVHmFXvE4csJtxWG2xo= 2024-03-04T17:53:27.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/REFVQoqxVHmFXvE4csJtxWG2xo=', 'Content-Length': '0'} 2024-03-04T17:53:27.640 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f4d74aa8f195751-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.688 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.688 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.688 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.688 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.689 DEBUG:boto:Method: GET 2024-03-04T17:53:27.689 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.689 DEBUG:boto:Data: 2024-03-04T17:53:27.689 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.689 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.689 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.689 DEBUG:boto:Params: {} 2024-03-04T17:53:27.689 DEBUG:boto:Token: None 2024-03-04T17:53:27.689 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:27.690 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/REFVQoqxVHmFXvE4csJtxWG2xo= 2024-03-04T17:53:27.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/REFVQoqxVHmFXvE4csJtxWG2xo=', 'Content-Length': '0'} 2024-03-04T17:53:27.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c00e5122d7ab70f5-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.693 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.693 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.693 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.693 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.693 DEBUG:boto:Method: GET 2024-03-04T17:53:27.693 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:27.693 DEBUG:boto:Data: 2024-03-04T17:53:27.693 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.694 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.694 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.694 DEBUG:boto:Params: {} 2024-03-04T17:53:27.694 DEBUG:boto:Token: None 2024-03-04T17:53:27.694 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.694 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f11949d5a22e5ba-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.699 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.700 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.700 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.700 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.700 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.700 DEBUG:boto:Method: GET 2024-03-04T17:53:27.700 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:27.700 DEBUG:boto:Data: 2024-03-04T17:53:27.700 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.700 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.700 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.701 DEBUG:boto:Params: {} 2024-03-04T17:53:27.701 DEBUG:boto:Token: None 2024-03-04T17:53:27.701 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.701 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001df29c1dce04db0a-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.748 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.748 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.748 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.749 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.749 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:27.749 DEBUG:boto:Data: 2024-03-04T17:53:27.749 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.749 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.749 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.749 DEBUG:boto:Params: {} 2024-03-04T17:53:27.749 DEBUG:boto:Token: None 2024-03-04T17:53:27.749 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj 2024-03-04T17:53:27.749 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lGGG2QYD50t4lSsvH6oK77uX+rA= 2024-03-04T17:53:27.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lGGG2QYD50t4lSsvH6oK77uX+rA=', 'Content-Length': '0'} 2024-03-04T17:53:27.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045a2fa7887495e5b-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.756 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.756 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.756 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.756 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:27.757 DEBUG:boto:Data: 2024-03-04T17:53:27.757 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.757 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.757 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.757 DEBUG:boto:Params: {} 2024-03-04T17:53:27.757 DEBUG:boto:Token: None 2024-03-04T17:53:27.757 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj 2024-03-04T17:53:27.757 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lGGG2QYD50t4lSsvH6oK77uX+rA= 2024-03-04T17:53:27.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lGGG2QYD50t4lSsvH6oK77uX+rA=', 'Content-Length': '0'} 2024-03-04T17:53:27.763 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006892de9ee2616a78-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:27.764 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.764 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.764 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.764 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.764 DEBUG:boto:Method: GET 2024-03-04T17:53:27.764 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.764 DEBUG:boto:Data: 2024-03-04T17:53:27.765 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.765 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.765 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.765 DEBUG:boto:Params: {} 2024-03-04T17:53:27.765 DEBUG:boto:Token: None 2024-03-04T17:53:27.765 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.765 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto= 2024-03-04T17:53:27.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto=', 'Content-Length': '0'} 2024-03-04T17:53:27.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018849e0b7335ec9f-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.768 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.769 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.769 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.769 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.769 DEBUG:boto:Method: GET 2024-03-04T17:53:27.769 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.769 DEBUG:boto:Data: 2024-03-04T17:53:27.769 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.769 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.769 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.770 DEBUG:boto:Params: {} 2024-03-04T17:53:27.770 DEBUG:boto:Token: None 2024-03-04T17:53:27.770 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.770 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto= 2024-03-04T17:53:27.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto=', 'Content-Length': '0'} 2024-03-04T17:53:27.773 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eee73c0462916dfd-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.773 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.773 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.773 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.773 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.774 DEBUG:boto:Method: GET 2024-03-04T17:53:27.774 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:27.774 DEBUG:boto:Data: 2024-03-04T17:53:27.774 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.774 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.774 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.774 DEBUG:boto:Params: {} 2024-03-04T17:53:27.774 DEBUG:boto:Token: None 2024-03-04T17:53:27.774 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.774 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0191a989fe06afd-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.820 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.820 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.821 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.821 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.821 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.821 DEBUG:boto:Method: GET 2024-03-04T17:53:27.821 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:27.821 DEBUG:boto:Data: 2024-03-04T17:53:27.821 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.821 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.821 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.821 DEBUG:boto:Params: {} 2024-03-04T17:53:27.821 DEBUG:boto:Token: None 2024-03-04T17:53:27.822 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.822 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaa7452b6cc303e7-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.827 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:27.828 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.828 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.828 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.828 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.828 DEBUG:boto:Method: GET 2024-03-04T17:53:27.828 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.828 DEBUG:boto:Data: 2024-03-04T17:53:27.828 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.828 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.829 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.829 DEBUG:boto:Params: {} 2024-03-04T17:53:27.829 DEBUG:boto:Token: None 2024-03-04T17:53:27.829 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.829 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+kEadMI1z/MgC/WtbaknsMXWiK0= 2024-03-04T17:53:27.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+kEadMI1z/MgC/WtbaknsMXWiK0=', 'Content-Length': '0'} 2024-03-04T17:53:27.832 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bef338f92878f5ad-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.832 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.832 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.832 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.833 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.833 DEBUG:boto:Method: GET 2024-03-04T17:53:27.833 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.833 DEBUG:boto:Data: 2024-03-04T17:53:27.833 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.833 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.833 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.833 DEBUG:boto:Params: {} 2024-03-04T17:53:27.833 DEBUG:boto:Token: None 2024-03-04T17:53:27.834 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.834 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+kEadMI1z/MgC/WtbaknsMXWiK0= 2024-03-04T17:53:27.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+kEadMI1z/MgC/WtbaknsMXWiK0=', 'Content-Length': '0'} 2024-03-04T17:53:27.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000141ed6e518e1a37b-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.880 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.880 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.880 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.880 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.880 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.880 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.881 DEBUG:boto:Data: 2024-03-04T17:53:27.881 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.881 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.881 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.881 DEBUG:boto:Params: {} 2024-03-04T17:53:27.881 DEBUG:boto:Token: None 2024-03-04T17:53:27.881 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.881 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:47h+vQrE/iXZfp8UQW1mN5182ko= 2024-03-04T17:53:27.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:47h+vQrE/iXZfp8UQW1mN5182ko=', 'Content-Length': '0'} 2024-03-04T17:53:27.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f96827a264676e6f-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.884 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.885 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.885 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.885 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.885 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.885 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.885 DEBUG:boto:Data: 2024-03-04T17:53:27.885 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.885 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.885 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.885 DEBUG:boto:Params: {} 2024-03-04T17:53:27.886 DEBUG:boto:Token: None 2024-03-04T17:53:27.886 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.886 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:47h+vQrE/iXZfp8UQW1mN5182ko= 2024-03-04T17:53:27.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:47h+vQrE/iXZfp8UQW1mN5182ko=', 'Content-Length': '0'} 2024-03-04T17:53:27.888 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1b4139b841a0782-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:27.889 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.889 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.889 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.889 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.889 DEBUG:boto:Method: GET 2024-03-04T17:53:27.890 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.890 DEBUG:boto:Data: 2024-03-04T17:53:27.890 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.890 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.890 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.890 DEBUG:boto:Params: {} 2024-03-04T17:53:27.890 DEBUG:boto:Token: None 2024-03-04T17:53:27.890 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.890 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+kEadMI1z/MgC/WtbaknsMXWiK0= 2024-03-04T17:53:27.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+kEadMI1z/MgC/WtbaknsMXWiK0=', 'Content-Length': '0'} 2024-03-04T17:53:27.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011eeafdd52fd8292-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.893 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.894 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.894 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.894 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.894 DEBUG:boto:Method: GET 2024-03-04T17:53:27.894 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.894 DEBUG:boto:Data: 2024-03-04T17:53:27.894 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.894 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.894 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.894 DEBUG:boto:Params: {} 2024-03-04T17:53:27.895 DEBUG:boto:Token: None 2024-03-04T17:53:27.895 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:27.895 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+kEadMI1z/MgC/WtbaknsMXWiK0= 2024-03-04T17:53:27.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+kEadMI1z/MgC/WtbaknsMXWiK0=', 'Content-Length': '0'} 2024-03-04T17:53:27.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a68552bbdbe294b1-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.898 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.898 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.898 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.898 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.898 DEBUG:boto:Method: GET 2024-03-04T17:53:27.898 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:27.898 DEBUG:boto:Data: 2024-03-04T17:53:27.898 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.899 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.899 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.899 DEBUG:boto:Params: {} 2024-03-04T17:53:27.899 DEBUG:boto:Token: None 2024-03-04T17:53:27.899 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.899 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c1d5013e0752955-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.944 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.944 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:27.945 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:27.945 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.945 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:27.945 DEBUG:boto:Method: GET 2024-03-04T17:53:27.945 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:27.945 DEBUG:boto:Data: 2024-03-04T17:53:27.945 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.945 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.945 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.945 DEBUG:boto:Params: {} 2024-03-04T17:53:27.946 DEBUG:boto:Token: None 2024-03-04T17:53:27.946 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/ 2024-03-04T17:53:27.946 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ= 2024-03-04T17:53:27.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:INo08NDnfA49Bv8bgDHEzZ2TfRQ=', 'Content-Length': '0'} 2024-03-04T17:53:27.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e58539e58526ea67-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.951 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:27.952 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.952 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.952 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.952 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:27.952 DEBUG:boto:Data: 2024-03-04T17:53:27.952 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.952 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.952 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.953 DEBUG:boto:Params: {} 2024-03-04T17:53:27.953 DEBUG:boto:Token: None 2024-03-04T17:53:27.953 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj 2024-03-04T17:53:27.953 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lGGG2QYD50t4lSsvH6oK77uX+rA= 2024-03-04T17:53:27.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lGGG2QYD50t4lSsvH6oK77uX+rA=', 'Content-Length': '0'} 2024-03-04T17:53:27.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6afb70939778dd3-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.959 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.959 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.959 DEBUG:boto:Method: HEAD 2024-03-04T17:53:27.959 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:27.959 DEBUG:boto:Data: 2024-03-04T17:53:27.959 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.960 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.960 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.960 DEBUG:boto:Params: {} 2024-03-04T17:53:27.960 DEBUG:boto:Token: None 2024-03-04T17:53:27.960 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj 2024-03-04T17:53:27.960 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lGGG2QYD50t4lSsvH6oK77uX+rA= 2024-03-04T17:53:27.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lGGG2QYD50t4lSsvH6oK77uX+rA=', 'Content-Length': '0'} 2024-03-04T17:53:27.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db9a137fd28794be-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:27.968 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.968 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.968 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.968 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.968 DEBUG:boto:Method: GET 2024-03-04T17:53:27.968 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.968 DEBUG:boto:Data: 2024-03-04T17:53:27.968 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.968 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:27.968 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.969 DEBUG:boto:Params: {} 2024-03-04T17:53:27.969 DEBUG:boto:Token: None 2024-03-04T17:53:27.969 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.969 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto= 2024-03-04T17:53:27.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto=', 'Content-Length': '0'} 2024-03-04T17:53:27.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008135dcc9e98f3ed2-0065e60a97-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:27.972 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:27.972 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:27.972 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.972 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.972 DEBUG:boto:Method: GET 2024-03-04T17:53:27.973 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.973 DEBUG:boto:Data: 2024-03-04T17:53:27.973 DEBUG:boto:Headers: {} 2024-03-04T17:53:27.973 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:27.973 DEBUG:boto:Port: 8000 2024-03-04T17:53:27.973 DEBUG:boto:Params: {} 2024-03-04T17:53:27.973 DEBUG:boto:Token: None 2024-03-04T17:53:27.973 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:27 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:27.973 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto= 2024-03-04T17:53:27.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NHZtisoGHCVcT03fe497OFyOdto=', 'Content-Length': '0'} 2024-03-04T17:53:27.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1555fd4b2bad822-0065e60a97-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.024 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.024 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.024 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.024 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.025 DEBUG:boto:Method: GET 2024-03-04T17:53:28.025 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.025 DEBUG:boto:Data: 2024-03-04T17:53:28.025 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.025 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.025 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.025 DEBUG:boto:Params: {} 2024-03-04T17:53:28.025 DEBUG:boto:Token: None 2024-03-04T17:53:28.025 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.025 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000280a3cb4d006960a-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.030 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.030 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.030 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.031 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.031 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.031 DEBUG:boto:Method: GET 2024-03-04T17:53:28.031 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.031 DEBUG:boto:Data: 2024-03-04T17:53:28.031 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.031 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.031 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.031 DEBUG:boto:Params: {} 2024-03-04T17:53:28.031 DEBUG:boto:Token: None 2024-03-04T17:53:28.031 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.032 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000522a6c19f595e864-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.036 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.037 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.037 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.037 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.037 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.037 DEBUG:boto:Method: GET 2024-03-04T17:53:28.037 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.038 DEBUG:boto:Data: 2024-03-04T17:53:28.038 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.038 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.038 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.038 DEBUG:boto:Params: {} 2024-03-04T17:53:28.038 DEBUG:boto:Token: None 2024-03-04T17:53:28.038 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.038 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:socAGfHPNAjxJYbSvH+5jAZ6znk= 2024-03-04T17:53:28.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:socAGfHPNAjxJYbSvH+5jAZ6znk=', 'Content-Length': '0'} 2024-03-04T17:53:28.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005040e82ee9948380-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.084 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.084 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.084 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.084 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.084 DEBUG:boto:Method: GET 2024-03-04T17:53:28.084 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.085 DEBUG:boto:Data: 2024-03-04T17:53:28.085 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.085 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.085 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.085 DEBUG:boto:Params: {} 2024-03-04T17:53:28.085 DEBUG:boto:Token: None 2024-03-04T17:53:28.085 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.085 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:socAGfHPNAjxJYbSvH+5jAZ6znk= 2024-03-04T17:53:28.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:socAGfHPNAjxJYbSvH+5jAZ6znk=', 'Content-Length': '0'} 2024-03-04T17:53:28.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056857d31d1a825cc-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.088 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.088 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.089 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.089 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.089 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.089 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.089 DEBUG:boto:Data: 2024-03-04T17:53:28.089 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.089 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.089 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.089 DEBUG:boto:Params: {} 2024-03-04T17:53:28.089 DEBUG:boto:Token: None 2024-03-04T17:53:28.090 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.090 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:igtvUQcSMo9V9kbtWrpjun6cSR8= 2024-03-04T17:53:28.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:igtvUQcSMo9V9kbtWrpjun6cSR8=', 'Content-Length': '0'} 2024-03-04T17:53:28.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c24d3807ae25059-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.093 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.093 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.093 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.093 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.093 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.093 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.094 DEBUG:boto:Data: 2024-03-04T17:53:28.094 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.094 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.094 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.094 DEBUG:boto:Params: {} 2024-03-04T17:53:28.094 DEBUG:boto:Token: None 2024-03-04T17:53:28.094 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.094 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:igtvUQcSMo9V9kbtWrpjun6cSR8= 2024-03-04T17:53:28.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:igtvUQcSMo9V9kbtWrpjun6cSR8=', 'Content-Length': '0'} 2024-03-04T17:53:28.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed007873bfc9b11f-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.098 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.098 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.098 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.098 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.098 DEBUG:boto:Method: GET 2024-03-04T17:53:28.098 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.098 DEBUG:boto:Data: 2024-03-04T17:53:28.098 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.098 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.098 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.098 DEBUG:boto:Params: {} 2024-03-04T17:53:28.099 DEBUG:boto:Token: None 2024-03-04T17:53:28.099 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.099 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:socAGfHPNAjxJYbSvH+5jAZ6znk= 2024-03-04T17:53:28.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:socAGfHPNAjxJYbSvH+5jAZ6znk=', 'Content-Length': '0'} 2024-03-04T17:53:28.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a45286a7bad9179-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.102 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.102 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.102 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.102 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.102 DEBUG:boto:Method: GET 2024-03-04T17:53:28.102 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.102 DEBUG:boto:Data: 2024-03-04T17:53:28.103 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.103 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.103 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.103 DEBUG:boto:Params: {} 2024-03-04T17:53:28.103 DEBUG:boto:Token: None 2024-03-04T17:53:28.103 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:28.103 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:socAGfHPNAjxJYbSvH+5jAZ6znk= 2024-03-04T17:53:28.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:socAGfHPNAjxJYbSvH+5jAZ6znk=', 'Content-Length': '0'} 2024-03-04T17:53:28.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aee80740cacb1c63-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.148 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.148 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.148 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.148 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.148 DEBUG:boto:Method: GET 2024-03-04T17:53:28.149 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.149 DEBUG:boto:Data: 2024-03-04T17:53:28.149 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.149 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.149 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.149 DEBUG:boto:Params: {} 2024-03-04T17:53:28.149 DEBUG:boto:Token: None 2024-03-04T17:53:28.149 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.149 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c9fcdb16c75a598-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.154 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.155 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.155 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.155 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.155 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.155 DEBUG:boto:Method: GET 2024-03-04T17:53:28.155 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.155 DEBUG:boto:Data: 2024-03-04T17:53:28.155 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.155 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.156 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.156 DEBUG:boto:Params: {} 2024-03-04T17:53:28.156 DEBUG:boto:Token: None 2024-03-04T17:53:28.156 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.156 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007362311e19864b44-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.161 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.161 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.161 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.161 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.161 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:28.162 DEBUG:boto:Data: 2024-03-04T17:53:28.162 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.162 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.162 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.162 DEBUG:boto:Params: {} 2024-03-04T17:53:28.162 DEBUG:boto:Token: None 2024-03-04T17:53:28.162 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj 2024-03-04T17:53:28.162 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0= 2024-03-04T17:53:28.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0=', 'Content-Length': '0'} 2024-03-04T17:53:28.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056d603cf9dd1a825-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.171 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.171 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.171 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.171 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:28.171 DEBUG:boto:Data: 2024-03-04T17:53:28.171 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.171 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.172 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.172 DEBUG:boto:Params: {} 2024-03-04T17:53:28.172 DEBUG:boto:Token: None 2024-03-04T17:53:28.172 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj 2024-03-04T17:53:28.172 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0= 2024-03-04T17:53:28.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0=', 'Content-Length': '0'} 2024-03-04T17:53:28.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053de6b4a0f333e03-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.179 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.179 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.179 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.179 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.179 DEBUG:boto:Method: GET 2024-03-04T17:53:28.179 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.179 DEBUG:boto:Data: 2024-03-04T17:53:28.179 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.179 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.179 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.180 DEBUG:boto:Params: {} 2024-03-04T17:53:28.180 DEBUG:boto:Token: None 2024-03-04T17:53:28.180 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.180 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg= 2024-03-04T17:53:28.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg=', 'Content-Length': '0'} 2024-03-04T17:53:28.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055e41ee90fc3890c-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.228 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.228 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.228 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.228 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.228 DEBUG:boto:Method: GET 2024-03-04T17:53:28.228 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.229 DEBUG:boto:Data: 2024-03-04T17:53:28.229 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.229 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.229 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.229 DEBUG:boto:Params: {} 2024-03-04T17:53:28.229 DEBUG:boto:Token: None 2024-03-04T17:53:28.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.229 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg= 2024-03-04T17:53:28.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg=', 'Content-Length': '0'} 2024-03-04T17:53:28.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad708aad39815c19-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.233 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.233 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.233 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.233 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.233 DEBUG:boto:Method: GET 2024-03-04T17:53:28.233 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.234 DEBUG:boto:Data: 2024-03-04T17:53:28.234 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.234 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.234 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.234 DEBUG:boto:Params: {} 2024-03-04T17:53:28.234 DEBUG:boto:Token: None 2024-03-04T17:53:28.234 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.234 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d4684b52c4bad0e-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.241 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.241 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.242 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.242 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.242 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.242 DEBUG:boto:Method: GET 2024-03-04T17:53:28.242 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.242 DEBUG:boto:Data: 2024-03-04T17:53:28.242 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.242 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.242 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.243 DEBUG:boto:Params: {} 2024-03-04T17:53:28.243 DEBUG:boto:Token: None 2024-03-04T17:53:28.243 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.243 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056f008c475e07612-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.288 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.288 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.289 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.289 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.289 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.289 DEBUG:boto:Method: GET 2024-03-04T17:53:28.289 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.289 DEBUG:boto:Data: 2024-03-04T17:53:28.289 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.289 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.289 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.289 DEBUG:boto:Params: {} 2024-03-04T17:53:28.289 DEBUG:boto:Token: None 2024-03-04T17:53:28.290 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.290 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xRlCAT/wTpXA5oMwqt5J5tUxERk= 2024-03-04T17:53:28.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xRlCAT/wTpXA5oMwqt5J5tUxERk=', 'Content-Length': '0'} 2024-03-04T17:53:28.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030794ee696717725-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.293 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.293 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.293 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.293 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.294 DEBUG:boto:Method: GET 2024-03-04T17:53:28.294 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.294 DEBUG:boto:Data: 2024-03-04T17:53:28.294 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.294 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.294 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.294 DEBUG:boto:Params: {} 2024-03-04T17:53:28.294 DEBUG:boto:Token: None 2024-03-04T17:53:28.294 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.295 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xRlCAT/wTpXA5oMwqt5J5tUxERk= 2024-03-04T17:53:28.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xRlCAT/wTpXA5oMwqt5J5tUxERk=', 'Content-Length': '0'} 2024-03-04T17:53:28.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087063cb438dffe12-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.298 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.298 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.298 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.299 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.299 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.299 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.299 DEBUG:boto:Data: 2024-03-04T17:53:28.299 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.299 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.299 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.299 DEBUG:boto:Params: {} 2024-03-04T17:53:28.299 DEBUG:boto:Token: None 2024-03-04T17:53:28.300 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.300 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dGfrZJbNyIOQ3ASi165eebFBX6U= 2024-03-04T17:53:28.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dGfrZJbNyIOQ3ASi165eebFBX6U=', 'Content-Length': '0'} 2024-03-04T17:53:28.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9a2c3efd352eceb-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.303 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.303 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.303 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.303 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.303 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.303 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.303 DEBUG:boto:Data: 2024-03-04T17:53:28.303 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.304 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.304 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.304 DEBUG:boto:Params: {} 2024-03-04T17:53:28.304 DEBUG:boto:Token: None 2024-03-04T17:53:28.304 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.304 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dGfrZJbNyIOQ3ASi165eebFBX6U= 2024-03-04T17:53:28.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dGfrZJbNyIOQ3ASi165eebFBX6U=', 'Content-Length': '0'} 2024-03-04T17:53:28.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe9a745a2233c8f4-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.308 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.308 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.308 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.308 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.308 DEBUG:boto:Method: GET 2024-03-04T17:53:28.308 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.308 DEBUG:boto:Data: 2024-03-04T17:53:28.308 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.308 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.308 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.309 DEBUG:boto:Params: {} 2024-03-04T17:53:28.309 DEBUG:boto:Token: None 2024-03-04T17:53:28.309 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.309 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xRlCAT/wTpXA5oMwqt5J5tUxERk= 2024-03-04T17:53:28.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xRlCAT/wTpXA5oMwqt5J5tUxERk=', 'Content-Length': '0'} 2024-03-04T17:53:28.312 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000120cbe04bcebd09c-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.312 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.312 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.312 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.312 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.312 DEBUG:boto:Method: GET 2024-03-04T17:53:28.313 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.313 DEBUG:boto:Data: 2024-03-04T17:53:28.313 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.313 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.313 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.313 DEBUG:boto:Params: {} 2024-03-04T17:53:28.313 DEBUG:boto:Token: None 2024-03-04T17:53:28.313 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:28.313 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xRlCAT/wTpXA5oMwqt5J5tUxERk= 2024-03-04T17:53:28.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xRlCAT/wTpXA5oMwqt5J5tUxERk=', 'Content-Length': '0'} 2024-03-04T17:53:28.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005c6fedffff29453-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.316 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.316 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.317 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.317 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.317 DEBUG:boto:Method: GET 2024-03-04T17:53:28.317 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.317 DEBUG:boto:Data: 2024-03-04T17:53:28.317 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.317 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.317 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.317 DEBUG:boto:Params: {} 2024-03-04T17:53:28.317 DEBUG:boto:Token: None 2024-03-04T17:53:28.318 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.318 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd8e701a2e8dcd61-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.364 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.364 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.364 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.365 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.365 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.365 DEBUG:boto:Method: GET 2024-03-04T17:53:28.365 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.365 DEBUG:boto:Data: 2024-03-04T17:53:28.365 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.365 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.365 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.365 DEBUG:boto:Params: {} 2024-03-04T17:53:28.366 DEBUG:boto:Token: None 2024-03-04T17:53:28.366 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.366 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc40d16dd5eb4635-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.371 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.372 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.372 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.372 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.372 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:28.372 DEBUG:boto:Data: 2024-03-04T17:53:28.372 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.372 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.372 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.372 DEBUG:boto:Params: {} 2024-03-04T17:53:28.372 DEBUG:boto:Token: None 2024-03-04T17:53:28.373 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj 2024-03-04T17:53:28.373 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0= 2024-03-04T17:53:28.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0=', 'Content-Length': '0'} 2024-03-04T17:53:28.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b99fa254584e3e5-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.379 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.379 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.379 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.379 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:28.379 DEBUG:boto:Data: 2024-03-04T17:53:28.380 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.380 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.380 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.380 DEBUG:boto:Params: {} 2024-03-04T17:53:28.380 DEBUG:boto:Token: None 2024-03-04T17:53:28.380 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj 2024-03-04T17:53:28.380 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0= 2024-03-04T17:53:28.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0=', 'Content-Length': '0'} 2024-03-04T17:53:28.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cddeb250f7a26212-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.386 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.386 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.387 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.387 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.387 DEBUG:boto:Method: GET 2024-03-04T17:53:28.387 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.387 DEBUG:boto:Data: 2024-03-04T17:53:28.387 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.387 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.387 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.387 DEBUG:boto:Params: {} 2024-03-04T17:53:28.387 DEBUG:boto:Token: None 2024-03-04T17:53:28.388 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.388 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg= 2024-03-04T17:53:28.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg=', 'Content-Length': '0'} 2024-03-04T17:53:28.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc6d9a600db194cf-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.391 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.391 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.391 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.392 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.392 DEBUG:boto:Method: GET 2024-03-04T17:53:28.392 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.392 DEBUG:boto:Data: 2024-03-04T17:53:28.392 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.392 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.392 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.392 DEBUG:boto:Params: {} 2024-03-04T17:53:28.392 DEBUG:boto:Token: None 2024-03-04T17:53:28.393 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.393 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg= 2024-03-04T17:53:28.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg=', 'Content-Length': '0'} 2024-03-04T17:53:28.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005193a425a1adfd9c-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.436 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.436 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.436 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.436 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.437 DEBUG:boto:Method: GET 2024-03-04T17:53:28.437 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.437 DEBUG:boto:Data: 2024-03-04T17:53:28.437 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.437 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.437 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.437 DEBUG:boto:Params: {} 2024-03-04T17:53:28.437 DEBUG:boto:Token: None 2024-03-04T17:53:28.437 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.438 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082ddb83cc96088c3-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.442 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.443 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.443 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.443 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.443 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.443 DEBUG:boto:Method: GET 2024-03-04T17:53:28.443 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.443 DEBUG:boto:Data: 2024-03-04T17:53:28.443 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.444 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.444 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.444 DEBUG:boto:Params: {} 2024-03-04T17:53:28.444 DEBUG:boto:Token: None 2024-03-04T17:53:28.444 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.444 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d54831b9185375bc-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.449 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.450 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.450 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.450 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.450 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.450 DEBUG:boto:Method: GET 2024-03-04T17:53:28.450 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.450 DEBUG:boto:Data: 2024-03-04T17:53:28.450 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.450 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.450 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.450 DEBUG:boto:Params: {} 2024-03-04T17:53:28.451 DEBUG:boto:Token: None 2024-03-04T17:53:28.451 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.451 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KnCoXVJqhijUXmyLJCjnaHPJPoA= 2024-03-04T17:53:28.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KnCoXVJqhijUXmyLJCjnaHPJPoA=', 'Content-Length': '0'} 2024-03-04T17:53:28.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007daf975a2f36d8d3-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.496 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.496 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.496 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.496 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.496 DEBUG:boto:Method: GET 2024-03-04T17:53:28.497 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.497 DEBUG:boto:Data: 2024-03-04T17:53:28.497 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.497 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.497 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.497 DEBUG:boto:Params: {} 2024-03-04T17:53:28.497 DEBUG:boto:Token: None 2024-03-04T17:53:28.497 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.497 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KnCoXVJqhijUXmyLJCjnaHPJPoA= 2024-03-04T17:53:28.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KnCoXVJqhijUXmyLJCjnaHPJPoA=', 'Content-Length': '0'} 2024-03-04T17:53:28.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089926482adc22de0-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.501 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.501 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.501 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.501 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.501 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.501 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.501 DEBUG:boto:Data: 2024-03-04T17:53:28.501 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.501 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.502 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.502 DEBUG:boto:Params: {} 2024-03-04T17:53:28.502 DEBUG:boto:Token: None 2024-03-04T17:53:28.502 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.502 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:a+77qUWQP5iUJTXjGFIuyZo9Yh8= 2024-03-04T17:53:28.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:a+77qUWQP5iUJTXjGFIuyZo9Yh8=', 'Content-Length': '0'} 2024-03-04T17:53:28.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbd69a593d078a4c-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.505 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.505 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.505 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.505 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.506 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.506 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.506 DEBUG:boto:Data: 2024-03-04T17:53:28.506 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.506 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.506 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.506 DEBUG:boto:Params: {} 2024-03-04T17:53:28.506 DEBUG:boto:Token: None 2024-03-04T17:53:28.506 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.506 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:a+77qUWQP5iUJTXjGFIuyZo9Yh8= 2024-03-04T17:53:28.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:a+77qUWQP5iUJTXjGFIuyZo9Yh8=', 'Content-Length': '0'} 2024-03-04T17:53:28.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000159657bc2c5b29c8-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.510 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.510 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.510 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.510 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.510 DEBUG:boto:Method: GET 2024-03-04T17:53:28.510 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.510 DEBUG:boto:Data: 2024-03-04T17:53:28.510 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.510 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.511 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.511 DEBUG:boto:Params: {} 2024-03-04T17:53:28.511 DEBUG:boto:Token: None 2024-03-04T17:53:28.511 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.511 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KnCoXVJqhijUXmyLJCjnaHPJPoA= 2024-03-04T17:53:28.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KnCoXVJqhijUXmyLJCjnaHPJPoA=', 'Content-Length': '0'} 2024-03-04T17:53:28.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f15dc60bc1772d2f-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.514 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.514 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.514 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.515 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.515 DEBUG:boto:Method: GET 2024-03-04T17:53:28.515 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.515 DEBUG:boto:Data: 2024-03-04T17:53:28.515 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.515 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.515 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.515 DEBUG:boto:Params: {} 2024-03-04T17:53:28.515 DEBUG:boto:Token: None 2024-03-04T17:53:28.516 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:28.516 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KnCoXVJqhijUXmyLJCjnaHPJPoA= 2024-03-04T17:53:28.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KnCoXVJqhijUXmyLJCjnaHPJPoA=', 'Content-Length': '0'} 2024-03-04T17:53:28.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa21589e3dff335e-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.560 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.560 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.560 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.560 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.561 DEBUG:boto:Method: GET 2024-03-04T17:53:28.561 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.561 DEBUG:boto:Data: 2024-03-04T17:53:28.561 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.561 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.561 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.561 DEBUG:boto:Params: {} 2024-03-04T17:53:28.561 DEBUG:boto:Token: None 2024-03-04T17:53:28.561 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.562 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063ec82c9e94c997b-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.566 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.567 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.567 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.567 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.567 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.567 DEBUG:boto:Method: GET 2024-03-04T17:53:28.567 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.567 DEBUG:boto:Data: 2024-03-04T17:53:28.567 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.568 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.568 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.568 DEBUG:boto:Params: {} 2024-03-04T17:53:28.568 DEBUG:boto:Token: None 2024-03-04T17:53:28.568 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.568 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d31d7daf1dedc7e-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.574 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.574 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.574 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.574 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.574 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:28.575 DEBUG:boto:Data: 2024-03-04T17:53:28.575 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.575 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.575 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.575 DEBUG:boto:Params: {} 2024-03-04T17:53:28.575 DEBUG:boto:Token: None 2024-03-04T17:53:28.575 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj 2024-03-04T17:53:28.575 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0= 2024-03-04T17:53:28.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0=', 'Content-Length': '0'} 2024-03-04T17:53:28.583 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd0183b588d955bc-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.583 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.583 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.583 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.583 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:28.583 DEBUG:boto:Data: 2024-03-04T17:53:28.584 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.584 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.584 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.584 DEBUG:boto:Params: {} 2024-03-04T17:53:28.584 DEBUG:boto:Token: None 2024-03-04T17:53:28.584 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj 2024-03-04T17:53:28.584 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0= 2024-03-04T17:53:28.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0=', 'Content-Length': '0'} 2024-03-04T17:53:28.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cac5b289c79caeb-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.591 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.591 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.591 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.591 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.591 DEBUG:boto:Method: GET 2024-03-04T17:53:28.591 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.592 DEBUG:boto:Data: 2024-03-04T17:53:28.592 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.592 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.592 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.592 DEBUG:boto:Params: {} 2024-03-04T17:53:28.592 DEBUG:boto:Token: None 2024-03-04T17:53:28.592 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.592 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg= 2024-03-04T17:53:28.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg=', 'Content-Length': '0'} 2024-03-04T17:53:28.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dffbc3c593f85f79-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.636 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.636 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.636 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.636 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.637 DEBUG:boto:Method: GET 2024-03-04T17:53:28.637 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.637 DEBUG:boto:Data: 2024-03-04T17:53:28.637 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.637 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.637 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.637 DEBUG:boto:Params: {} 2024-03-04T17:53:28.637 DEBUG:boto:Token: None 2024-03-04T17:53:28.637 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.638 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg= 2024-03-04T17:53:28.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg=', 'Content-Length': '0'} 2024-03-04T17:53:28.641 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f775c9494e109bb6-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.642 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.642 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.642 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.642 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.642 DEBUG:boto:Method: GET 2024-03-04T17:53:28.642 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.642 DEBUG:boto:Data: 2024-03-04T17:53:28.642 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.642 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.643 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.643 DEBUG:boto:Params: {} 2024-03-04T17:53:28.643 DEBUG:boto:Token: None 2024-03-04T17:53:28.643 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.643 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d9ddbb1f70e498a-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.648 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.649 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.649 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.649 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.649 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.649 DEBUG:boto:Method: GET 2024-03-04T17:53:28.649 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.649 DEBUG:boto:Data: 2024-03-04T17:53:28.650 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.650 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.650 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.650 DEBUG:boto:Params: {} 2024-03-04T17:53:28.650 DEBUG:boto:Token: None 2024-03-04T17:53:28.650 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.650 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000617f6dc2e401edf8-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.700 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.700 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.701 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.701 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.701 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.701 DEBUG:boto:Method: GET 2024-03-04T17:53:28.701 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.701 DEBUG:boto:Data: 2024-03-04T17:53:28.701 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.701 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.701 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.701 DEBUG:boto:Params: {} 2024-03-04T17:53:28.702 DEBUG:boto:Token: None 2024-03-04T17:53:28.702 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.702 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sCwPLqJRdnKESTKOrjKcRbYHdSA= 2024-03-04T17:53:28.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sCwPLqJRdnKESTKOrjKcRbYHdSA=', 'Content-Length': '0'} 2024-03-04T17:53:28.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ae66a83af1ec096-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.705 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.705 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.705 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.706 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.706 DEBUG:boto:Method: GET 2024-03-04T17:53:28.706 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.706 DEBUG:boto:Data: 2024-03-04T17:53:28.706 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.706 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.706 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.706 DEBUG:boto:Params: {} 2024-03-04T17:53:28.706 DEBUG:boto:Token: None 2024-03-04T17:53:28.706 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.707 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sCwPLqJRdnKESTKOrjKcRbYHdSA= 2024-03-04T17:53:28.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sCwPLqJRdnKESTKOrjKcRbYHdSA=', 'Content-Length': '0'} 2024-03-04T17:53:28.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc7b06622fd7b29e-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.710 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.710 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.710 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.710 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.710 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.710 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.710 DEBUG:boto:Data: 2024-03-04T17:53:28.710 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.711 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.711 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.711 DEBUG:boto:Params: {} 2024-03-04T17:53:28.711 DEBUG:boto:Token: None 2024-03-04T17:53:28.711 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.711 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V7CfIFG4YIsa+Ox48VUHdcsE6mQ= 2024-03-04T17:53:28.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V7CfIFG4YIsa+Ox48VUHdcsE6mQ=', 'Content-Length': '0'} 2024-03-04T17:53:28.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c51ab0baa229e013-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.714 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.714 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.714 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.715 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.715 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.715 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.715 DEBUG:boto:Data: 2024-03-04T17:53:28.715 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.715 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.715 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.715 DEBUG:boto:Params: {} 2024-03-04T17:53:28.715 DEBUG:boto:Token: None 2024-03-04T17:53:28.715 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.716 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:V7CfIFG4YIsa+Ox48VUHdcsE6mQ= 2024-03-04T17:53:28.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:V7CfIFG4YIsa+Ox48VUHdcsE6mQ=', 'Content-Length': '0'} 2024-03-04T17:53:28.718 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bbe5d1c6eaeeb08-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.719 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.719 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.719 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.719 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.719 DEBUG:boto:Method: GET 2024-03-04T17:53:28.719 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.719 DEBUG:boto:Data: 2024-03-04T17:53:28.719 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.719 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.719 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.720 DEBUG:boto:Params: {} 2024-03-04T17:53:28.720 DEBUG:boto:Token: None 2024-03-04T17:53:28.720 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.720 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sCwPLqJRdnKESTKOrjKcRbYHdSA= 2024-03-04T17:53:28.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sCwPLqJRdnKESTKOrjKcRbYHdSA=', 'Content-Length': '0'} 2024-03-04T17:53:28.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005330869de0eb1f99-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.768 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.768 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.768 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.768 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.768 DEBUG:boto:Method: GET 2024-03-04T17:53:28.769 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.769 DEBUG:boto:Data: 2024-03-04T17:53:28.769 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.769 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.769 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.769 DEBUG:boto:Params: {} 2024-03-04T17:53:28.769 DEBUG:boto:Token: None 2024-03-04T17:53:28.769 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:28.769 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sCwPLqJRdnKESTKOrjKcRbYHdSA= 2024-03-04T17:53:28.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sCwPLqJRdnKESTKOrjKcRbYHdSA=', 'Content-Length': '0'} 2024-03-04T17:53:28.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fd32e7f03d7741f-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.772 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.773 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.773 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.773 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.773 DEBUG:boto:Method: GET 2024-03-04T17:53:28.773 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.773 DEBUG:boto:Data: 2024-03-04T17:53:28.773 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.773 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.773 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.773 DEBUG:boto:Params: {} 2024-03-04T17:53:28.774 DEBUG:boto:Token: None 2024-03-04T17:53:28.774 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.774 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acc92665e0ce09e8-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.778 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.779 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.779 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.779 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.779 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.779 DEBUG:boto:Method: GET 2024-03-04T17:53:28.779 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.779 DEBUG:boto:Data: 2024-03-04T17:53:28.779 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.780 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.780 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.780 DEBUG:boto:Params: {} 2024-03-04T17:53:28.780 DEBUG:boto:Token: None 2024-03-04T17:53:28.780 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.780 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1054af0a0e421e8-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.831 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.831 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.832 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.832 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.832 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:28.832 DEBUG:boto:Data: 2024-03-04T17:53:28.832 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.832 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.832 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.832 DEBUG:boto:Params: {} 2024-03-04T17:53:28.833 DEBUG:boto:Token: None 2024-03-04T17:53:28.833 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj 2024-03-04T17:53:28.833 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0= 2024-03-04T17:53:28.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0=', 'Content-Length': '0'} 2024-03-04T17:53:28.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cd22a97808d8e28-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.839 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.839 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.840 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.840 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:28.840 DEBUG:boto:Data: 2024-03-04T17:53:28.840 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.840 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.840 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.840 DEBUG:boto:Params: {} 2024-03-04T17:53:28.840 DEBUG:boto:Token: None 2024-03-04T17:53:28.841 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj 2024-03-04T17:53:28.841 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0= 2024-03-04T17:53:28.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Bk5+jCxsWIDnY12BxVJ2IpWj0=', 'Content-Length': '0'} 2024-03-04T17:53:28.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016dd66e0e7a53924-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.847 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.847 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.847 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.847 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.847 DEBUG:boto:Method: GET 2024-03-04T17:53:28.847 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.847 DEBUG:boto:Data: 2024-03-04T17:53:28.847 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.847 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.847 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.848 DEBUG:boto:Params: {} 2024-03-04T17:53:28.848 DEBUG:boto:Token: None 2024-03-04T17:53:28.848 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.848 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg= 2024-03-04T17:53:28.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg=', 'Content-Length': '0'} 2024-03-04T17:53:28.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000204c725df49f8bf3-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.852 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.852 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.852 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.852 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.852 DEBUG:boto:Method: GET 2024-03-04T17:53:28.852 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.852 DEBUG:boto:Data: 2024-03-04T17:53:28.852 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.852 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.853 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.853 DEBUG:boto:Params: {} 2024-03-04T17:53:28.853 DEBUG:boto:Token: None 2024-03-04T17:53:28.853 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:28.853 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg= 2024-03-04T17:53:28.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eTROqpjSuxypFci66cd2HNFS+rg=', 'Content-Length': '0'} 2024-03-04T17:53:28.856 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037d59715f00eaaed-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.857 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.857 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.857 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.857 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.857 DEBUG:boto:Method: GET 2024-03-04T17:53:28.857 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.857 DEBUG:boto:Data: 2024-03-04T17:53:28.857 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.857 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.858 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.858 DEBUG:boto:Params: {} 2024-03-04T17:53:28.858 DEBUG:boto:Token: None 2024-03-04T17:53:28.858 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.858 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e8bb836b3a87a6a-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.863 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.863 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.863 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.863 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.864 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.864 DEBUG:boto:Method: GET 2024-03-04T17:53:28.864 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.864 DEBUG:boto:Data: 2024-03-04T17:53:28.864 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.864 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.864 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.864 DEBUG:boto:Params: {} 2024-03-04T17:53:28.864 DEBUG:boto:Token: None 2024-03-04T17:53:28.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.865 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018a179b0bff31b14-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.912 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.912 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.913 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.913 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.913 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.913 DEBUG:boto:Method: GET 2024-03-04T17:53:28.913 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.913 DEBUG:boto:Data: 2024-03-04T17:53:28.913 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.913 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.913 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.913 DEBUG:boto:Params: {} 2024-03-04T17:53:28.914 DEBUG:boto:Token: None 2024-03-04T17:53:28.914 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.914 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nsGyM8C8Nr1UZsU5z51FHD4GuSw= 2024-03-04T17:53:28.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nsGyM8C8Nr1UZsU5z51FHD4GuSw=', 'Content-Length': '0'} 2024-03-04T17:53:28.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079a048ba2d8e3a69-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.917 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.917 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.917 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.918 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.918 DEBUG:boto:Method: GET 2024-03-04T17:53:28.918 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.918 DEBUG:boto:Data: 2024-03-04T17:53:28.918 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.918 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.918 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.918 DEBUG:boto:Params: {} 2024-03-04T17:53:28.918 DEBUG:boto:Token: None 2024-03-04T17:53:28.918 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.919 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nsGyM8C8Nr1UZsU5z51FHD4GuSw= 2024-03-04T17:53:28.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nsGyM8C8Nr1UZsU5z51FHD4GuSw=', 'Content-Length': '0'} 2024-03-04T17:53:28.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee0450d882a1ee5c-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.922 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.922 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.922 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.922 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.922 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.922 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.923 DEBUG:boto:Data: 2024-03-04T17:53:28.923 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.923 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.923 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.923 DEBUG:boto:Params: {} 2024-03-04T17:53:28.923 DEBUG:boto:Token: None 2024-03-04T17:53:28.923 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.923 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CKEcvkZWuMD5QAgqxjRKWtpIPZ4= 2024-03-04T17:53:28.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CKEcvkZWuMD5QAgqxjRKWtpIPZ4=', 'Content-Length': '0'} 2024-03-04T17:53:28.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ad0136e9a17762e-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.926 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.926 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.927 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.927 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.927 DEBUG:boto:Method: HEAD 2024-03-04T17:53:28.927 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.927 DEBUG:boto:Data: 2024-03-04T17:53:28.927 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.927 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.927 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.927 DEBUG:boto:Params: {} 2024-03-04T17:53:28.927 DEBUG:boto:Token: None 2024-03-04T17:53:28.928 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.928 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CKEcvkZWuMD5QAgqxjRKWtpIPZ4= 2024-03-04T17:53:28.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CKEcvkZWuMD5QAgqxjRKWtpIPZ4=', 'Content-Length': '0'} 2024-03-04T17:53:28.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab37e25a96fe7a79-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:28.931 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.931 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.931 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.931 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.931 DEBUG:boto:Method: GET 2024-03-04T17:53:28.932 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.932 DEBUG:boto:Data: 2024-03-04T17:53:28.932 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.932 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.932 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.932 DEBUG:boto:Params: {} 2024-03-04T17:53:28.932 DEBUG:boto:Token: None 2024-03-04T17:53:28.932 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.932 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nsGyM8C8Nr1UZsU5z51FHD4GuSw= 2024-03-04T17:53:28.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nsGyM8C8Nr1UZsU5z51FHD4GuSw=', 'Content-Length': '0'} 2024-03-04T17:53:28.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a691156463f57c7-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.976 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:28.976 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:28.976 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.976 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.976 DEBUG:boto:Method: GET 2024-03-04T17:53:28.976 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.977 DEBUG:boto:Data: 2024-03-04T17:53:28.977 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.977 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.977 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.977 DEBUG:boto:Params: {} 2024-03-04T17:53:28.977 DEBUG:boto:Token: None 2024-03-04T17:53:28.977 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:28.977 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nsGyM8C8Nr1UZsU5z51FHD4GuSw= 2024-03-04T17:53:28.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nsGyM8C8Nr1UZsU5z51FHD4GuSw=', 'Content-Length': '0'} 2024-03-04T17:53:28.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c17f4b1a123d5a7b-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.980 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.981 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.981 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.981 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.981 DEBUG:boto:Method: GET 2024-03-04T17:53:28.981 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.981 DEBUG:boto:Data: 2024-03-04T17:53:28.981 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.981 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:28.981 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.981 DEBUG:boto:Params: {} 2024-03-04T17:53:28.982 DEBUG:boto:Token: None 2024-03-04T17:53:28.982 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.982 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f573eb9d452039f-0065e60a98-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:28.987 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:28.987 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:28.987 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:28.987 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.987 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:28.988 DEBUG:boto:Method: GET 2024-03-04T17:53:28.988 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:28.988 DEBUG:boto:Data: 2024-03-04T17:53:28.988 DEBUG:boto:Headers: {} 2024-03-04T17:53:28.988 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:28.988 DEBUG:boto:Port: 8000 2024-03-04T17:53:28.988 DEBUG:boto:Params: {} 2024-03-04T17:53:28.988 DEBUG:boto:Token: None 2024-03-04T17:53:28.988 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:28 GMT /ppfahn-22/ 2024-03-04T17:53:28.989 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8= 2024-03-04T17:53:28.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DxPN9rGZ1tO+o6SIccgZQhcPIj8=', 'Content-Length': '0'} 2024-03-04T17:53:28.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a402ee07e8a95a16-0065e60a98-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.036 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.036 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.037 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.037 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.037 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:29.037 DEBUG:boto:Data: 2024-03-04T17:53:29.037 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.037 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.037 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.037 DEBUG:boto:Params: {} 2024-03-04T17:53:29.037 DEBUG:boto:Token: None 2024-03-04T17:53:29.037 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj 2024-03-04T17:53:29.038 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y= 2024-03-04T17:53:29.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y=', 'Content-Length': '0'} 2024-03-04T17:53:29.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000704efc6c6202384e-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.044 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.044 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.044 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.044 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:29.044 DEBUG:boto:Data: 2024-03-04T17:53:29.044 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.044 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.045 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.045 DEBUG:boto:Params: {} 2024-03-04T17:53:29.045 DEBUG:boto:Token: None 2024-03-04T17:53:29.045 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj 2024-03-04T17:53:29.045 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y= 2024-03-04T17:53:29.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y=', 'Content-Length': '0'} 2024-03-04T17:53:29.051 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7de004a4056056b-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.051 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.051 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.052 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.052 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.052 DEBUG:boto:Method: GET 2024-03-04T17:53:29.052 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.052 DEBUG:boto:Data: 2024-03-04T17:53:29.052 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.052 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.052 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.052 DEBUG:boto:Params: {} 2024-03-04T17:53:29.052 DEBUG:boto:Token: None 2024-03-04T17:53:29.053 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.053 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE= 2024-03-04T17:53:29.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE=', 'Content-Length': '0'} 2024-03-04T17:53:29.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be3d1ab5c257aaef-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.056 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.056 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.057 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.057 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.057 DEBUG:boto:Method: GET 2024-03-04T17:53:29.057 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.057 DEBUG:boto:Data: 2024-03-04T17:53:29.057 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.057 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.057 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.057 DEBUG:boto:Params: {} 2024-03-04T17:53:29.057 DEBUG:boto:Token: None 2024-03-04T17:53:29.058 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.058 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE= 2024-03-04T17:53:29.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE=', 'Content-Length': '0'} 2024-03-04T17:53:29.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081614547ac93ed6b-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.061 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.061 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.061 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.061 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.062 DEBUG:boto:Method: GET 2024-03-04T17:53:29.062 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.062 DEBUG:boto:Data: 2024-03-04T17:53:29.062 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.062 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.062 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.062 DEBUG:boto:Params: {} 2024-03-04T17:53:29.062 DEBUG:boto:Token: None 2024-03-04T17:53:29.062 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.062 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a4a301970114d43-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.108 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.108 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.109 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.109 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.109 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.109 DEBUG:boto:Method: GET 2024-03-04T17:53:29.109 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.109 DEBUG:boto:Data: 2024-03-04T17:53:29.109 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.109 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.109 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.109 DEBUG:boto:Params: {} 2024-03-04T17:53:29.109 DEBUG:boto:Token: None 2024-03-04T17:53:29.110 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.110 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ba4a7283f226936-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.115 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.115 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.115 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.115 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.115 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.115 DEBUG:boto:Method: GET 2024-03-04T17:53:29.116 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.116 DEBUG:boto:Data: 2024-03-04T17:53:29.116 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.116 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.116 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.116 DEBUG:boto:Params: {} 2024-03-04T17:53:29.116 DEBUG:boto:Token: None 2024-03-04T17:53:29.116 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.116 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0nXYO4BfSNqwBXZg/xpCManVWT4= 2024-03-04T17:53:29.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0nXYO4BfSNqwBXZg/xpCManVWT4=', 'Content-Length': '0'} 2024-03-04T17:53:29.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008dc9a2bf2a95f75-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.120 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.120 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.120 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.120 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.121 DEBUG:boto:Method: GET 2024-03-04T17:53:29.121 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.121 DEBUG:boto:Data: 2024-03-04T17:53:29.121 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.121 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.121 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.121 DEBUG:boto:Params: {} 2024-03-04T17:53:29.121 DEBUG:boto:Token: None 2024-03-04T17:53:29.121 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.122 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0nXYO4BfSNqwBXZg/xpCManVWT4= 2024-03-04T17:53:29.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0nXYO4BfSNqwBXZg/xpCManVWT4=', 'Content-Length': '0'} 2024-03-04T17:53:29.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007176dc809f333b45-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.168 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.168 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.168 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.168 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.169 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.169 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.169 DEBUG:boto:Data: 2024-03-04T17:53:29.169 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.169 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.169 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.169 DEBUG:boto:Params: {} 2024-03-04T17:53:29.169 DEBUG:boto:Token: None 2024-03-04T17:53:29.169 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.169 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:g/ER7NUp8JVGqLAIqZ2aWfd4//4= 2024-03-04T17:53:29.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:g/ER7NUp8JVGqLAIqZ2aWfd4//4=', 'Content-Length': '0'} 2024-03-04T17:53:29.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1cdf7c24b54e94b-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.173 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.173 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.173 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.173 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.173 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.173 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.174 DEBUG:boto:Data: 2024-03-04T17:53:29.174 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.174 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.174 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.174 DEBUG:boto:Params: {} 2024-03-04T17:53:29.174 DEBUG:boto:Token: None 2024-03-04T17:53:29.174 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.174 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:g/ER7NUp8JVGqLAIqZ2aWfd4//4= 2024-03-04T17:53:29.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:g/ER7NUp8JVGqLAIqZ2aWfd4//4=', 'Content-Length': '0'} 2024-03-04T17:53:29.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039bb5b558fe5e5ba-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.177 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.177 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.178 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.178 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.178 DEBUG:boto:Method: GET 2024-03-04T17:53:29.178 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.178 DEBUG:boto:Data: 2024-03-04T17:53:29.178 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.178 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.178 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.178 DEBUG:boto:Params: {} 2024-03-04T17:53:29.178 DEBUG:boto:Token: None 2024-03-04T17:53:29.179 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.179 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0nXYO4BfSNqwBXZg/xpCManVWT4= 2024-03-04T17:53:29.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0nXYO4BfSNqwBXZg/xpCManVWT4=', 'Content-Length': '0'} 2024-03-04T17:53:29.182 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb2ab4a014c97722-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.182 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.182 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.182 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.182 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.182 DEBUG:boto:Method: GET 2024-03-04T17:53:29.183 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.183 DEBUG:boto:Data: 2024-03-04T17:53:29.183 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.183 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.183 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.183 DEBUG:boto:Params: {} 2024-03-04T17:53:29.183 DEBUG:boto:Token: None 2024-03-04T17:53:29.183 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:29.183 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0nXYO4BfSNqwBXZg/xpCManVWT4= 2024-03-04T17:53:29.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0nXYO4BfSNqwBXZg/xpCManVWT4=', 'Content-Length': '0'} 2024-03-04T17:53:29.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c78f1e696bf2242e-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.187 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.187 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.187 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.187 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.187 DEBUG:boto:Method: GET 2024-03-04T17:53:29.188 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.188 DEBUG:boto:Data: 2024-03-04T17:53:29.188 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.188 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.188 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.188 DEBUG:boto:Params: {} 2024-03-04T17:53:29.188 DEBUG:boto:Token: None 2024-03-04T17:53:29.189 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.189 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a8a8fb473994153-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.236 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.236 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.237 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.237 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.237 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.237 DEBUG:boto:Method: GET 2024-03-04T17:53:29.237 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.237 DEBUG:boto:Data: 2024-03-04T17:53:29.237 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.237 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.237 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.237 DEBUG:boto:Params: {} 2024-03-04T17:53:29.238 DEBUG:boto:Token: None 2024-03-04T17:53:29.238 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.238 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006145644a20628768-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.244 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.244 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.244 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.244 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.244 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:29.244 DEBUG:boto:Data: 2024-03-04T17:53:29.245 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.245 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.245 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.245 DEBUG:boto:Params: {} 2024-03-04T17:53:29.245 DEBUG:boto:Token: None 2024-03-04T17:53:29.245 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj 2024-03-04T17:53:29.245 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y= 2024-03-04T17:53:29.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y=', 'Content-Length': '0'} 2024-03-04T17:53:29.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec42fdcb934043cd-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.252 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.252 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.253 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.253 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:29.253 DEBUG:boto:Data: 2024-03-04T17:53:29.253 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.253 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.253 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.253 DEBUG:boto:Params: {} 2024-03-04T17:53:29.253 DEBUG:boto:Token: None 2024-03-04T17:53:29.254 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj 2024-03-04T17:53:29.254 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y= 2024-03-04T17:53:29.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y=', 'Content-Length': '0'} 2024-03-04T17:53:29.260 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039750631fe001eaf-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.260 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.261 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.261 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.261 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.261 DEBUG:boto:Method: GET 2024-03-04T17:53:29.261 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.261 DEBUG:boto:Data: 2024-03-04T17:53:29.261 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.261 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.261 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.261 DEBUG:boto:Params: {} 2024-03-04T17:53:29.262 DEBUG:boto:Token: None 2024-03-04T17:53:29.262 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.262 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE= 2024-03-04T17:53:29.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE=', 'Content-Length': '0'} 2024-03-04T17:53:29.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a612c55b1c6d54d7-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.265 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.266 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.266 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.266 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.266 DEBUG:boto:Method: GET 2024-03-04T17:53:29.266 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.266 DEBUG:boto:Data: 2024-03-04T17:53:29.266 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.266 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.266 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.267 DEBUG:boto:Params: {} 2024-03-04T17:53:29.267 DEBUG:boto:Token: None 2024-03-04T17:53:29.267 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.267 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE= 2024-03-04T17:53:29.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE=', 'Content-Length': '0'} 2024-03-04T17:53:29.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ded28fec288714e9-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.312 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.312 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.312 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.312 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.312 DEBUG:boto:Method: GET 2024-03-04T17:53:29.313 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.313 DEBUG:boto:Data: 2024-03-04T17:53:29.313 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.313 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.313 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.313 DEBUG:boto:Params: {} 2024-03-04T17:53:29.313 DEBUG:boto:Token: None 2024-03-04T17:53:29.313 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.313 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbb9a5c9c3831823-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.318 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.319 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.319 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.319 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.319 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.319 DEBUG:boto:Method: GET 2024-03-04T17:53:29.319 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.319 DEBUG:boto:Data: 2024-03-04T17:53:29.319 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.320 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.320 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.320 DEBUG:boto:Params: {} 2024-03-04T17:53:29.320 DEBUG:boto:Token: None 2024-03-04T17:53:29.320 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.320 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000a1651c4564d063-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.325 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.326 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.326 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.326 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.326 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.326 DEBUG:boto:Method: GET 2024-03-04T17:53:29.326 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.326 DEBUG:boto:Data: 2024-03-04T17:53:29.327 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.327 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.327 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.327 DEBUG:boto:Params: {} 2024-03-04T17:53:29.327 DEBUG:boto:Token: None 2024-03-04T17:53:29.327 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.327 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ySSMXKLKL+mKvDccazVirRRE30U= 2024-03-04T17:53:29.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ySSMXKLKL+mKvDccazVirRRE30U=', 'Content-Length': '0'} 2024-03-04T17:53:29.331 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043499e719f5d2cec-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.372 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.372 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.372 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.373 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.373 DEBUG:boto:Method: GET 2024-03-04T17:53:29.373 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.373 DEBUG:boto:Data: 2024-03-04T17:53:29.373 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.373 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.373 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.373 DEBUG:boto:Params: {} 2024-03-04T17:53:29.373 DEBUG:boto:Token: None 2024-03-04T17:53:29.374 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.374 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ySSMXKLKL+mKvDccazVirRRE30U= 2024-03-04T17:53:29.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ySSMXKLKL+mKvDccazVirRRE30U=', 'Content-Length': '0'} 2024-03-04T17:53:29.377 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e52568e0e17eef92-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.377 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.377 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.377 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.377 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.377 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.377 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.378 DEBUG:boto:Data: 2024-03-04T17:53:29.378 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.378 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.378 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.378 DEBUG:boto:Params: {} 2024-03-04T17:53:29.378 DEBUG:boto:Token: None 2024-03-04T17:53:29.378 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.378 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Q8ss4IkESlXKMwfJ3fqZW1sym4c= 2024-03-04T17:53:29.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Q8ss4IkESlXKMwfJ3fqZW1sym4c=', 'Content-Length': '0'} 2024-03-04T17:53:29.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073fe75ab3dc6854c-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.382 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.382 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.382 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.382 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.382 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.382 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.382 DEBUG:boto:Data: 2024-03-04T17:53:29.382 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.382 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.382 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.383 DEBUG:boto:Params: {} 2024-03-04T17:53:29.383 DEBUG:boto:Token: None 2024-03-04T17:53:29.383 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.383 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Q8ss4IkESlXKMwfJ3fqZW1sym4c= 2024-03-04T17:53:29.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Q8ss4IkESlXKMwfJ3fqZW1sym4c=', 'Content-Length': '0'} 2024-03-04T17:53:29.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000faab3a4f14591b17-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.386 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.386 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.386 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.386 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.386 DEBUG:boto:Method: GET 2024-03-04T17:53:29.387 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.387 DEBUG:boto:Data: 2024-03-04T17:53:29.387 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.387 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.387 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.387 DEBUG:boto:Params: {} 2024-03-04T17:53:29.387 DEBUG:boto:Token: None 2024-03-04T17:53:29.387 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.387 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ySSMXKLKL+mKvDccazVirRRE30U= 2024-03-04T17:53:29.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ySSMXKLKL+mKvDccazVirRRE30U=', 'Content-Length': '0'} 2024-03-04T17:53:29.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a08a6214b8338949-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.391 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.391 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.391 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.391 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.391 DEBUG:boto:Method: GET 2024-03-04T17:53:29.391 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.391 DEBUG:boto:Data: 2024-03-04T17:53:29.391 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.391 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.392 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.392 DEBUG:boto:Params: {} 2024-03-04T17:53:29.392 DEBUG:boto:Token: None 2024-03-04T17:53:29.392 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:29.392 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ySSMXKLKL+mKvDccazVirRRE30U= 2024-03-04T17:53:29.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ySSMXKLKL+mKvDccazVirRRE30U=', 'Content-Length': '0'} 2024-03-04T17:53:29.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ca83195f9fec876-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.436 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.436 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.436 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.436 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.436 DEBUG:boto:Method: GET 2024-03-04T17:53:29.436 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.437 DEBUG:boto:Data: 2024-03-04T17:53:29.437 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.437 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.437 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.437 DEBUG:boto:Params: {} 2024-03-04T17:53:29.437 DEBUG:boto:Token: None 2024-03-04T17:53:29.437 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.437 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1f75f256735299c-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.442 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.443 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.443 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.443 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.443 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.443 DEBUG:boto:Method: GET 2024-03-04T17:53:29.443 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.443 DEBUG:boto:Data: 2024-03-04T17:53:29.443 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.443 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.444 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.444 DEBUG:boto:Params: {} 2024-03-04T17:53:29.444 DEBUG:boto:Token: None 2024-03-04T17:53:29.444 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.444 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070460ec64dd19449-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.449 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.449 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.449 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.450 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.450 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:29.450 DEBUG:boto:Data: 2024-03-04T17:53:29.450 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.450 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.450 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.450 DEBUG:boto:Params: {} 2024-03-04T17:53:29.450 DEBUG:boto:Token: None 2024-03-04T17:53:29.450 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj 2024-03-04T17:53:29.450 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y= 2024-03-04T17:53:29.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y=', 'Content-Length': '0'} 2024-03-04T17:53:29.456 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000420844d4d19ded40-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.457 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.457 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.457 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.457 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:29.457 DEBUG:boto:Data: 2024-03-04T17:53:29.457 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.457 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.457 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.457 DEBUG:boto:Params: {} 2024-03-04T17:53:29.458 DEBUG:boto:Token: None 2024-03-04T17:53:29.458 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj 2024-03-04T17:53:29.458 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y= 2024-03-04T17:53:29.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y=', 'Content-Length': '0'} 2024-03-04T17:53:29.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a532d206c24729c-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.465 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.465 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.465 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.465 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.465 DEBUG:boto:Method: GET 2024-03-04T17:53:29.465 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.466 DEBUG:boto:Data: 2024-03-04T17:53:29.466 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.466 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.466 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.466 DEBUG:boto:Params: {} 2024-03-04T17:53:29.466 DEBUG:boto:Token: None 2024-03-04T17:53:29.466 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.466 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE= 2024-03-04T17:53:29.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE=', 'Content-Length': '0'} 2024-03-04T17:53:29.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000827ea5e18780a914-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.512 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.512 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.512 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.513 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.513 DEBUG:boto:Method: GET 2024-03-04T17:53:29.513 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.513 DEBUG:boto:Data: 2024-03-04T17:53:29.513 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.513 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.513 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.513 DEBUG:boto:Params: {} 2024-03-04T17:53:29.513 DEBUG:boto:Token: None 2024-03-04T17:53:29.514 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.514 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE= 2024-03-04T17:53:29.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE=', 'Content-Length': '0'} 2024-03-04T17:53:29.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026a7f4a59ebe1349-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.518 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.518 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.518 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.518 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.518 DEBUG:boto:Method: GET 2024-03-04T17:53:29.518 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.519 DEBUG:boto:Data: 2024-03-04T17:53:29.519 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.519 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.519 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.519 DEBUG:boto:Params: {} 2024-03-04T17:53:29.519 DEBUG:boto:Token: None 2024-03-04T17:53:29.519 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.520 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adcf4764cf7611ab-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.525 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.525 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.526 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.526 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.526 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.526 DEBUG:boto:Method: GET 2024-03-04T17:53:29.526 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.526 DEBUG:boto:Data: 2024-03-04T17:53:29.526 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.526 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.526 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.526 DEBUG:boto:Params: {} 2024-03-04T17:53:29.527 DEBUG:boto:Token: None 2024-03-04T17:53:29.527 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.527 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004401bf6d0d7768dc-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.572 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.573 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.573 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.573 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.573 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.573 DEBUG:boto:Method: GET 2024-03-04T17:53:29.574 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.574 DEBUG:boto:Data: 2024-03-04T17:53:29.574 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.574 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.574 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.574 DEBUG:boto:Params: {} 2024-03-04T17:53:29.574 DEBUG:boto:Token: None 2024-03-04T17:53:29.575 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.575 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:05Mg7HK60geF26sKaDLMRpwqfro= 2024-03-04T17:53:29.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:05Mg7HK60geF26sKaDLMRpwqfro=', 'Content-Length': '0'} 2024-03-04T17:53:29.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000288575d881e286f1-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.578 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.578 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.579 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.579 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.579 DEBUG:boto:Method: GET 2024-03-04T17:53:29.579 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.579 DEBUG:boto:Data: 2024-03-04T17:53:29.579 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.579 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.579 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.579 DEBUG:boto:Params: {} 2024-03-04T17:53:29.579 DEBUG:boto:Token: None 2024-03-04T17:53:29.580 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.580 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:05Mg7HK60geF26sKaDLMRpwqfro= 2024-03-04T17:53:29.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:05Mg7HK60geF26sKaDLMRpwqfro=', 'Content-Length': '0'} 2024-03-04T17:53:29.583 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfc5ebbcb6daad88-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.583 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.584 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.584 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.584 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.584 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.584 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.584 DEBUG:boto:Data: 2024-03-04T17:53:29.584 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.585 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.585 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.585 DEBUG:boto:Params: {} 2024-03-04T17:53:29.585 DEBUG:boto:Token: None 2024-03-04T17:53:29.585 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.586 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+n4k2JKpjcYncOzpTzcx/UNWuqo= 2024-03-04T17:53:29.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+n4k2JKpjcYncOzpTzcx/UNWuqo=', 'Content-Length': '0'} 2024-03-04T17:53:29.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077b90e96cdf73fb2-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.589 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.589 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.589 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.589 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.590 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.590 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.590 DEBUG:boto:Data: 2024-03-04T17:53:29.590 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.590 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.590 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.590 DEBUG:boto:Params: {} 2024-03-04T17:53:29.591 DEBUG:boto:Token: None 2024-03-04T17:53:29.591 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.591 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+n4k2JKpjcYncOzpTzcx/UNWuqo= 2024-03-04T17:53:29.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+n4k2JKpjcYncOzpTzcx/UNWuqo=', 'Content-Length': '0'} 2024-03-04T17:53:29.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000392d45e1b1d99746-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.594 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.594 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.595 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.595 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.595 DEBUG:boto:Method: GET 2024-03-04T17:53:29.595 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.595 DEBUG:boto:Data: 2024-03-04T17:53:29.595 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.595 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.596 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.596 DEBUG:boto:Params: {} 2024-03-04T17:53:29.596 DEBUG:boto:Token: None 2024-03-04T17:53:29.596 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.597 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:05Mg7HK60geF26sKaDLMRpwqfro= 2024-03-04T17:53:29.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:05Mg7HK60geF26sKaDLMRpwqfro=', 'Content-Length': '0'} 2024-03-04T17:53:29.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037e952fa0ee8a6e9-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.640 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.640 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.640 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.641 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.641 DEBUG:boto:Method: GET 2024-03-04T17:53:29.641 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.641 DEBUG:boto:Data: 2024-03-04T17:53:29.641 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.641 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.642 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.642 DEBUG:boto:Params: {} 2024-03-04T17:53:29.642 DEBUG:boto:Token: None 2024-03-04T17:53:29.642 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:29.642 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:05Mg7HK60geF26sKaDLMRpwqfro= 2024-03-04T17:53:29.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:05Mg7HK60geF26sKaDLMRpwqfro=', 'Content-Length': '0'} 2024-03-04T17:53:29.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be43566ff4b51cb9-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.646 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.647 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.647 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.647 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.647 DEBUG:boto:Method: GET 2024-03-04T17:53:29.647 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.647 DEBUG:boto:Data: 2024-03-04T17:53:29.647 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.647 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.647 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.647 DEBUG:boto:Params: {} 2024-03-04T17:53:29.648 DEBUG:boto:Token: None 2024-03-04T17:53:29.648 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.648 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b589baf7f488f66-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.653 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.653 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.653 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.653 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.654 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.654 DEBUG:boto:Method: GET 2024-03-04T17:53:29.654 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.654 DEBUG:boto:Data: 2024-03-04T17:53:29.654 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.654 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.654 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.655 DEBUG:boto:Params: {} 2024-03-04T17:53:29.655 DEBUG:boto:Token: None 2024-03-04T17:53:29.655 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.655 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c02a7dbb4aa25ea7-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.660 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.660 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.660 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.661 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.661 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:29.661 DEBUG:boto:Data: 2024-03-04T17:53:29.661 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.661 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.661 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.661 DEBUG:boto:Params: {} 2024-03-04T17:53:29.661 DEBUG:boto:Token: None 2024-03-04T17:53:29.661 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj 2024-03-04T17:53:29.662 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y= 2024-03-04T17:53:29.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y=', 'Content-Length': '0'} 2024-03-04T17:53:29.668 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec89fa32faee3081-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.668 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.668 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.668 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.669 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:29.669 DEBUG:boto:Data: 2024-03-04T17:53:29.669 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.669 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.669 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.669 DEBUG:boto:Params: {} 2024-03-04T17:53:29.669 DEBUG:boto:Token: None 2024-03-04T17:53:29.669 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj 2024-03-04T17:53:29.669 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y= 2024-03-04T17:53:29.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y=', 'Content-Length': '0'} 2024-03-04T17:53:29.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bb307711e8f995f-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.676 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.676 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.676 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.676 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.676 DEBUG:boto:Method: GET 2024-03-04T17:53:29.676 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.676 DEBUG:boto:Data: 2024-03-04T17:53:29.676 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.677 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.677 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.677 DEBUG:boto:Params: {} 2024-03-04T17:53:29.677 DEBUG:boto:Token: None 2024-03-04T17:53:29.677 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.677 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE= 2024-03-04T17:53:29.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE=', 'Content-Length': '0'} 2024-03-04T17:53:29.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc2ee17762f127da-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.724 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.724 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.724 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.724 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.724 DEBUG:boto:Method: GET 2024-03-04T17:53:29.725 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.725 DEBUG:boto:Data: 2024-03-04T17:53:29.725 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.725 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.725 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.725 DEBUG:boto:Params: {} 2024-03-04T17:53:29.725 DEBUG:boto:Token: None 2024-03-04T17:53:29.725 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.726 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE= 2024-03-04T17:53:29.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE=', 'Content-Length': '0'} 2024-03-04T17:53:29.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f429df44016384d-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.729 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.729 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.729 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.729 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.729 DEBUG:boto:Method: GET 2024-03-04T17:53:29.729 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.729 DEBUG:boto:Data: 2024-03-04T17:53:29.729 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.730 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.730 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.730 DEBUG:boto:Params: {} 2024-03-04T17:53:29.730 DEBUG:boto:Token: None 2024-03-04T17:53:29.730 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.730 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009506388a42bd9bf0-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.735 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.735 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.735 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.736 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.736 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.736 DEBUG:boto:Method: GET 2024-03-04T17:53:29.736 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.736 DEBUG:boto:Data: 2024-03-04T17:53:29.736 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.736 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.736 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.736 DEBUG:boto:Params: {} 2024-03-04T17:53:29.736 DEBUG:boto:Token: None 2024-03-04T17:53:29.737 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.737 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eff26c27e96f83d2-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.742 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.743 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.743 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.743 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.743 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.743 DEBUG:boto:Method: GET 2024-03-04T17:53:29.743 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.743 DEBUG:boto:Data: 2024-03-04T17:53:29.743 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.743 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.744 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.744 DEBUG:boto:Params: {} 2024-03-04T17:53:29.744 DEBUG:boto:Token: None 2024-03-04T17:53:29.744 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.744 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hNj3lGq/daCMwo/JaUuInPs4FfA= 2024-03-04T17:53:29.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hNj3lGq/daCMwo/JaUuInPs4FfA=', 'Content-Length': '0'} 2024-03-04T17:53:29.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec27dd8329e3d53c-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.748 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.748 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.748 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.748 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.748 DEBUG:boto:Method: GET 2024-03-04T17:53:29.748 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.748 DEBUG:boto:Data: 2024-03-04T17:53:29.748 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.748 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.748 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.749 DEBUG:boto:Params: {} 2024-03-04T17:53:29.749 DEBUG:boto:Token: None 2024-03-04T17:53:29.749 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.749 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hNj3lGq/daCMwo/JaUuInPs4FfA= 2024-03-04T17:53:29.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hNj3lGq/daCMwo/JaUuInPs4FfA=', 'Content-Length': '0'} 2024-03-04T17:53:29.752 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038ab23b4f4beaebe-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.752 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.752 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.753 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.753 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.753 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.753 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.753 DEBUG:boto:Data: 2024-03-04T17:53:29.753 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.753 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.753 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.753 DEBUG:boto:Params: {} 2024-03-04T17:53:29.753 DEBUG:boto:Token: None 2024-03-04T17:53:29.753 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.754 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:I4/e8cjzpBKdQ3qddRSz4vSA8BM= 2024-03-04T17:53:29.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:I4/e8cjzpBKdQ3qddRSz4vSA8BM=', 'Content-Length': '0'} 2024-03-04T17:53:29.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004947ec5cfcbfbef-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.757 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.757 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.757 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.757 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.757 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.757 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.757 DEBUG:boto:Data: 2024-03-04T17:53:29.757 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.758 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.758 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.758 DEBUG:boto:Params: {} 2024-03-04T17:53:29.758 DEBUG:boto:Token: None 2024-03-04T17:53:29.758 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.758 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:I4/e8cjzpBKdQ3qddRSz4vSA8BM= 2024-03-04T17:53:29.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:I4/e8cjzpBKdQ3qddRSz4vSA8BM=', 'Content-Length': '0'} 2024-03-04T17:53:29.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc9f0afe689c37bb-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.761 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.761 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.761 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.762 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.762 DEBUG:boto:Method: GET 2024-03-04T17:53:29.762 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.762 DEBUG:boto:Data: 2024-03-04T17:53:29.762 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.762 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.762 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.762 DEBUG:boto:Params: {} 2024-03-04T17:53:29.762 DEBUG:boto:Token: None 2024-03-04T17:53:29.763 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.763 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hNj3lGq/daCMwo/JaUuInPs4FfA= 2024-03-04T17:53:29.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hNj3lGq/daCMwo/JaUuInPs4FfA=', 'Content-Length': '0'} 2024-03-04T17:53:29.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdab5da104754347-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.808 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.808 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.808 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.808 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.808 DEBUG:boto:Method: GET 2024-03-04T17:53:29.808 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.809 DEBUG:boto:Data: 2024-03-04T17:53:29.809 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.809 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.809 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.809 DEBUG:boto:Params: {} 2024-03-04T17:53:29.809 DEBUG:boto:Token: None 2024-03-04T17:53:29.809 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:29.809 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hNj3lGq/daCMwo/JaUuInPs4FfA= 2024-03-04T17:53:29.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hNj3lGq/daCMwo/JaUuInPs4FfA=', 'Content-Length': '0'} 2024-03-04T17:53:29.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012856b6920881bff-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.813 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.813 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.813 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.813 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.813 DEBUG:boto:Method: GET 2024-03-04T17:53:29.813 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.813 DEBUG:boto:Data: 2024-03-04T17:53:29.814 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.814 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.814 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.814 DEBUG:boto:Params: {} 2024-03-04T17:53:29.814 DEBUG:boto:Token: None 2024-03-04T17:53:29.814 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.814 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026b0b4169b3c3cfc-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.820 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.820 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.820 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.821 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.821 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.821 DEBUG:boto:Method: GET 2024-03-04T17:53:29.821 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.821 DEBUG:boto:Data: 2024-03-04T17:53:29.821 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.821 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.821 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.821 DEBUG:boto:Params: {} 2024-03-04T17:53:29.821 DEBUG:boto:Token: None 2024-03-04T17:53:29.822 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.822 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d2beb33680ebd8d-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.827 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.827 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.828 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.828 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.828 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:29.828 DEBUG:boto:Data: 2024-03-04T17:53:29.828 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.828 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.828 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.828 DEBUG:boto:Params: {} 2024-03-04T17:53:29.828 DEBUG:boto:Token: None 2024-03-04T17:53:29.828 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj 2024-03-04T17:53:29.829 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y= 2024-03-04T17:53:29.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y=', 'Content-Length': '0'} 2024-03-04T17:53:29.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048c8aa98b94eaad5-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.834 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.835 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.835 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.835 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:29.835 DEBUG:boto:Data: 2024-03-04T17:53:29.835 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.835 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.835 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.835 DEBUG:boto:Params: {} 2024-03-04T17:53:29.835 DEBUG:boto:Token: None 2024-03-04T17:53:29.836 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj 2024-03-04T17:53:29.836 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y= 2024-03-04T17:53:29.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/onn68UMewwolEspAoAEGb79L5Y=', 'Content-Length': '0'} 2024-03-04T17:53:29.842 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024ad63ecd44190c7-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.842 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.842 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.842 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.842 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.842 DEBUG:boto:Method: GET 2024-03-04T17:53:29.843 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.843 DEBUG:boto:Data: 2024-03-04T17:53:29.843 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.843 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.843 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.843 DEBUG:boto:Params: {} 2024-03-04T17:53:29.843 DEBUG:boto:Token: None 2024-03-04T17:53:29.843 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.843 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE= 2024-03-04T17:53:29.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE=', 'Content-Length': '0'} 2024-03-04T17:53:29.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097cdaf14a20ec253-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.888 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.888 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.888 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.888 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.888 DEBUG:boto:Method: GET 2024-03-04T17:53:29.889 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.889 DEBUG:boto:Data: 2024-03-04T17:53:29.889 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.889 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.889 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.889 DEBUG:boto:Params: {} 2024-03-04T17:53:29.889 DEBUG:boto:Token: None 2024-03-04T17:53:29.889 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:29.889 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE= 2024-03-04T17:53:29.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ePmzW2vqB/YTEEYEa3GRZK6XriE=', 'Content-Length': '0'} 2024-03-04T17:53:29.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b744b7ed23fe7c54-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.893 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.893 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.893 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.893 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.893 DEBUG:boto:Method: GET 2024-03-04T17:53:29.893 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.893 DEBUG:boto:Data: 2024-03-04T17:53:29.894 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.894 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.894 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.894 DEBUG:boto:Params: {} 2024-03-04T17:53:29.894 DEBUG:boto:Token: None 2024-03-04T17:53:29.894 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.894 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068db19096d26cd0b-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.899 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.900 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.900 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.900 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.900 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.900 DEBUG:boto:Method: GET 2024-03-04T17:53:29.900 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.901 DEBUG:boto:Data: 2024-03-04T17:53:29.901 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.901 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.901 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.901 DEBUG:boto:Params: {} 2024-03-04T17:53:29.901 DEBUG:boto:Token: None 2024-03-04T17:53:29.901 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.901 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d1a0cd6be1c27d6-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.948 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:29.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.948 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.949 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.949 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.949 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.949 DEBUG:boto:Method: GET 2024-03-04T17:53:29.949 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.949 DEBUG:boto:Data: 2024-03-04T17:53:29.949 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.949 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.949 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.950 DEBUG:boto:Params: {} 2024-03-04T17:53:29.950 DEBUG:boto:Token: None 2024-03-04T17:53:29.950 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.950 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f1HFYtUi4eWX6Jk+wOqi0pumvsM= 2024-03-04T17:53:29.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f1HFYtUi4eWX6Jk+wOqi0pumvsM=', 'Content-Length': '0'} 2024-03-04T17:53:29.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003285c13f0c8dd054-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.953 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.953 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.954 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.954 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.954 DEBUG:boto:Method: GET 2024-03-04T17:53:29.954 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.954 DEBUG:boto:Data: 2024-03-04T17:53:29.954 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.954 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.954 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.954 DEBUG:boto:Params: {} 2024-03-04T17:53:29.955 DEBUG:boto:Token: None 2024-03-04T17:53:29.955 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.955 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f1HFYtUi4eWX6Jk+wOqi0pumvsM= 2024-03-04T17:53:29.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f1HFYtUi4eWX6Jk+wOqi0pumvsM=', 'Content-Length': '0'} 2024-03-04T17:53:29.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b1f7acdd2f0f4f5-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.958 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.958 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.959 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.959 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.959 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.959 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.959 DEBUG:boto:Data: 2024-03-04T17:53:29.959 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.959 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.959 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.959 DEBUG:boto:Params: {} 2024-03-04T17:53:29.960 DEBUG:boto:Token: None 2024-03-04T17:53:29.960 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.960 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWHtsgnkv7KusA6x8yh6tVLvB/c= 2024-03-04T17:53:29.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWHtsgnkv7KusA6x8yh6tVLvB/c=', 'Content-Length': '0'} 2024-03-04T17:53:29.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdfb43077df20d52-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.963 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.963 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.963 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.963 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.963 DEBUG:boto:Method: HEAD 2024-03-04T17:53:29.964 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.964 DEBUG:boto:Data: 2024-03-04T17:53:29.964 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.964 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.964 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.964 DEBUG:boto:Params: {} 2024-03-04T17:53:29.964 DEBUG:boto:Token: None 2024-03-04T17:53:29.964 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.965 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VWHtsgnkv7KusA6x8yh6tVLvB/c= 2024-03-04T17:53:29.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VWHtsgnkv7KusA6x8yh6tVLvB/c=', 'Content-Length': '0'} 2024-03-04T17:53:29.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032f6a90073c8d337-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:29.968 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.968 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.968 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.968 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.969 DEBUG:boto:Method: GET 2024-03-04T17:53:29.969 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.969 DEBUG:boto:Data: 2024-03-04T17:53:29.969 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.969 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.969 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.969 DEBUG:boto:Params: {} 2024-03-04T17:53:29.969 DEBUG:boto:Token: None 2024-03-04T17:53:29.969 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.970 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f1HFYtUi4eWX6Jk+wOqi0pumvsM= 2024-03-04T17:53:29.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f1HFYtUi4eWX6Jk+wOqi0pumvsM=', 'Content-Length': '0'} 2024-03-04T17:53:29.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f960db0ca62e065-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.973 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:29.973 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:29.973 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.973 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.973 DEBUG:boto:Method: GET 2024-03-04T17:53:29.973 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.973 DEBUG:boto:Data: 2024-03-04T17:53:29.973 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.974 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:29.974 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.974 DEBUG:boto:Params: {} 2024-03-04T17:53:29.974 DEBUG:boto:Token: None 2024-03-04T17:53:29.974 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:29.974 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f1HFYtUi4eWX6Jk+wOqi0pumvsM= 2024-03-04T17:53:29.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f1HFYtUi4eWX6Jk+wOqi0pumvsM=', 'Content-Length': '0'} 2024-03-04T17:53:29.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005679cdb8d37c81c6-0065e60a99-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:29.977 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:29.978 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:29.978 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.978 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:29.978 DEBUG:boto:Method: GET 2024-03-04T17:53:29.978 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:29.978 DEBUG:boto:Data: 2024-03-04T17:53:29.978 DEBUG:boto:Headers: {} 2024-03-04T17:53:29.978 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:29.978 DEBUG:boto:Port: 8000 2024-03-04T17:53:29.978 DEBUG:boto:Params: {} 2024-03-04T17:53:29.979 DEBUG:boto:Token: None 2024-03-04T17:53:29.979 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:29 GMT /ppfahn-22/ 2024-03-04T17:53:29.979 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g= 2024-03-04T17:53:29.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p3cS+QlKig3Hfyy/AvmoiBhdy4g=', 'Content-Length': '0'} 2024-03-04T17:53:29.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042ad31cee68e5c2d-0065e60a99-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.024 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.024 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.025 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.025 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.025 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.025 DEBUG:boto:Method: GET 2024-03-04T17:53:30.025 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.025 DEBUG:boto:Data: 2024-03-04T17:53:30.025 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.025 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.026 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.026 DEBUG:boto:Params: {} 2024-03-04T17:53:30.026 DEBUG:boto:Token: None 2024-03-04T17:53:30.026 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.026 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001de9a3e23466a74e-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.032 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.032 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.032 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.032 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.032 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:30.032 DEBUG:boto:Data: 2024-03-04T17:53:30.032 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.033 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.033 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.033 DEBUG:boto:Params: {} 2024-03-04T17:53:30.033 DEBUG:boto:Token: None 2024-03-04T17:53:30.033 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj 2024-03-04T17:53:30.033 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw= 2024-03-04T17:53:30.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw=', 'Content-Length': '0'} 2024-03-04T17:53:30.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024848eb1e356bc8f-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.040 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.041 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.041 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.041 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:30.041 DEBUG:boto:Data: 2024-03-04T17:53:30.041 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.041 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.041 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.041 DEBUG:boto:Params: {} 2024-03-04T17:53:30.041 DEBUG:boto:Token: None 2024-03-04T17:53:30.042 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj 2024-03-04T17:53:30.042 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw= 2024-03-04T17:53:30.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw=', 'Content-Length': '0'} 2024-03-04T17:53:30.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009acdf495a3b3dfb1-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:30.049 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.049 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.049 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.049 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.049 DEBUG:boto:Method: GET 2024-03-04T17:53:30.049 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.049 DEBUG:boto:Data: 2024-03-04T17:53:30.049 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.050 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.050 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.050 DEBUG:boto:Params: {} 2024-03-04T17:53:30.050 DEBUG:boto:Token: None 2024-03-04T17:53:30.050 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.050 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I= 2024-03-04T17:53:30.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I=', 'Content-Length': '0'} 2024-03-04T17:53:30.054 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096edcbb4cffce7ad-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.096 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.096 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.096 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.097 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.097 DEBUG:boto:Method: GET 2024-03-04T17:53:30.097 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.097 DEBUG:boto:Data: 2024-03-04T17:53:30.097 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.097 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.097 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.097 DEBUG:boto:Params: {} 2024-03-04T17:53:30.097 DEBUG:boto:Token: None 2024-03-04T17:53:30.098 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.098 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I= 2024-03-04T17:53:30.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I=', 'Content-Length': '0'} 2024-03-04T17:53:30.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055fdd7f6c0e53133-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.101 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.101 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.102 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.102 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.102 DEBUG:boto:Method: GET 2024-03-04T17:53:30.102 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.102 DEBUG:boto:Data: 2024-03-04T17:53:30.102 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.102 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.102 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.102 DEBUG:boto:Params: {} 2024-03-04T17:53:30.103 DEBUG:boto:Token: None 2024-03-04T17:53:30.103 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.103 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005df614149569f40c-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.148 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.148 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.149 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.149 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.149 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.149 DEBUG:boto:Method: GET 2024-03-04T17:53:30.149 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.149 DEBUG:boto:Data: 2024-03-04T17:53:30.149 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.149 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.149 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.149 DEBUG:boto:Params: {} 2024-03-04T17:53:30.149 DEBUG:boto:Token: None 2024-03-04T17:53:30.150 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.150 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000655f382f87483c74-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.155 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:30.155 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.155 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.156 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.156 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.156 DEBUG:boto:Method: GET 2024-03-04T17:53:30.156 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.156 DEBUG:boto:Data: 2024-03-04T17:53:30.156 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.156 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.156 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.156 DEBUG:boto:Params: {} 2024-03-04T17:53:30.156 DEBUG:boto:Token: None 2024-03-04T17:53:30.157 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.157 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bup2OUSxrSJtoUZcxdVtz12crA8= 2024-03-04T17:53:30.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bup2OUSxrSJtoUZcxdVtz12crA8=', 'Content-Length': '0'} 2024-03-04T17:53:30.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b455a04371c5a42d-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.200 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.200 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.200 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.200 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.200 DEBUG:boto:Method: GET 2024-03-04T17:53:30.201 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.201 DEBUG:boto:Data: 2024-03-04T17:53:30.201 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.201 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.201 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.201 DEBUG:boto:Params: {} 2024-03-04T17:53:30.201 DEBUG:boto:Token: None 2024-03-04T17:53:30.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.201 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bup2OUSxrSJtoUZcxdVtz12crA8= 2024-03-04T17:53:30.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bup2OUSxrSJtoUZcxdVtz12crA8=', 'Content-Length': '0'} 2024-03-04T17:53:30.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d0be1f53a11b9f9-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.205 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.205 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.205 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.205 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.205 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.205 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.205 DEBUG:boto:Data: 2024-03-04T17:53:30.206 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.206 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.206 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.206 DEBUG:boto:Params: {} 2024-03-04T17:53:30.206 DEBUG:boto:Token: None 2024-03-04T17:53:30.206 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.206 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KJuXQT1cWbC+KpUQEkioHVDBva0= 2024-03-04T17:53:30.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KJuXQT1cWbC+KpUQEkioHVDBva0=', 'Content-Length': '0'} 2024-03-04T17:53:30.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000200c4b378e0fa17-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.209 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.210 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.210 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.210 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.210 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.210 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.210 DEBUG:boto:Data: 2024-03-04T17:53:30.210 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.210 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.210 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.210 DEBUG:boto:Params: {} 2024-03-04T17:53:30.211 DEBUG:boto:Token: None 2024-03-04T17:53:30.211 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.211 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KJuXQT1cWbC+KpUQEkioHVDBva0= 2024-03-04T17:53:30.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KJuXQT1cWbC+KpUQEkioHVDBva0=', 'Content-Length': '0'} 2024-03-04T17:53:30.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f4d225d64e31f77-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:30.214 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.214 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.214 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.214 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.214 DEBUG:boto:Method: GET 2024-03-04T17:53:30.214 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.215 DEBUG:boto:Data: 2024-03-04T17:53:30.215 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.215 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.215 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.215 DEBUG:boto:Params: {} 2024-03-04T17:53:30.215 DEBUG:boto:Token: None 2024-03-04T17:53:30.215 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.215 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bup2OUSxrSJtoUZcxdVtz12crA8= 2024-03-04T17:53:30.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bup2OUSxrSJtoUZcxdVtz12crA8=', 'Content-Length': '0'} 2024-03-04T17:53:30.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001e71be3d013bddc-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.264 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.264 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.264 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.264 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.264 DEBUG:boto:Method: GET 2024-03-04T17:53:30.264 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.265 DEBUG:boto:Data: 2024-03-04T17:53:30.265 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.265 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.265 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.265 DEBUG:boto:Params: {} 2024-03-04T17:53:30.265 DEBUG:boto:Token: None 2024-03-04T17:53:30.265 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:30.265 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bup2OUSxrSJtoUZcxdVtz12crA8= 2024-03-04T17:53:30.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bup2OUSxrSJtoUZcxdVtz12crA8=', 'Content-Length': '0'} 2024-03-04T17:53:30.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000952d91bf70c3846a-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.268 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.269 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.269 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.269 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.269 DEBUG:boto:Method: GET 2024-03-04T17:53:30.269 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.269 DEBUG:boto:Data: 2024-03-04T17:53:30.269 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.269 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.269 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.269 DEBUG:boto:Params: {} 2024-03-04T17:53:30.270 DEBUG:boto:Token: None 2024-03-04T17:53:30.270 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.270 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cac95e27a96899f-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.275 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.276 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.276 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.276 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.276 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.276 DEBUG:boto:Method: GET 2024-03-04T17:53:30.276 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.276 DEBUG:boto:Data: 2024-03-04T17:53:30.277 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.277 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.277 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.277 DEBUG:boto:Params: {} 2024-03-04T17:53:30.277 DEBUG:boto:Token: None 2024-03-04T17:53:30.277 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.277 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037de3226d9e530e0-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.283 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.283 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.283 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.284 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.284 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:30.284 DEBUG:boto:Data: 2024-03-04T17:53:30.284 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.284 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.284 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.284 DEBUG:boto:Params: {} 2024-03-04T17:53:30.284 DEBUG:boto:Token: None 2024-03-04T17:53:30.284 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj 2024-03-04T17:53:30.285 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw= 2024-03-04T17:53:30.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw=', 'Content-Length': '0'} 2024-03-04T17:53:30.291 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4507ad32e213651-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.291 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.291 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.291 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.291 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:30.291 DEBUG:boto:Data: 2024-03-04T17:53:30.291 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.291 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.292 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.292 DEBUG:boto:Params: {} 2024-03-04T17:53:30.292 DEBUG:boto:Token: None 2024-03-04T17:53:30.292 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj 2024-03-04T17:53:30.292 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw= 2024-03-04T17:53:30.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw=', 'Content-Length': '0'} 2024-03-04T17:53:30.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009140335d56da120f-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:30.298 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.298 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.298 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.299 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.299 DEBUG:boto:Method: GET 2024-03-04T17:53:30.299 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.299 DEBUG:boto:Data: 2024-03-04T17:53:30.299 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.299 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.299 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.299 DEBUG:boto:Params: {} 2024-03-04T17:53:30.299 DEBUG:boto:Token: None 2024-03-04T17:53:30.299 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.300 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I= 2024-03-04T17:53:30.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I=', 'Content-Length': '0'} 2024-03-04T17:53:30.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008eed0ee29f26931e-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.344 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.344 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.344 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.344 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.344 DEBUG:boto:Method: GET 2024-03-04T17:53:30.344 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.345 DEBUG:boto:Data: 2024-03-04T17:53:30.345 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.345 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.345 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.345 DEBUG:boto:Params: {} 2024-03-04T17:53:30.345 DEBUG:boto:Token: None 2024-03-04T17:53:30.345 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.345 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I= 2024-03-04T17:53:30.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I=', 'Content-Length': '0'} 2024-03-04T17:53:30.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a9c2fcbdd9bf462-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.349 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.349 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.349 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.349 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.349 DEBUG:boto:Method: GET 2024-03-04T17:53:30.349 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.349 DEBUG:boto:Data: 2024-03-04T17:53:30.349 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.349 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.349 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.350 DEBUG:boto:Params: {} 2024-03-04T17:53:30.350 DEBUG:boto:Token: None 2024-03-04T17:53:30.350 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.350 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bedf7bd888fca8b4-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.354 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.355 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.355 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.355 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.355 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.355 DEBUG:boto:Method: GET 2024-03-04T17:53:30.355 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.355 DEBUG:boto:Data: 2024-03-04T17:53:30.355 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.355 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.356 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.356 DEBUG:boto:Params: {} 2024-03-04T17:53:30.356 DEBUG:boto:Token: None 2024-03-04T17:53:30.356 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.356 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bce67ca58c6bc8f0-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.361 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:30.361 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.362 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.362 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.362 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.362 DEBUG:boto:Method: GET 2024-03-04T17:53:30.362 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.362 DEBUG:boto:Data: 2024-03-04T17:53:30.362 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.362 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.362 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.362 DEBUG:boto:Params: {} 2024-03-04T17:53:30.362 DEBUG:boto:Token: None 2024-03-04T17:53:30.363 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.363 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:u2jNje+OFnpsdqVGjBTIupPx3hA= 2024-03-04T17:53:30.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:u2jNje+OFnpsdqVGjBTIupPx3hA=', 'Content-Length': '0'} 2024-03-04T17:53:30.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e967b7d38b0f9be-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.366 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.366 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.366 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.366 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.366 DEBUG:boto:Method: GET 2024-03-04T17:53:30.366 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.366 DEBUG:boto:Data: 2024-03-04T17:53:30.367 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.367 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.367 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.367 DEBUG:boto:Params: {} 2024-03-04T17:53:30.367 DEBUG:boto:Token: None 2024-03-04T17:53:30.367 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.368 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:u2jNje+OFnpsdqVGjBTIupPx3hA= 2024-03-04T17:53:30.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:u2jNje+OFnpsdqVGjBTIupPx3hA=', 'Content-Length': '0'} 2024-03-04T17:53:30.370 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df7ca80e640b756b-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.371 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.371 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.371 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.371 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.371 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.371 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.371 DEBUG:boto:Data: 2024-03-04T17:53:30.371 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.372 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.372 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.372 DEBUG:boto:Params: {} 2024-03-04T17:53:30.372 DEBUG:boto:Token: None 2024-03-04T17:53:30.372 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.372 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZbcA7Yz5g8ESyX+4yn0oaj4t7Is= 2024-03-04T17:53:30.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZbcA7Yz5g8ESyX+4yn0oaj4t7Is=', 'Content-Length': '0'} 2024-03-04T17:53:30.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053368c9477b5fc60-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.375 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.375 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.376 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.376 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.376 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.376 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.376 DEBUG:boto:Data: 2024-03-04T17:53:30.376 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.376 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.376 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.376 DEBUG:boto:Params: {} 2024-03-04T17:53:30.377 DEBUG:boto:Token: None 2024-03-04T17:53:30.377 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.377 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZbcA7Yz5g8ESyX+4yn0oaj4t7Is= 2024-03-04T17:53:30.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZbcA7Yz5g8ESyX+4yn0oaj4t7Is=', 'Content-Length': '0'} 2024-03-04T17:53:30.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b866c3a0e3d2b9e0-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:30.380 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.380 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.380 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.380 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.381 DEBUG:boto:Method: GET 2024-03-04T17:53:30.381 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.381 DEBUG:boto:Data: 2024-03-04T17:53:30.381 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.381 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.381 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.381 DEBUG:boto:Params: {} 2024-03-04T17:53:30.381 DEBUG:boto:Token: None 2024-03-04T17:53:30.381 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.382 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:u2jNje+OFnpsdqVGjBTIupPx3hA= 2024-03-04T17:53:30.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:u2jNje+OFnpsdqVGjBTIupPx3hA=', 'Content-Length': '0'} 2024-03-04T17:53:30.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000546ddd321ff3e3a6-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.428 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.428 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.428 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.428 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.428 DEBUG:boto:Method: GET 2024-03-04T17:53:30.429 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.429 DEBUG:boto:Data: 2024-03-04T17:53:30.429 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.429 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.429 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.429 DEBUG:boto:Params: {} 2024-03-04T17:53:30.429 DEBUG:boto:Token: None 2024-03-04T17:53:30.429 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:30.429 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:u2jNje+OFnpsdqVGjBTIupPx3hA= 2024-03-04T17:53:30.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:u2jNje+OFnpsdqVGjBTIupPx3hA=', 'Content-Length': '0'} 2024-03-04T17:53:30.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e3bbcc6aede7520-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.433 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.433 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.433 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.433 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.433 DEBUG:boto:Method: GET 2024-03-04T17:53:30.433 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.433 DEBUG:boto:Data: 2024-03-04T17:53:30.433 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.433 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.433 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.434 DEBUG:boto:Params: {} 2024-03-04T17:53:30.434 DEBUG:boto:Token: None 2024-03-04T17:53:30.434 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.434 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d3a2befd74d3510-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.438 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.439 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.439 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.439 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.439 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.439 DEBUG:boto:Method: GET 2024-03-04T17:53:30.439 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.439 DEBUG:boto:Data: 2024-03-04T17:53:30.439 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.439 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.440 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.440 DEBUG:boto:Params: {} 2024-03-04T17:53:30.440 DEBUG:boto:Token: None 2024-03-04T17:53:30.440 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.440 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ce5a2c55cac7ce1-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.488 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.488 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.488 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.488 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.489 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:30.489 DEBUG:boto:Data: 2024-03-04T17:53:30.489 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.489 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.489 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.489 DEBUG:boto:Params: {} 2024-03-04T17:53:30.489 DEBUG:boto:Token: None 2024-03-04T17:53:30.489 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj 2024-03-04T17:53:30.489 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw= 2024-03-04T17:53:30.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw=', 'Content-Length': '0'} 2024-03-04T17:53:30.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e84f9f0b656a9c40-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.496 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.496 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.496 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.496 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:30.497 DEBUG:boto:Data: 2024-03-04T17:53:30.497 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.497 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.497 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.497 DEBUG:boto:Params: {} 2024-03-04T17:53:30.497 DEBUG:boto:Token: None 2024-03-04T17:53:30.497 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj 2024-03-04T17:53:30.497 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw= 2024-03-04T17:53:30.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw=', 'Content-Length': '0'} 2024-03-04T17:53:30.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e07345b5de26296-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:30.503 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.503 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.503 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.503 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.503 DEBUG:boto:Method: GET 2024-03-04T17:53:30.504 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.504 DEBUG:boto:Data: 2024-03-04T17:53:30.504 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.504 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.504 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.504 DEBUG:boto:Params: {} 2024-03-04T17:53:30.504 DEBUG:boto:Token: None 2024-03-04T17:53:30.504 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.504 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I= 2024-03-04T17:53:30.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I=', 'Content-Length': '0'} 2024-03-04T17:53:30.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff472230b112cd26-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.548 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.548 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.548 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.548 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.549 DEBUG:boto:Method: GET 2024-03-04T17:53:30.549 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.549 DEBUG:boto:Data: 2024-03-04T17:53:30.549 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.549 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.549 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.549 DEBUG:boto:Params: {} 2024-03-04T17:53:30.549 DEBUG:boto:Token: None 2024-03-04T17:53:30.550 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.550 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I= 2024-03-04T17:53:30.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I=', 'Content-Length': '0'} 2024-03-04T17:53:30.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004bf0fc9244eb19e-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.553 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.553 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.553 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.554 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.554 DEBUG:boto:Method: GET 2024-03-04T17:53:30.554 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.554 DEBUG:boto:Data: 2024-03-04T17:53:30.554 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.554 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.554 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.554 DEBUG:boto:Params: {} 2024-03-04T17:53:30.554 DEBUG:boto:Token: None 2024-03-04T17:53:30.555 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.555 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dde314355712ee4-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.600 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.600 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.600 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.601 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.601 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.601 DEBUG:boto:Method: GET 2024-03-04T17:53:30.601 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.601 DEBUG:boto:Data: 2024-03-04T17:53:30.601 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.601 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.601 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.601 DEBUG:boto:Params: {} 2024-03-04T17:53:30.602 DEBUG:boto:Token: None 2024-03-04T17:53:30.602 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.602 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000478b88b4154af003-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.607 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:30.608 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.608 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.608 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.608 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.608 DEBUG:boto:Method: GET 2024-03-04T17:53:30.608 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.608 DEBUG:boto:Data: 2024-03-04T17:53:30.608 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.609 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.609 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.609 DEBUG:boto:Params: {} 2024-03-04T17:53:30.609 DEBUG:boto:Token: None 2024-03-04T17:53:30.609 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.609 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TXewJI8Ssr0pHEI5ZTuKLWA+nAs= 2024-03-04T17:53:30.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TXewJI8Ssr0pHEI5ZTuKLWA+nAs=', 'Content-Length': '0'} 2024-03-04T17:53:30.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006223763f03c4181e-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.612 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.613 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.613 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.613 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.613 DEBUG:boto:Method: GET 2024-03-04T17:53:30.613 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.613 DEBUG:boto:Data: 2024-03-04T17:53:30.613 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.613 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.613 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.613 DEBUG:boto:Params: {} 2024-03-04T17:53:30.614 DEBUG:boto:Token: None 2024-03-04T17:53:30.614 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.614 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TXewJI8Ssr0pHEI5ZTuKLWA+nAs= 2024-03-04T17:53:30.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TXewJI8Ssr0pHEI5ZTuKLWA+nAs=', 'Content-Length': '0'} 2024-03-04T17:53:30.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5c9b4db23dd0f8c-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.617 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.617 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.618 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.618 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.618 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.618 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.618 DEBUG:boto:Data: 2024-03-04T17:53:30.618 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.618 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.618 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.618 DEBUG:boto:Params: {} 2024-03-04T17:53:30.619 DEBUG:boto:Token: None 2024-03-04T17:53:30.619 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.619 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TmoDBJ2LcKMW4gu8JIl6f7i5zXg= 2024-03-04T17:53:30.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TmoDBJ2LcKMW4gu8JIl6f7i5zXg=', 'Content-Length': '0'} 2024-03-04T17:53:30.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f0ca06e6fc05388-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.622 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.622 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.622 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.622 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.623 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.623 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.623 DEBUG:boto:Data: 2024-03-04T17:53:30.623 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.623 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.623 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.623 DEBUG:boto:Params: {} 2024-03-04T17:53:30.623 DEBUG:boto:Token: None 2024-03-04T17:53:30.624 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.624 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TmoDBJ2LcKMW4gu8JIl6f7i5zXg= 2024-03-04T17:53:30.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TmoDBJ2LcKMW4gu8JIl6f7i5zXg=', 'Content-Length': '0'} 2024-03-04T17:53:30.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb05a0c9d0eb59f3-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:30.627 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.627 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.627 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.627 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.628 DEBUG:boto:Method: GET 2024-03-04T17:53:30.628 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.628 DEBUG:boto:Data: 2024-03-04T17:53:30.628 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.628 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.628 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.628 DEBUG:boto:Params: {} 2024-03-04T17:53:30.628 DEBUG:boto:Token: None 2024-03-04T17:53:30.628 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.629 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TXewJI8Ssr0pHEI5ZTuKLWA+nAs= 2024-03-04T17:53:30.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TXewJI8Ssr0pHEI5ZTuKLWA+nAs=', 'Content-Length': '0'} 2024-03-04T17:53:30.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000980744f52a65f6f3-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.672 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.672 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.672 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.672 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.673 DEBUG:boto:Method: GET 2024-03-04T17:53:30.673 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.673 DEBUG:boto:Data: 2024-03-04T17:53:30.673 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.673 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.673 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.673 DEBUG:boto:Params: {} 2024-03-04T17:53:30.673 DEBUG:boto:Token: None 2024-03-04T17:53:30.673 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:30.674 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TXewJI8Ssr0pHEI5ZTuKLWA+nAs= 2024-03-04T17:53:30.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TXewJI8Ssr0pHEI5ZTuKLWA+nAs=', 'Content-Length': '0'} 2024-03-04T17:53:30.677 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000dc5fdf0cc2e0a71-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.677 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.677 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.677 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.677 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.677 DEBUG:boto:Method: GET 2024-03-04T17:53:30.677 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.678 DEBUG:boto:Data: 2024-03-04T17:53:30.678 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.678 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.678 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.678 DEBUG:boto:Params: {} 2024-03-04T17:53:30.678 DEBUG:boto:Token: None 2024-03-04T17:53:30.678 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.678 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008833745eb36e5610-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.683 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.683 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.684 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.684 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.684 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.684 DEBUG:boto:Method: GET 2024-03-04T17:53:30.684 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.684 DEBUG:boto:Data: 2024-03-04T17:53:30.684 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.684 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.685 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.685 DEBUG:boto:Params: {} 2024-03-04T17:53:30.685 DEBUG:boto:Token: None 2024-03-04T17:53:30.685 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.685 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004240186c73079dc4-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.690 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.690 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.690 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.690 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.690 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:30.691 DEBUG:boto:Data: 2024-03-04T17:53:30.691 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.691 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.691 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.691 DEBUG:boto:Params: {} 2024-03-04T17:53:30.691 DEBUG:boto:Token: None 2024-03-04T17:53:30.691 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj 2024-03-04T17:53:30.691 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw= 2024-03-04T17:53:30.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw=', 'Content-Length': '0'} 2024-03-04T17:53:30.698 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e98e661ee16f23aa-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.698 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.698 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.698 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.698 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:30.699 DEBUG:boto:Data: 2024-03-04T17:53:30.699 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.699 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.699 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.699 DEBUG:boto:Params: {} 2024-03-04T17:53:30.699 DEBUG:boto:Token: None 2024-03-04T17:53:30.699 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj 2024-03-04T17:53:30.699 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw= 2024-03-04T17:53:30.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw=', 'Content-Length': '0'} 2024-03-04T17:53:30.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008457af049fea3817-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:30.706 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.706 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.706 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.706 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.706 DEBUG:boto:Method: GET 2024-03-04T17:53:30.706 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.706 DEBUG:boto:Data: 2024-03-04T17:53:30.706 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.706 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.707 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.707 DEBUG:boto:Params: {} 2024-03-04T17:53:30.707 DEBUG:boto:Token: None 2024-03-04T17:53:30.707 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.707 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I= 2024-03-04T17:53:30.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I=', 'Content-Length': '0'} 2024-03-04T17:53:30.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000162f1272000c035b-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.752 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.752 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.752 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.752 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.752 DEBUG:boto:Method: GET 2024-03-04T17:53:30.753 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.753 DEBUG:boto:Data: 2024-03-04T17:53:30.753 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.753 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.753 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.753 DEBUG:boto:Params: {} 2024-03-04T17:53:30.753 DEBUG:boto:Token: None 2024-03-04T17:53:30.753 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.753 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I= 2024-03-04T17:53:30.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I=', 'Content-Length': '0'} 2024-03-04T17:53:30.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3992cdb4c3e0f5c-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.757 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.757 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.757 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.757 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.757 DEBUG:boto:Method: GET 2024-03-04T17:53:30.757 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.757 DEBUG:boto:Data: 2024-03-04T17:53:30.757 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.757 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.757 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.758 DEBUG:boto:Params: {} 2024-03-04T17:53:30.758 DEBUG:boto:Token: None 2024-03-04T17:53:30.758 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.758 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000428cdec2d2b73694-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.762 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.763 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.763 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.763 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.763 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.763 DEBUG:boto:Method: GET 2024-03-04T17:53:30.763 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.763 DEBUG:boto:Data: 2024-03-04T17:53:30.763 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.764 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.764 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.764 DEBUG:boto:Params: {} 2024-03-04T17:53:30.764 DEBUG:boto:Token: None 2024-03-04T17:53:30.764 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.764 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c50e19cb2d49b04b-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.816 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:30.816 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.817 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.817 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.817 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.817 DEBUG:boto:Method: GET 2024-03-04T17:53:30.817 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.817 DEBUG:boto:Data: 2024-03-04T17:53:30.817 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.817 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.817 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.817 DEBUG:boto:Params: {} 2024-03-04T17:53:30.817 DEBUG:boto:Token: None 2024-03-04T17:53:30.818 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.818 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LhNWEfdccvRku214zWwJHZZPAkQ= 2024-03-04T17:53:30.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LhNWEfdccvRku214zWwJHZZPAkQ=', 'Content-Length': '0'} 2024-03-04T17:53:30.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000170133aac3c9c9ad-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.821 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.821 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.821 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.821 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.821 DEBUG:boto:Method: GET 2024-03-04T17:53:30.821 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.822 DEBUG:boto:Data: 2024-03-04T17:53:30.822 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.822 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.822 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.822 DEBUG:boto:Params: {} 2024-03-04T17:53:30.822 DEBUG:boto:Token: None 2024-03-04T17:53:30.822 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.822 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LhNWEfdccvRku214zWwJHZZPAkQ= 2024-03-04T17:53:30.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LhNWEfdccvRku214zWwJHZZPAkQ=', 'Content-Length': '0'} 2024-03-04T17:53:30.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff4c79a2ee5d6a1f-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.825 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.825 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.826 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.826 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.826 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.826 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.826 DEBUG:boto:Data: 2024-03-04T17:53:30.826 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.826 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.826 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.826 DEBUG:boto:Params: {} 2024-03-04T17:53:30.826 DEBUG:boto:Token: None 2024-03-04T17:53:30.827 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.827 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qtpd0+GBx6R4bfz99kA+MzHpDI4= 2024-03-04T17:53:30.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qtpd0+GBx6R4bfz99kA+MzHpDI4=', 'Content-Length': '0'} 2024-03-04T17:53:30.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005216c571619f821b-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.830 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.830 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.830 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.830 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.830 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.830 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.830 DEBUG:boto:Data: 2024-03-04T17:53:30.831 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.831 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.831 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.831 DEBUG:boto:Params: {} 2024-03-04T17:53:30.831 DEBUG:boto:Token: None 2024-03-04T17:53:30.831 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.831 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qtpd0+GBx6R4bfz99kA+MzHpDI4= 2024-03-04T17:53:30.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qtpd0+GBx6R4bfz99kA+MzHpDI4=', 'Content-Length': '0'} 2024-03-04T17:53:30.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a06c8659e828c3e-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:30.835 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.835 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.835 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.835 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.835 DEBUG:boto:Method: GET 2024-03-04T17:53:30.835 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.835 DEBUG:boto:Data: 2024-03-04T17:53:30.835 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.835 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.835 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.836 DEBUG:boto:Params: {} 2024-03-04T17:53:30.836 DEBUG:boto:Token: None 2024-03-04T17:53:30.836 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.836 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LhNWEfdccvRku214zWwJHZZPAkQ= 2024-03-04T17:53:30.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LhNWEfdccvRku214zWwJHZZPAkQ=', 'Content-Length': '0'} 2024-03-04T17:53:30.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000443434126dbd8fc1-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.880 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.880 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.880 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.880 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.880 DEBUG:boto:Method: GET 2024-03-04T17:53:30.880 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.881 DEBUG:boto:Data: 2024-03-04T17:53:30.881 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.881 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.881 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.881 DEBUG:boto:Params: {} 2024-03-04T17:53:30.881 DEBUG:boto:Token: None 2024-03-04T17:53:30.881 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:30.881 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LhNWEfdccvRku214zWwJHZZPAkQ= 2024-03-04T17:53:30.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LhNWEfdccvRku214zWwJHZZPAkQ=', 'Content-Length': '0'} 2024-03-04T17:53:30.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000effd99ee25171119-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.885 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.885 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.885 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.885 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.885 DEBUG:boto:Method: GET 2024-03-04T17:53:30.885 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.885 DEBUG:boto:Data: 2024-03-04T17:53:30.885 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.886 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.886 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.886 DEBUG:boto:Params: {} 2024-03-04T17:53:30.886 DEBUG:boto:Token: None 2024-03-04T17:53:30.886 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.886 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009eeff68abc8998b-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.891 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.891 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:30.891 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:30.892 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.892 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:30.892 DEBUG:boto:Method: GET 2024-03-04T17:53:30.892 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:30.892 DEBUG:boto:Data: 2024-03-04T17:53:30.892 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.892 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.892 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.892 DEBUG:boto:Params: {} 2024-03-04T17:53:30.892 DEBUG:boto:Token: None 2024-03-04T17:53:30.893 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/ 2024-03-04T17:53:30.893 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M= 2024-03-04T17:53:30.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ycU5ZLFEdJs3auzlfxXrOfjmp6M=', 'Content-Length': '0'} 2024-03-04T17:53:30.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090f8af00a97237e3-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.940 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:30.940 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.940 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.941 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.941 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:30.941 DEBUG:boto:Data: 2024-03-04T17:53:30.941 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.941 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.941 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.941 DEBUG:boto:Params: {} 2024-03-04T17:53:30.941 DEBUG:boto:Token: None 2024-03-04T17:53:30.941 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj 2024-03-04T17:53:30.941 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw= 2024-03-04T17:53:30.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw=', 'Content-Length': '0'} 2024-03-04T17:53:30.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3fcd85a89d0332e-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.948 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.948 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.948 DEBUG:boto:Method: HEAD 2024-03-04T17:53:30.948 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:30.948 DEBUG:boto:Data: 2024-03-04T17:53:30.948 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.949 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:30.949 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.949 DEBUG:boto:Params: {} 2024-03-04T17:53:30.949 DEBUG:boto:Token: None 2024-03-04T17:53:30.949 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj 2024-03-04T17:53:30.949 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw= 2024-03-04T17:53:30.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IKUw8xUUAFykFaMrGaf/SjCfBOw=', 'Content-Length': '0'} 2024-03-04T17:53:30.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090cce5a2b4bee88a-0065e60a9a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:30.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:30.955 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:30.955 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:30.956 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.956 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.956 DEBUG:boto:Method: GET 2024-03-04T17:53:30.956 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.956 DEBUG:boto:Data: 2024-03-04T17:53:30.956 DEBUG:boto:Headers: {} 2024-03-04T17:53:30.956 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:30.956 DEBUG:boto:Port: 8000 2024-03-04T17:53:30.956 DEBUG:boto:Params: {} 2024-03-04T17:53:30.956 DEBUG:boto:Token: None 2024-03-04T17:53:30.957 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:30 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:30.957 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I= 2024-03-04T17:53:30.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qF7e5PE1KQL4nKVm2sScCg6c76I=', 'Content-Length': '0'} 2024-03-04T17:53:30.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aebf8272031ecb63-0065e60a9a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.000 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.000 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.000 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.000 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.000 DEBUG:boto:Method: GET 2024-03-04T17:53:31.000 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.001 DEBUG:boto:Data: 2024-03-04T17:53:31.001 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.001 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.001 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.001 DEBUG:boto:Params: {} 2024-03-04T17:53:31.001 DEBUG:boto:Token: None 2024-03-04T17:53:31.001 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.001 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo= 2024-03-04T17:53:31.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo=', 'Content-Length': '0'} 2024-03-04T17:53:31.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba6c745762e53434-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.004 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.005 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.005 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.005 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.005 DEBUG:boto:Method: GET 2024-03-04T17:53:31.005 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.005 DEBUG:boto:Data: 2024-03-04T17:53:31.005 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.005 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.005 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.005 DEBUG:boto:Params: {} 2024-03-04T17:53:31.005 DEBUG:boto:Token: None 2024-03-04T17:53:31.006 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.006 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002315ca1d4ab9eaaa-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.052 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.052 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.053 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.053 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.053 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.053 DEBUG:boto:Method: GET 2024-03-04T17:53:31.053 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.053 DEBUG:boto:Data: 2024-03-04T17:53:31.053 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.053 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.053 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.053 DEBUG:boto:Params: {} 2024-03-04T17:53:31.054 DEBUG:boto:Token: None 2024-03-04T17:53:31.054 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.054 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071ae9353b1dc0d7c-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.058 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:31.059 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.059 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.059 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.059 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.059 DEBUG:boto:Method: GET 2024-03-04T17:53:31.060 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.060 DEBUG:boto:Data: 2024-03-04T17:53:31.060 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.060 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.060 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.060 DEBUG:boto:Params: {} 2024-03-04T17:53:31.060 DEBUG:boto:Token: None 2024-03-04T17:53:31.060 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.060 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PTpZ66tIjLNNqt05s92ISDdL43s= 2024-03-04T17:53:31.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PTpZ66tIjLNNqt05s92ISDdL43s=', 'Content-Length': '0'} 2024-03-04T17:53:31.063 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e41f23266000571c-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.064 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.064 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.064 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.064 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.064 DEBUG:boto:Method: GET 2024-03-04T17:53:31.064 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.064 DEBUG:boto:Data: 2024-03-04T17:53:31.064 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.065 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.065 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.065 DEBUG:boto:Params: {} 2024-03-04T17:53:31.065 DEBUG:boto:Token: None 2024-03-04T17:53:31.065 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.065 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PTpZ66tIjLNNqt05s92ISDdL43s= 2024-03-04T17:53:31.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PTpZ66tIjLNNqt05s92ISDdL43s=', 'Content-Length': '0'} 2024-03-04T17:53:31.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006500bd98933d876d-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.068 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.069 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.069 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.069 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.069 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.069 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.069 DEBUG:boto:Data: 2024-03-04T17:53:31.069 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.069 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.069 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.069 DEBUG:boto:Params: {} 2024-03-04T17:53:31.070 DEBUG:boto:Token: None 2024-03-04T17:53:31.070 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.070 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s/xs9IpXnJsHnCO74xZZBoiUZxw= 2024-03-04T17:53:31.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s/xs9IpXnJsHnCO74xZZBoiUZxw=', 'Content-Length': '0'} 2024-03-04T17:53:31.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085cb55b2b9a371e0-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.073 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.073 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.073 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.073 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.073 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.073 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.074 DEBUG:boto:Data: 2024-03-04T17:53:31.074 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.074 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.074 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.074 DEBUG:boto:Params: {} 2024-03-04T17:53:31.074 DEBUG:boto:Token: None 2024-03-04T17:53:31.074 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.074 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s/xs9IpXnJsHnCO74xZZBoiUZxw= 2024-03-04T17:53:31.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s/xs9IpXnJsHnCO74xZZBoiUZxw=', 'Content-Length': '0'} 2024-03-04T17:53:31.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5e7755605f6bb3a-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:31.078 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.078 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.078 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.078 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.078 DEBUG:boto:Method: GET 2024-03-04T17:53:31.078 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.078 DEBUG:boto:Data: 2024-03-04T17:53:31.078 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.079 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.079 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.079 DEBUG:boto:Params: {} 2024-03-04T17:53:31.079 DEBUG:boto:Token: None 2024-03-04T17:53:31.079 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.079 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PTpZ66tIjLNNqt05s92ISDdL43s= 2024-03-04T17:53:31.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PTpZ66tIjLNNqt05s92ISDdL43s=', 'Content-Length': '0'} 2024-03-04T17:53:31.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002286dd377e2fff41-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.124 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.124 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.124 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.124 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.124 DEBUG:boto:Method: GET 2024-03-04T17:53:31.124 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.125 DEBUG:boto:Data: 2024-03-04T17:53:31.125 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.125 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.125 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.125 DEBUG:boto:Params: {} 2024-03-04T17:53:31.125 DEBUG:boto:Token: None 2024-03-04T17:53:31.125 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:31.125 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PTpZ66tIjLNNqt05s92ISDdL43s= 2024-03-04T17:53:31.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PTpZ66tIjLNNqt05s92ISDdL43s=', 'Content-Length': '0'} 2024-03-04T17:53:31.128 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d77b623e6cf3cf15-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.128 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.129 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.129 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.129 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.129 DEBUG:boto:Method: GET 2024-03-04T17:53:31.129 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.129 DEBUG:boto:Data: 2024-03-04T17:53:31.129 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.129 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.129 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.129 DEBUG:boto:Params: {} 2024-03-04T17:53:31.129 DEBUG:boto:Token: None 2024-03-04T17:53:31.130 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.130 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c248d53d6e43b64-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.180 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.180 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.180 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.181 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.181 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.181 DEBUG:boto:Method: GET 2024-03-04T17:53:31.181 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.181 DEBUG:boto:Data: 2024-03-04T17:53:31.181 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.181 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.181 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.181 DEBUG:boto:Params: {} 2024-03-04T17:53:31.181 DEBUG:boto:Token: None 2024-03-04T17:53:31.181 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.182 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cb1b34dfa1f8019-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.187 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.187 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.187 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.187 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.187 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:31.187 DEBUG:boto:Data: 2024-03-04T17:53:31.187 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.187 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.188 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.188 DEBUG:boto:Params: {} 2024-03-04T17:53:31.188 DEBUG:boto:Token: None 2024-03-04T17:53:31.188 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj 2024-03-04T17:53:31.188 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw= 2024-03-04T17:53:31.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw=', 'Content-Length': '0'} 2024-03-04T17:53:31.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016dcd8d61ecc88f5-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.195 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.195 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.195 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.195 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:31.195 DEBUG:boto:Data: 2024-03-04T17:53:31.195 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.195 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.195 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.196 DEBUG:boto:Params: {} 2024-03-04T17:53:31.196 DEBUG:boto:Token: None 2024-03-04T17:53:31.196 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj 2024-03-04T17:53:31.196 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw= 2024-03-04T17:53:31.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw=', 'Content-Length': '0'} 2024-03-04T17:53:31.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df4b0e27f3f40517-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:31.202 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.203 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.203 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.203 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.203 DEBUG:boto:Method: GET 2024-03-04T17:53:31.203 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.203 DEBUG:boto:Data: 2024-03-04T17:53:31.203 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.203 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.203 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.203 DEBUG:boto:Params: {} 2024-03-04T17:53:31.204 DEBUG:boto:Token: None 2024-03-04T17:53:31.204 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.204 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo= 2024-03-04T17:53:31.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo=', 'Content-Length': '0'} 2024-03-04T17:53:31.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ada7ba1006b4b7f0-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.248 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.248 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.248 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.248 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.249 DEBUG:boto:Method: GET 2024-03-04T17:53:31.249 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.249 DEBUG:boto:Data: 2024-03-04T17:53:31.249 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.249 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.249 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.249 DEBUG:boto:Params: {} 2024-03-04T17:53:31.249 DEBUG:boto:Token: None 2024-03-04T17:53:31.249 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.250 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo= 2024-03-04T17:53:31.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo=', 'Content-Length': '0'} 2024-03-04T17:53:31.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000297a1f6c7b6248b4-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.253 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.253 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.254 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.254 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.254 DEBUG:boto:Method: GET 2024-03-04T17:53:31.254 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.254 DEBUG:boto:Data: 2024-03-04T17:53:31.254 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.254 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.254 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.254 DEBUG:boto:Params: {} 2024-03-04T17:53:31.254 DEBUG:boto:Token: None 2024-03-04T17:53:31.255 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.255 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2f214bc3a998acb-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.300 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.300 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.300 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.301 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.301 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.301 DEBUG:boto:Method: GET 2024-03-04T17:53:31.301 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.301 DEBUG:boto:Data: 2024-03-04T17:53:31.301 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.301 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.301 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.301 DEBUG:boto:Params: {} 2024-03-04T17:53:31.301 DEBUG:boto:Token: None 2024-03-04T17:53:31.302 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.302 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b14eccfe93b3124-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.309 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:31.310 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.310 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.310 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.310 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.310 DEBUG:boto:Method: GET 2024-03-04T17:53:31.310 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.310 DEBUG:boto:Data: 2024-03-04T17:53:31.311 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.311 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.311 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.311 DEBUG:boto:Params: {} 2024-03-04T17:53:31.311 DEBUG:boto:Token: None 2024-03-04T17:53:31.311 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.311 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YzfE8/GNkbGuF+kWhWjctVtftVk= 2024-03-04T17:53:31.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YzfE8/GNkbGuF+kWhWjctVtftVk=', 'Content-Length': '0'} 2024-03-04T17:53:31.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067a641c309b77ded-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.314 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.314 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.315 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.315 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.315 DEBUG:boto:Method: GET 2024-03-04T17:53:31.315 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.315 DEBUG:boto:Data: 2024-03-04T17:53:31.315 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.315 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.315 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.315 DEBUG:boto:Params: {} 2024-03-04T17:53:31.315 DEBUG:boto:Token: None 2024-03-04T17:53:31.316 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.316 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YzfE8/GNkbGuF+kWhWjctVtftVk= 2024-03-04T17:53:31.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YzfE8/GNkbGuF+kWhWjctVtftVk=', 'Content-Length': '0'} 2024-03-04T17:53:31.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f4513b7430eeadb-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.319 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.319 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.319 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.319 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.320 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.320 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.320 DEBUG:boto:Data: 2024-03-04T17:53:31.320 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.320 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.320 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.320 DEBUG:boto:Params: {} 2024-03-04T17:53:31.320 DEBUG:boto:Token: None 2024-03-04T17:53:31.320 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.321 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5W3rE20KTvifFtYMtRMd+CPX21w= 2024-03-04T17:53:31.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5W3rE20KTvifFtYMtRMd+CPX21w=', 'Content-Length': '0'} 2024-03-04T17:53:31.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051f3e430fe4663d2-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.323 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.324 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.324 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.324 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.324 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.324 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.324 DEBUG:boto:Data: 2024-03-04T17:53:31.324 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.324 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.324 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.325 DEBUG:boto:Params: {} 2024-03-04T17:53:31.325 DEBUG:boto:Token: None 2024-03-04T17:53:31.325 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.325 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5W3rE20KTvifFtYMtRMd+CPX21w= 2024-03-04T17:53:31.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5W3rE20KTvifFtYMtRMd+CPX21w=', 'Content-Length': '0'} 2024-03-04T17:53:31.328 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfa166a3fc306928-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:31.329 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.329 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.329 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.329 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.329 DEBUG:boto:Method: GET 2024-03-04T17:53:31.329 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.329 DEBUG:boto:Data: 2024-03-04T17:53:31.329 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.329 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.329 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.329 DEBUG:boto:Params: {} 2024-03-04T17:53:31.330 DEBUG:boto:Token: None 2024-03-04T17:53:31.330 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.330 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YzfE8/GNkbGuF+kWhWjctVtftVk= 2024-03-04T17:53:31.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YzfE8/GNkbGuF+kWhWjctVtftVk=', 'Content-Length': '0'} 2024-03-04T17:53:31.333 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9155be46e354896-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.333 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.333 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.333 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.333 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.333 DEBUG:boto:Method: GET 2024-03-04T17:53:31.334 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.334 DEBUG:boto:Data: 2024-03-04T17:53:31.334 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.334 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.334 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.334 DEBUG:boto:Params: {} 2024-03-04T17:53:31.334 DEBUG:boto:Token: None 2024-03-04T17:53:31.334 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:31.334 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YzfE8/GNkbGuF+kWhWjctVtftVk= 2024-03-04T17:53:31.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YzfE8/GNkbGuF+kWhWjctVtftVk=', 'Content-Length': '0'} 2024-03-04T17:53:31.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066ddbe9c34a49ed9-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.380 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.380 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.380 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.381 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.381 DEBUG:boto:Method: GET 2024-03-04T17:53:31.381 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.381 DEBUG:boto:Data: 2024-03-04T17:53:31.381 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.381 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.381 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.381 DEBUG:boto:Params: {} 2024-03-04T17:53:31.381 DEBUG:boto:Token: None 2024-03-04T17:53:31.381 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.382 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3dfc0a3be450539-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.386 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.387 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.387 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.387 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.387 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.387 DEBUG:boto:Method: GET 2024-03-04T17:53:31.387 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.387 DEBUG:boto:Data: 2024-03-04T17:53:31.387 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.387 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.387 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.388 DEBUG:boto:Params: {} 2024-03-04T17:53:31.388 DEBUG:boto:Token: None 2024-03-04T17:53:31.388 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.388 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afba189186604fff-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.393 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.394 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.394 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.394 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.394 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:31.394 DEBUG:boto:Data: 2024-03-04T17:53:31.394 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.394 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.394 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.394 DEBUG:boto:Params: {} 2024-03-04T17:53:31.395 DEBUG:boto:Token: None 2024-03-04T17:53:31.395 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj 2024-03-04T17:53:31.395 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw= 2024-03-04T17:53:31.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw=', 'Content-Length': '0'} 2024-03-04T17:53:31.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029cadae362a65e6d-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.401 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.401 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.401 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.402 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:31.402 DEBUG:boto:Data: 2024-03-04T17:53:31.402 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.402 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.402 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.402 DEBUG:boto:Params: {} 2024-03-04T17:53:31.402 DEBUG:boto:Token: None 2024-03-04T17:53:31.402 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj 2024-03-04T17:53:31.402 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw= 2024-03-04T17:53:31.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw=', 'Content-Length': '0'} 2024-03-04T17:53:31.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fe129218f23c32f-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:31.409 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.409 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.409 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.409 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.410 DEBUG:boto:Method: GET 2024-03-04T17:53:31.410 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.410 DEBUG:boto:Data: 2024-03-04T17:53:31.410 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.410 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.410 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.410 DEBUG:boto:Params: {} 2024-03-04T17:53:31.410 DEBUG:boto:Token: None 2024-03-04T17:53:31.411 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.411 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo= 2024-03-04T17:53:31.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo=', 'Content-Length': '0'} 2024-03-04T17:53:31.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000181ca7dea22ffdd8-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.456 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.456 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.456 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.456 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.456 DEBUG:boto:Method: GET 2024-03-04T17:53:31.456 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.457 DEBUG:boto:Data: 2024-03-04T17:53:31.457 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.457 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.457 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.457 DEBUG:boto:Params: {} 2024-03-04T17:53:31.457 DEBUG:boto:Token: None 2024-03-04T17:53:31.457 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.457 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo= 2024-03-04T17:53:31.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo=', 'Content-Length': '0'} 2024-03-04T17:53:31.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d545ac424d4d3df9-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.461 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.461 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.461 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.461 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.461 DEBUG:boto:Method: GET 2024-03-04T17:53:31.461 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.461 DEBUG:boto:Data: 2024-03-04T17:53:31.461 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.461 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.461 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.462 DEBUG:boto:Params: {} 2024-03-04T17:53:31.462 DEBUG:boto:Token: None 2024-03-04T17:53:31.462 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.462 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d99ce4103011cdf9-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.467 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.467 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.468 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.468 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.468 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.468 DEBUG:boto:Method: GET 2024-03-04T17:53:31.468 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.468 DEBUG:boto:Data: 2024-03-04T17:53:31.468 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.468 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.468 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.469 DEBUG:boto:Params: {} 2024-03-04T17:53:31.469 DEBUG:boto:Token: None 2024-03-04T17:53:31.469 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.469 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de3be018e0ff06ab-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.516 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:31.516 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.516 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.517 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.517 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.517 DEBUG:boto:Method: GET 2024-03-04T17:53:31.517 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.517 DEBUG:boto:Data: 2024-03-04T17:53:31.517 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.517 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.517 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.517 DEBUG:boto:Params: {} 2024-03-04T17:53:31.517 DEBUG:boto:Token: None 2024-03-04T17:53:31.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.518 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F40C7ba4aLxJWpRDLNQCrw9B0O4= 2024-03-04T17:53:31.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F40C7ba4aLxJWpRDLNQCrw9B0O4=', 'Content-Length': '0'} 2024-03-04T17:53:31.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007244d505c0e96330-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.521 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.521 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.521 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.521 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.521 DEBUG:boto:Method: GET 2024-03-04T17:53:31.521 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.521 DEBUG:boto:Data: 2024-03-04T17:53:31.522 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.522 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.522 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.522 DEBUG:boto:Params: {} 2024-03-04T17:53:31.522 DEBUG:boto:Token: None 2024-03-04T17:53:31.522 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.522 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F40C7ba4aLxJWpRDLNQCrw9B0O4= 2024-03-04T17:53:31.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F40C7ba4aLxJWpRDLNQCrw9B0O4=', 'Content-Length': '0'} 2024-03-04T17:53:31.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9aa7213c3a7af5b-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.525 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.526 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.526 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.526 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.526 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.526 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.526 DEBUG:boto:Data: 2024-03-04T17:53:31.526 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.526 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.527 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.527 DEBUG:boto:Params: {} 2024-03-04T17:53:31.527 DEBUG:boto:Token: None 2024-03-04T17:53:31.527 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.527 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mCFJNN1QKMWk3F+eUyWgYToGWw= 2024-03-04T17:53:31.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mCFJNN1QKMWk3F+eUyWgYToGWw=', 'Content-Length': '0'} 2024-03-04T17:53:31.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097a004d4b6bccba4-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.530 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.531 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.531 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.531 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.531 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.531 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.531 DEBUG:boto:Data: 2024-03-04T17:53:31.531 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.531 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.531 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.531 DEBUG:boto:Params: {} 2024-03-04T17:53:31.531 DEBUG:boto:Token: None 2024-03-04T17:53:31.532 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.532 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mCFJNN1QKMWk3F+eUyWgYToGWw= 2024-03-04T17:53:31.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mCFJNN1QKMWk3F+eUyWgYToGWw=', 'Content-Length': '0'} 2024-03-04T17:53:31.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c1daf16d12fcd53-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:31.535 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.535 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.535 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.535 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.536 DEBUG:boto:Method: GET 2024-03-04T17:53:31.536 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.536 DEBUG:boto:Data: 2024-03-04T17:53:31.536 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.536 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.536 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.536 DEBUG:boto:Params: {} 2024-03-04T17:53:31.536 DEBUG:boto:Token: None 2024-03-04T17:53:31.536 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.537 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F40C7ba4aLxJWpRDLNQCrw9B0O4= 2024-03-04T17:53:31.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F40C7ba4aLxJWpRDLNQCrw9B0O4=', 'Content-Length': '0'} 2024-03-04T17:53:31.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f52029fe50cf39d-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.580 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.580 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.580 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.580 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.580 DEBUG:boto:Method: GET 2024-03-04T17:53:31.580 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.581 DEBUG:boto:Data: 2024-03-04T17:53:31.581 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.581 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.581 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.581 DEBUG:boto:Params: {} 2024-03-04T17:53:31.581 DEBUG:boto:Token: None 2024-03-04T17:53:31.581 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:31.581 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F40C7ba4aLxJWpRDLNQCrw9B0O4= 2024-03-04T17:53:31.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F40C7ba4aLxJWpRDLNQCrw9B0O4=', 'Content-Length': '0'} 2024-03-04T17:53:31.584 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3697f5198a8eb66-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.584 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.585 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.585 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.585 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.585 DEBUG:boto:Method: GET 2024-03-04T17:53:31.585 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.585 DEBUG:boto:Data: 2024-03-04T17:53:31.585 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.585 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.585 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.586 DEBUG:boto:Params: {} 2024-03-04T17:53:31.586 DEBUG:boto:Token: None 2024-03-04T17:53:31.586 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.586 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026e5242effc12dc7-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.591 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.592 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.592 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.592 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.592 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.592 DEBUG:boto:Method: GET 2024-03-04T17:53:31.592 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.592 DEBUG:boto:Data: 2024-03-04T17:53:31.593 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.593 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.593 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.593 DEBUG:boto:Params: {} 2024-03-04T17:53:31.593 DEBUG:boto:Token: None 2024-03-04T17:53:31.593 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.593 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7855e076bae61d0-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.640 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.640 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.640 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.640 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.641 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:31.641 DEBUG:boto:Data: 2024-03-04T17:53:31.641 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.641 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.641 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.641 DEBUG:boto:Params: {} 2024-03-04T17:53:31.641 DEBUG:boto:Token: None 2024-03-04T17:53:31.641 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj 2024-03-04T17:53:31.641 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw= 2024-03-04T17:53:31.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw=', 'Content-Length': '0'} 2024-03-04T17:53:31.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f520b3101718c032-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.648 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.648 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.649 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.649 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:31.649 DEBUG:boto:Data: 2024-03-04T17:53:31.649 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.649 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.649 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.649 DEBUG:boto:Params: {} 2024-03-04T17:53:31.649 DEBUG:boto:Token: None 2024-03-04T17:53:31.649 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj 2024-03-04T17:53:31.649 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw= 2024-03-04T17:53:31.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw=', 'Content-Length': '0'} 2024-03-04T17:53:31.655 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa9f4b2ef5889bcb-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:31.656 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.656 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.656 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.656 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.656 DEBUG:boto:Method: GET 2024-03-04T17:53:31.656 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.656 DEBUG:boto:Data: 2024-03-04T17:53:31.657 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.657 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.657 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.657 DEBUG:boto:Params: {} 2024-03-04T17:53:31.657 DEBUG:boto:Token: None 2024-03-04T17:53:31.657 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.657 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo= 2024-03-04T17:53:31.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo=', 'Content-Length': '0'} 2024-03-04T17:53:31.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a245e1152a6382b5-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.704 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.704 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.704 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.704 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.705 DEBUG:boto:Method: GET 2024-03-04T17:53:31.705 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.705 DEBUG:boto:Data: 2024-03-04T17:53:31.705 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.705 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.705 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.705 DEBUG:boto:Params: {} 2024-03-04T17:53:31.705 DEBUG:boto:Token: None 2024-03-04T17:53:31.705 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.705 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo= 2024-03-04T17:53:31.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo=', 'Content-Length': '0'} 2024-03-04T17:53:31.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001956a59b29cacd17-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.709 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.709 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.710 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.710 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.710 DEBUG:boto:Method: GET 2024-03-04T17:53:31.710 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.710 DEBUG:boto:Data: 2024-03-04T17:53:31.710 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.710 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.710 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.710 DEBUG:boto:Params: {} 2024-03-04T17:53:31.711 DEBUG:boto:Token: None 2024-03-04T17:53:31.711 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.711 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f1c143cbc7b2d52-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.756 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.756 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.756 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.757 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.757 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.757 DEBUG:boto:Method: GET 2024-03-04T17:53:31.757 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.757 DEBUG:boto:Data: 2024-03-04T17:53:31.757 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.757 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.757 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.757 DEBUG:boto:Params: {} 2024-03-04T17:53:31.758 DEBUG:boto:Token: None 2024-03-04T17:53:31.758 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.758 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad78afae66cbd311-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.763 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:31.763 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.763 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.763 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.763 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.764 DEBUG:boto:Method: GET 2024-03-04T17:53:31.764 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.764 DEBUG:boto:Data: 2024-03-04T17:53:31.764 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.764 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.764 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.764 DEBUG:boto:Params: {} 2024-03-04T17:53:31.764 DEBUG:boto:Token: None 2024-03-04T17:53:31.764 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.764 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DTmcpC4FhD9mES+P5ja8E5FOSag= 2024-03-04T17:53:31.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DTmcpC4FhD9mES+P5ja8E5FOSag=', 'Content-Length': '0'} 2024-03-04T17:53:31.767 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063d2694006afdb22-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.768 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.768 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.768 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.768 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.768 DEBUG:boto:Method: GET 2024-03-04T17:53:31.768 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.768 DEBUG:boto:Data: 2024-03-04T17:53:31.769 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.769 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.769 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.769 DEBUG:boto:Params: {} 2024-03-04T17:53:31.769 DEBUG:boto:Token: None 2024-03-04T17:53:31.769 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.769 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DTmcpC4FhD9mES+P5ja8E5FOSag= 2024-03-04T17:53:31.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DTmcpC4FhD9mES+P5ja8E5FOSag=', 'Content-Length': '0'} 2024-03-04T17:53:31.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013de05dbdb2d9b9f-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.773 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.773 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.773 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.773 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.773 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.773 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.773 DEBUG:boto:Data: 2024-03-04T17:53:31.774 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.774 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.774 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.774 DEBUG:boto:Params: {} 2024-03-04T17:53:31.774 DEBUG:boto:Token: None 2024-03-04T17:53:31.774 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.774 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:n7ttdfLftoYbbnj6KbtMMeKN+pg= 2024-03-04T17:53:31.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:n7ttdfLftoYbbnj6KbtMMeKN+pg=', 'Content-Length': '0'} 2024-03-04T17:53:31.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed64dbe5bbb1c67a-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.777 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.777 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.777 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.777 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.778 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.778 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.778 DEBUG:boto:Data: 2024-03-04T17:53:31.778 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.778 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.778 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.778 DEBUG:boto:Params: {} 2024-03-04T17:53:31.778 DEBUG:boto:Token: None 2024-03-04T17:53:31.778 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.779 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:n7ttdfLftoYbbnj6KbtMMeKN+pg= 2024-03-04T17:53:31.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:n7ttdfLftoYbbnj6KbtMMeKN+pg=', 'Content-Length': '0'} 2024-03-04T17:53:31.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee2c8b55a430cda7-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:31.782 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.782 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.782 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.782 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.782 DEBUG:boto:Method: GET 2024-03-04T17:53:31.783 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.783 DEBUG:boto:Data: 2024-03-04T17:53:31.783 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.783 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.783 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.783 DEBUG:boto:Params: {} 2024-03-04T17:53:31.783 DEBUG:boto:Token: None 2024-03-04T17:53:31.783 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.783 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DTmcpC4FhD9mES+P5ja8E5FOSag= 2024-03-04T17:53:31.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DTmcpC4FhD9mES+P5ja8E5FOSag=', 'Content-Length': '0'} 2024-03-04T17:53:31.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec115a009e1609ec-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.828 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.828 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.828 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.828 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.828 DEBUG:boto:Method: GET 2024-03-04T17:53:31.829 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.829 DEBUG:boto:Data: 2024-03-04T17:53:31.829 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.829 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.829 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.829 DEBUG:boto:Params: {} 2024-03-04T17:53:31.829 DEBUG:boto:Token: None 2024-03-04T17:53:31.829 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:31.830 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DTmcpC4FhD9mES+P5ja8E5FOSag= 2024-03-04T17:53:31.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DTmcpC4FhD9mES+P5ja8E5FOSag=', 'Content-Length': '0'} 2024-03-04T17:53:31.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9c45567ea7857ec-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.833 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.833 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.833 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.833 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.834 DEBUG:boto:Method: GET 2024-03-04T17:53:31.834 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.834 DEBUG:boto:Data: 2024-03-04T17:53:31.834 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.834 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.834 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.834 DEBUG:boto:Params: {} 2024-03-04T17:53:31.834 DEBUG:boto:Token: None 2024-03-04T17:53:31.834 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.835 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6ed1cb6363e7914-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.884 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.884 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.885 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.885 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.885 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.885 DEBUG:boto:Method: GET 2024-03-04T17:53:31.885 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.885 DEBUG:boto:Data: 2024-03-04T17:53:31.885 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.885 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.885 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.885 DEBUG:boto:Params: {} 2024-03-04T17:53:31.885 DEBUG:boto:Token: None 2024-03-04T17:53:31.886 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.886 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007905cfc7f5032c31-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.891 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:31.891 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.891 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.891 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.891 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:31.891 DEBUG:boto:Data: 2024-03-04T17:53:31.891 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.891 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.892 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.892 DEBUG:boto:Params: {} 2024-03-04T17:53:31.892 DEBUG:boto:Token: None 2024-03-04T17:53:31.892 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj 2024-03-04T17:53:31.892 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw= 2024-03-04T17:53:31.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw=', 'Content-Length': '0'} 2024-03-04T17:53:31.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f225ce82d79e8562-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.898 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.898 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.898 DEBUG:boto:Method: HEAD 2024-03-04T17:53:31.899 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:31.899 DEBUG:boto:Data: 2024-03-04T17:53:31.899 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.899 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.899 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.899 DEBUG:boto:Params: {} 2024-03-04T17:53:31.899 DEBUG:boto:Token: None 2024-03-04T17:53:31.899 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj 2024-03-04T17:53:31.899 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw= 2024-03-04T17:53:31.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7qPWha0JG7utMf1l7gRZXkLCxnw=', 'Content-Length': '0'} 2024-03-04T17:53:31.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000444930bc311996fb-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:31.906 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.906 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.906 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.906 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.906 DEBUG:boto:Method: GET 2024-03-04T17:53:31.906 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.906 DEBUG:boto:Data: 2024-03-04T17:53:31.906 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.906 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.906 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.907 DEBUG:boto:Params: {} 2024-03-04T17:53:31.907 DEBUG:boto:Token: None 2024-03-04T17:53:31.907 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.907 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo= 2024-03-04T17:53:31.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo=', 'Content-Length': '0'} 2024-03-04T17:53:31.910 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f237c26c8cb9284-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.952 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:31.952 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:31.952 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.952 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.952 DEBUG:boto:Method: GET 2024-03-04T17:53:31.953 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.953 DEBUG:boto:Data: 2024-03-04T17:53:31.953 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.953 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:31.953 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.953 DEBUG:boto:Params: {} 2024-03-04T17:53:31.953 DEBUG:boto:Token: None 2024-03-04T17:53:31.953 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:31.953 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo= 2024-03-04T17:53:31.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5PTYsEOMJZ/UtmrNem2VHhIVjxo=', 'Content-Length': '0'} 2024-03-04T17:53:31.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097feda335955d4d6-0065e60a9b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:31.957 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:31.957 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:31.957 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.957 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:31.957 DEBUG:boto:Method: GET 2024-03-04T17:53:31.958 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:31.958 DEBUG:boto:Data: 2024-03-04T17:53:31.958 DEBUG:boto:Headers: {} 2024-03-04T17:53:31.958 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:31.958 DEBUG:boto:Port: 8000 2024-03-04T17:53:31.958 DEBUG:boto:Params: {} 2024-03-04T17:53:31.958 DEBUG:boto:Token: None 2024-03-04T17:53:31.958 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:31 GMT /ppfahn-22/ 2024-03-04T17:53:31.958 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY= 2024-03-04T17:53:31.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:31 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYRIHu1l0fkjJxTrdVsEqIRxibY=', 'Content-Length': '0'} 2024-03-04T17:53:31.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1d8ef6feb3f5095-0065e60a9b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.004 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.004 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.004 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.004 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.005 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.005 DEBUG:boto:Method: GET 2024-03-04T17:53:32.005 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.005 DEBUG:boto:Data: 2024-03-04T17:53:32.005 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.005 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.005 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.005 DEBUG:boto:Params: {} 2024-03-04T17:53:32.005 DEBUG:boto:Token: None 2024-03-04T17:53:32.005 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.006 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7d71d3198217e28-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.010 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:32.011 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.011 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.011 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.011 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.011 DEBUG:boto:Method: GET 2024-03-04T17:53:32.011 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.011 DEBUG:boto:Data: 2024-03-04T17:53:32.011 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.011 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.012 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.012 DEBUG:boto:Params: {} 2024-03-04T17:53:32.012 DEBUG:boto:Token: None 2024-03-04T17:53:32.012 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.012 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SMXok51coJloEiXR3jyoYxgsSBE= 2024-03-04T17:53:32.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SMXok51coJloEiXR3jyoYxgsSBE=', 'Content-Length': '0'} 2024-03-04T17:53:32.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0c9f32991d1dcbc-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.016 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.016 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.016 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.016 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.016 DEBUG:boto:Method: GET 2024-03-04T17:53:32.016 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.016 DEBUG:boto:Data: 2024-03-04T17:53:32.016 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.016 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.017 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.017 DEBUG:boto:Params: {} 2024-03-04T17:53:32.017 DEBUG:boto:Token: None 2024-03-04T17:53:32.017 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.017 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SMXok51coJloEiXR3jyoYxgsSBE= 2024-03-04T17:53:32.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SMXok51coJloEiXR3jyoYxgsSBE=', 'Content-Length': '0'} 2024-03-04T17:53:32.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a09d7c23040e7d0e-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.020 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.020 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.020 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.020 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.021 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.021 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.021 DEBUG:boto:Data: 2024-03-04T17:53:32.021 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.021 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.021 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.021 DEBUG:boto:Params: {} 2024-03-04T17:53:32.021 DEBUG:boto:Token: None 2024-03-04T17:53:32.021 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.021 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6Z0BoLpEcirWcXcds4HDnOy29L8= 2024-03-04T17:53:32.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6Z0BoLpEcirWcXcds4HDnOy29L8=', 'Content-Length': '0'} 2024-03-04T17:53:32.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fa85f61af7ac864-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.025 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.025 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.025 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.025 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.026 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.026 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.026 DEBUG:boto:Data: 2024-03-04T17:53:32.026 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.026 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.026 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.026 DEBUG:boto:Params: {} 2024-03-04T17:53:32.026 DEBUG:boto:Token: None 2024-03-04T17:53:32.026 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.027 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6Z0BoLpEcirWcXcds4HDnOy29L8= 2024-03-04T17:53:32.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6Z0BoLpEcirWcXcds4HDnOy29L8=', 'Content-Length': '0'} 2024-03-04T17:53:32.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9b98cdae2ba381f-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:32.030 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.030 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.030 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.031 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.031 DEBUG:boto:Method: GET 2024-03-04T17:53:32.031 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.031 DEBUG:boto:Data: 2024-03-04T17:53:32.031 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.031 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.031 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.031 DEBUG:boto:Params: {} 2024-03-04T17:53:32.031 DEBUG:boto:Token: None 2024-03-04T17:53:32.032 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.032 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SMXok51coJloEiXR3jyoYxgsSBE= 2024-03-04T17:53:32.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SMXok51coJloEiXR3jyoYxgsSBE=', 'Content-Length': '0'} 2024-03-04T17:53:32.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e7cec703d1a4af1-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.035 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.035 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.035 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.035 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.036 DEBUG:boto:Method: GET 2024-03-04T17:53:32.036 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.036 DEBUG:boto:Data: 2024-03-04T17:53:32.036 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.036 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.036 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.036 DEBUG:boto:Params: {} 2024-03-04T17:53:32.036 DEBUG:boto:Token: None 2024-03-04T17:53:32.036 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:32.037 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SMXok51coJloEiXR3jyoYxgsSBE= 2024-03-04T17:53:32.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SMXok51coJloEiXR3jyoYxgsSBE=', 'Content-Length': '0'} 2024-03-04T17:53:32.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5ae8a945561da91-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.080 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.080 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.080 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.080 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.080 DEBUG:boto:Method: GET 2024-03-04T17:53:32.080 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.081 DEBUG:boto:Data: 2024-03-04T17:53:32.081 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.081 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.081 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.081 DEBUG:boto:Params: {} 2024-03-04T17:53:32.081 DEBUG:boto:Token: None 2024-03-04T17:53:32.081 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.081 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000854f29e00b520f08-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.087 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.087 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.087 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.087 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.087 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.087 DEBUG:boto:Method: GET 2024-03-04T17:53:32.088 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.088 DEBUG:boto:Data: 2024-03-04T17:53:32.088 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.088 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.088 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.088 DEBUG:boto:Params: {} 2024-03-04T17:53:32.088 DEBUG:boto:Token: None 2024-03-04T17:53:32.088 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.089 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4e247cc942a33ef-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.093 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.094 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.094 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.094 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.094 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:32.094 DEBUG:boto:Data: 2024-03-04T17:53:32.094 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.094 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.094 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.094 DEBUG:boto:Params: {} 2024-03-04T17:53:32.095 DEBUG:boto:Token: None 2024-03-04T17:53:32.095 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj 2024-03-04T17:53:32.095 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI= 2024-03-04T17:53:32.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI=', 'Content-Length': '0'} 2024-03-04T17:53:32.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eaea0082a5eab051-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.101 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.101 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.101 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.101 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:32.101 DEBUG:boto:Data: 2024-03-04T17:53:32.102 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.102 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.102 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.102 DEBUG:boto:Params: {} 2024-03-04T17:53:32.102 DEBUG:boto:Token: None 2024-03-04T17:53:32.102 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj 2024-03-04T17:53:32.102 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI= 2024-03-04T17:53:32.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI=', 'Content-Length': '0'} 2024-03-04T17:53:32.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082907f5eaef47e48-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:32.108 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.108 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.108 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.109 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.109 DEBUG:boto:Method: GET 2024-03-04T17:53:32.109 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.109 DEBUG:boto:Data: 2024-03-04T17:53:32.109 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.109 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.109 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.109 DEBUG:boto:Params: {} 2024-03-04T17:53:32.109 DEBUG:boto:Token: None 2024-03-04T17:53:32.109 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.110 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI= 2024-03-04T17:53:32.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI=', 'Content-Length': '0'} 2024-03-04T17:53:32.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000171254498c6e530f-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.156 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.156 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.156 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.156 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.156 DEBUG:boto:Method: GET 2024-03-04T17:53:32.157 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.157 DEBUG:boto:Data: 2024-03-04T17:53:32.157 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.157 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.157 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.157 DEBUG:boto:Params: {} 2024-03-04T17:53:32.157 DEBUG:boto:Token: None 2024-03-04T17:53:32.158 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.158 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI= 2024-03-04T17:53:32.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI=', 'Content-Length': '0'} 2024-03-04T17:53:32.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b2dec927b56096f-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.161 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.161 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.161 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.162 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.162 DEBUG:boto:Method: GET 2024-03-04T17:53:32.162 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.162 DEBUG:boto:Data: 2024-03-04T17:53:32.162 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.162 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.162 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.162 DEBUG:boto:Params: {} 2024-03-04T17:53:32.162 DEBUG:boto:Token: None 2024-03-04T17:53:32.162 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.163 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbde377824a1d24a-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.168 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.168 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.168 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.168 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.169 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.169 DEBUG:boto:Method: GET 2024-03-04T17:53:32.169 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.169 DEBUG:boto:Data: 2024-03-04T17:53:32.169 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.169 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.169 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.169 DEBUG:boto:Params: {} 2024-03-04T17:53:32.169 DEBUG:boto:Token: None 2024-03-04T17:53:32.169 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.170 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000725ae69292455b63-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.216 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:32.216 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.217 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.217 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.217 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.217 DEBUG:boto:Method: GET 2024-03-04T17:53:32.217 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.217 DEBUG:boto:Data: 2024-03-04T17:53:32.217 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.217 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.217 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.217 DEBUG:boto:Params: {} 2024-03-04T17:53:32.218 DEBUG:boto:Token: None 2024-03-04T17:53:32.218 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.218 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aHHV67UBm81Z9twko6PnbCwdLBk= 2024-03-04T17:53:32.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aHHV67UBm81Z9twko6PnbCwdLBk=', 'Content-Length': '0'} 2024-03-04T17:53:32.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093cc9e766e3ed4e6-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.221 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.221 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.221 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.221 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.221 DEBUG:boto:Method: GET 2024-03-04T17:53:32.222 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.222 DEBUG:boto:Data: 2024-03-04T17:53:32.222 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.222 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.222 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.222 DEBUG:boto:Params: {} 2024-03-04T17:53:32.222 DEBUG:boto:Token: None 2024-03-04T17:53:32.222 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.222 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aHHV67UBm81Z9twko6PnbCwdLBk= 2024-03-04T17:53:32.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aHHV67UBm81Z9twko6PnbCwdLBk=', 'Content-Length': '0'} 2024-03-04T17:53:32.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fae05844d208a66a-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.226 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.226 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.226 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.226 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.226 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.226 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.226 DEBUG:boto:Data: 2024-03-04T17:53:32.226 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.226 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.227 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.227 DEBUG:boto:Params: {} 2024-03-04T17:53:32.227 DEBUG:boto:Token: None 2024-03-04T17:53:32.227 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.227 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Bgsote2PBMNj6BqYgBj/uBTbuLg= 2024-03-04T17:53:32.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Bgsote2PBMNj6BqYgBj/uBTbuLg=', 'Content-Length': '0'} 2024-03-04T17:53:32.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bff6699b95482d30-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.230 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.230 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.230 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.230 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.231 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.231 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.231 DEBUG:boto:Data: 2024-03-04T17:53:32.231 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.231 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.231 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.231 DEBUG:boto:Params: {} 2024-03-04T17:53:32.231 DEBUG:boto:Token: None 2024-03-04T17:53:32.232 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.232 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Bgsote2PBMNj6BqYgBj/uBTbuLg= 2024-03-04T17:53:32.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Bgsote2PBMNj6BqYgBj/uBTbuLg=', 'Content-Length': '0'} 2024-03-04T17:53:32.234 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fa264c49e840d2c-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:32.235 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.235 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.235 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.235 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.235 DEBUG:boto:Method: GET 2024-03-04T17:53:32.235 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.235 DEBUG:boto:Data: 2024-03-04T17:53:32.235 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.235 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.236 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.236 DEBUG:boto:Params: {} 2024-03-04T17:53:32.236 DEBUG:boto:Token: None 2024-03-04T17:53:32.236 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.236 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aHHV67UBm81Z9twko6PnbCwdLBk= 2024-03-04T17:53:32.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aHHV67UBm81Z9twko6PnbCwdLBk=', 'Content-Length': '0'} 2024-03-04T17:53:32.239 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041ac64db33728243-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.280 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.280 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.280 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.280 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.280 DEBUG:boto:Method: GET 2024-03-04T17:53:32.280 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.281 DEBUG:boto:Data: 2024-03-04T17:53:32.281 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.281 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.281 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.281 DEBUG:boto:Params: {} 2024-03-04T17:53:32.281 DEBUG:boto:Token: None 2024-03-04T17:53:32.281 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:32.281 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aHHV67UBm81Z9twko6PnbCwdLBk= 2024-03-04T17:53:32.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aHHV67UBm81Z9twko6PnbCwdLBk=', 'Content-Length': '0'} 2024-03-04T17:53:32.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002e4fde98420269d-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.285 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.285 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.285 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.285 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.285 DEBUG:boto:Method: GET 2024-03-04T17:53:32.285 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.285 DEBUG:boto:Data: 2024-03-04T17:53:32.285 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.285 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.286 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.286 DEBUG:boto:Params: {} 2024-03-04T17:53:32.286 DEBUG:boto:Token: None 2024-03-04T17:53:32.286 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.286 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f2b92eeeeca805c-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.291 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.292 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.292 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.292 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.292 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.292 DEBUG:boto:Method: GET 2024-03-04T17:53:32.292 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.293 DEBUG:boto:Data: 2024-03-04T17:53:32.293 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.293 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.293 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.293 DEBUG:boto:Params: {} 2024-03-04T17:53:32.293 DEBUG:boto:Token: None 2024-03-04T17:53:32.293 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.293 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b1e7e315164e0cf-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.340 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.340 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.340 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.341 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.341 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:32.341 DEBUG:boto:Data: 2024-03-04T17:53:32.341 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.341 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.341 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.341 DEBUG:boto:Params: {} 2024-03-04T17:53:32.341 DEBUG:boto:Token: None 2024-03-04T17:53:32.341 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj 2024-03-04T17:53:32.341 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI= 2024-03-04T17:53:32.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI=', 'Content-Length': '0'} 2024-03-04T17:53:32.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074cdaab781d0ff98-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.348 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.348 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.348 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.348 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:32.348 DEBUG:boto:Data: 2024-03-04T17:53:32.348 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.348 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.348 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.348 DEBUG:boto:Params: {} 2024-03-04T17:53:32.349 DEBUG:boto:Token: None 2024-03-04T17:53:32.349 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj 2024-03-04T17:53:32.349 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI= 2024-03-04T17:53:32.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI=', 'Content-Length': '0'} 2024-03-04T17:53:32.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002872dcfef84b0b7f-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:32.355 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.355 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.355 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.355 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.355 DEBUG:boto:Method: GET 2024-03-04T17:53:32.355 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.355 DEBUG:boto:Data: 2024-03-04T17:53:32.355 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.356 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.356 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.356 DEBUG:boto:Params: {} 2024-03-04T17:53:32.356 DEBUG:boto:Token: None 2024-03-04T17:53:32.356 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.356 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI= 2024-03-04T17:53:32.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI=', 'Content-Length': '0'} 2024-03-04T17:53:32.359 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce36008ebde878b2-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.400 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.400 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.400 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.400 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.400 DEBUG:boto:Method: GET 2024-03-04T17:53:32.401 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.401 DEBUG:boto:Data: 2024-03-04T17:53:32.401 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.401 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.401 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.401 DEBUG:boto:Params: {} 2024-03-04T17:53:32.401 DEBUG:boto:Token: None 2024-03-04T17:53:32.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.401 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI= 2024-03-04T17:53:32.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI=', 'Content-Length': '0'} 2024-03-04T17:53:32.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002442e227b5103e68-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.405 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.405 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.405 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.405 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.405 DEBUG:boto:Method: GET 2024-03-04T17:53:32.405 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.405 DEBUG:boto:Data: 2024-03-04T17:53:32.406 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.406 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.406 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.406 DEBUG:boto:Params: {} 2024-03-04T17:53:32.406 DEBUG:boto:Token: None 2024-03-04T17:53:32.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.406 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077392b88b7cf7323-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.452 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.453 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.453 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.453 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.453 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.453 DEBUG:boto:Method: GET 2024-03-04T17:53:32.453 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.453 DEBUG:boto:Data: 2024-03-04T17:53:32.453 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.453 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.454 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.454 DEBUG:boto:Params: {} 2024-03-04T17:53:32.454 DEBUG:boto:Token: None 2024-03-04T17:53:32.454 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.454 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006167754498eb9f74-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.460 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:32.461 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.461 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.461 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.461 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.462 DEBUG:boto:Method: GET 2024-03-04T17:53:32.462 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.462 DEBUG:boto:Data: 2024-03-04T17:53:32.462 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.462 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.462 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.462 DEBUG:boto:Params: {} 2024-03-04T17:53:32.462 DEBUG:boto:Token: None 2024-03-04T17:53:32.462 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.462 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bsIhCEE15qemq9Cd4JRgopYaezM= 2024-03-04T17:53:32.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bsIhCEE15qemq9Cd4JRgopYaezM=', 'Content-Length': '0'} 2024-03-04T17:53:32.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080bcdc985b7bdb87-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.466 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.466 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.466 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.466 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.466 DEBUG:boto:Method: GET 2024-03-04T17:53:32.466 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.467 DEBUG:boto:Data: 2024-03-04T17:53:32.467 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.467 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.467 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.467 DEBUG:boto:Params: {} 2024-03-04T17:53:32.467 DEBUG:boto:Token: None 2024-03-04T17:53:32.467 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.467 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bsIhCEE15qemq9Cd4JRgopYaezM= 2024-03-04T17:53:32.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bsIhCEE15qemq9Cd4JRgopYaezM=', 'Content-Length': '0'} 2024-03-04T17:53:32.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003eade44b1a1763ea-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.471 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.471 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.471 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.471 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.471 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.471 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.472 DEBUG:boto:Data: 2024-03-04T17:53:32.472 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.472 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.472 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.472 DEBUG:boto:Params: {} 2024-03-04T17:53:32.472 DEBUG:boto:Token: None 2024-03-04T17:53:32.472 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.472 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M+mn/9YtLSiKWnhE+VgpiflNTKY= 2024-03-04T17:53:32.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M+mn/9YtLSiKWnhE+VgpiflNTKY=', 'Content-Length': '0'} 2024-03-04T17:53:32.475 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad6d524792b71e3f-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.475 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.475 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.475 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.476 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.476 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.476 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.476 DEBUG:boto:Data: 2024-03-04T17:53:32.476 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.476 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.476 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.476 DEBUG:boto:Params: {} 2024-03-04T17:53:32.476 DEBUG:boto:Token: None 2024-03-04T17:53:32.476 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.477 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M+mn/9YtLSiKWnhE+VgpiflNTKY= 2024-03-04T17:53:32.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M+mn/9YtLSiKWnhE+VgpiflNTKY=', 'Content-Length': '0'} 2024-03-04T17:53:32.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000710bc62733cef878-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:32.480 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.480 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.480 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.480 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.480 DEBUG:boto:Method: GET 2024-03-04T17:53:32.480 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.480 DEBUG:boto:Data: 2024-03-04T17:53:32.480 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.481 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.481 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.481 DEBUG:boto:Params: {} 2024-03-04T17:53:32.481 DEBUG:boto:Token: None 2024-03-04T17:53:32.481 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.481 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bsIhCEE15qemq9Cd4JRgopYaezM= 2024-03-04T17:53:32.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bsIhCEE15qemq9Cd4JRgopYaezM=', 'Content-Length': '0'} 2024-03-04T17:53:32.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c58944cfd7e76d96-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.528 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.528 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.528 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.528 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.528 DEBUG:boto:Method: GET 2024-03-04T17:53:32.529 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.529 DEBUG:boto:Data: 2024-03-04T17:53:32.529 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.529 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.529 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.529 DEBUG:boto:Params: {} 2024-03-04T17:53:32.529 DEBUG:boto:Token: None 2024-03-04T17:53:32.529 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:32.529 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bsIhCEE15qemq9Cd4JRgopYaezM= 2024-03-04T17:53:32.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bsIhCEE15qemq9Cd4JRgopYaezM=', 'Content-Length': '0'} 2024-03-04T17:53:32.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000623a32a6acbf84a0-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.533 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.533 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.533 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.533 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.533 DEBUG:boto:Method: GET 2024-03-04T17:53:32.533 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.533 DEBUG:boto:Data: 2024-03-04T17:53:32.533 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.533 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.534 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.534 DEBUG:boto:Params: {} 2024-03-04T17:53:32.534 DEBUG:boto:Token: None 2024-03-04T17:53:32.534 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.534 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dc40ca04d2424da-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.580 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.580 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.581 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.581 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.581 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.581 DEBUG:boto:Method: GET 2024-03-04T17:53:32.581 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.581 DEBUG:boto:Data: 2024-03-04T17:53:32.581 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.581 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.581 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.581 DEBUG:boto:Params: {} 2024-03-04T17:53:32.581 DEBUG:boto:Token: None 2024-03-04T17:53:32.582 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.582 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b3bc6b915f8cc1e-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.587 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.587 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.587 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.587 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.587 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:32.587 DEBUG:boto:Data: 2024-03-04T17:53:32.587 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.588 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.588 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.588 DEBUG:boto:Params: {} 2024-03-04T17:53:32.588 DEBUG:boto:Token: None 2024-03-04T17:53:32.588 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj 2024-03-04T17:53:32.588 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI= 2024-03-04T17:53:32.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI=', 'Content-Length': '0'} 2024-03-04T17:53:32.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049d858c88df995cc-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.594 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.595 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.595 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.595 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:32.595 DEBUG:boto:Data: 2024-03-04T17:53:32.595 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.595 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.595 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.595 DEBUG:boto:Params: {} 2024-03-04T17:53:32.595 DEBUG:boto:Token: None 2024-03-04T17:53:32.595 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj 2024-03-04T17:53:32.596 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI= 2024-03-04T17:53:32.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI=', 'Content-Length': '0'} 2024-03-04T17:53:32.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e9e52befe1d61eb-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:32.602 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.602 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.602 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.603 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.603 DEBUG:boto:Method: GET 2024-03-04T17:53:32.603 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.603 DEBUG:boto:Data: 2024-03-04T17:53:32.603 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.603 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.603 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.603 DEBUG:boto:Params: {} 2024-03-04T17:53:32.603 DEBUG:boto:Token: None 2024-03-04T17:53:32.603 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.604 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI= 2024-03-04T17:53:32.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI=', 'Content-Length': '0'} 2024-03-04T17:53:32.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000316a28ac9b80bd58-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.652 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.652 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.652 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.653 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.653 DEBUG:boto:Method: GET 2024-03-04T17:53:32.653 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.653 DEBUG:boto:Data: 2024-03-04T17:53:32.653 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.653 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.653 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.653 DEBUG:boto:Params: {} 2024-03-04T17:53:32.653 DEBUG:boto:Token: None 2024-03-04T17:53:32.653 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.654 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI= 2024-03-04T17:53:32.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI=', 'Content-Length': '0'} 2024-03-04T17:53:32.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098bd1fc16ded9aa5-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.657 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.657 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.658 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.658 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.658 DEBUG:boto:Method: GET 2024-03-04T17:53:32.658 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.658 DEBUG:boto:Data: 2024-03-04T17:53:32.658 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.658 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.658 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.658 DEBUG:boto:Params: {} 2024-03-04T17:53:32.658 DEBUG:boto:Token: None 2024-03-04T17:53:32.659 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.659 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029263704f33423f3-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.704 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.704 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.704 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.705 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.705 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.705 DEBUG:boto:Method: GET 2024-03-04T17:53:32.705 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.705 DEBUG:boto:Data: 2024-03-04T17:53:32.705 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.705 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.705 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.705 DEBUG:boto:Params: {} 2024-03-04T17:53:32.705 DEBUG:boto:Token: None 2024-03-04T17:53:32.706 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.706 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046ca604b05c785e3-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.711 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:32.711 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.711 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.711 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.711 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.711 DEBUG:boto:Method: GET 2024-03-04T17:53:32.712 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.712 DEBUG:boto:Data: 2024-03-04T17:53:32.712 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.712 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.712 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.712 DEBUG:boto:Params: {} 2024-03-04T17:53:32.712 DEBUG:boto:Token: None 2024-03-04T17:53:32.712 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.713 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DCzsRkN+Vp3SQruYR3sZdPGyzkE= 2024-03-04T17:53:32.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DCzsRkN+Vp3SQruYR3sZdPGyzkE=', 'Content-Length': '0'} 2024-03-04T17:53:32.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d64b61c904d78b66-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.716 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.716 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.716 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.717 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.717 DEBUG:boto:Method: GET 2024-03-04T17:53:32.717 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.717 DEBUG:boto:Data: 2024-03-04T17:53:32.717 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.717 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.717 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.717 DEBUG:boto:Params: {} 2024-03-04T17:53:32.717 DEBUG:boto:Token: None 2024-03-04T17:53:32.718 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.718 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DCzsRkN+Vp3SQruYR3sZdPGyzkE= 2024-03-04T17:53:32.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DCzsRkN+Vp3SQruYR3sZdPGyzkE=', 'Content-Length': '0'} 2024-03-04T17:53:32.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6e450f4b36f8e9c-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.721 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.721 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.722 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.722 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.722 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.722 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.722 DEBUG:boto:Data: 2024-03-04T17:53:32.722 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.722 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.722 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.722 DEBUG:boto:Params: {} 2024-03-04T17:53:32.723 DEBUG:boto:Token: None 2024-03-04T17:53:32.723 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.723 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+4remzHxO4PBNKJnvoe19/ZQ6kw= 2024-03-04T17:53:32.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+4remzHxO4PBNKJnvoe19/ZQ6kw=', 'Content-Length': '0'} 2024-03-04T17:53:32.726 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004585937f73182f43-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.726 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.726 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.726 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.726 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.726 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.727 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.727 DEBUG:boto:Data: 2024-03-04T17:53:32.727 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.727 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.727 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.727 DEBUG:boto:Params: {} 2024-03-04T17:53:32.727 DEBUG:boto:Token: None 2024-03-04T17:53:32.727 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.728 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+4remzHxO4PBNKJnvoe19/ZQ6kw= 2024-03-04T17:53:32.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+4remzHxO4PBNKJnvoe19/ZQ6kw=', 'Content-Length': '0'} 2024-03-04T17:53:32.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1a9848679d3d564-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:32.731 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.731 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.731 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.731 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.731 DEBUG:boto:Method: GET 2024-03-04T17:53:32.731 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.732 DEBUG:boto:Data: 2024-03-04T17:53:32.732 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.732 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.732 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.732 DEBUG:boto:Params: {} 2024-03-04T17:53:32.732 DEBUG:boto:Token: None 2024-03-04T17:53:32.732 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.732 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DCzsRkN+Vp3SQruYR3sZdPGyzkE= 2024-03-04T17:53:32.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DCzsRkN+Vp3SQruYR3sZdPGyzkE=', 'Content-Length': '0'} 2024-03-04T17:53:32.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000697d5e5da588d3f9-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.776 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.776 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.776 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.776 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.776 DEBUG:boto:Method: GET 2024-03-04T17:53:32.776 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.777 DEBUG:boto:Data: 2024-03-04T17:53:32.777 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.777 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.777 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.777 DEBUG:boto:Params: {} 2024-03-04T17:53:32.777 DEBUG:boto:Token: None 2024-03-04T17:53:32.777 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:32.777 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DCzsRkN+Vp3SQruYR3sZdPGyzkE= 2024-03-04T17:53:32.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DCzsRkN+Vp3SQruYR3sZdPGyzkE=', 'Content-Length': '0'} 2024-03-04T17:53:32.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030894c3749b123a0-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.781 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.781 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.781 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.781 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.781 DEBUG:boto:Method: GET 2024-03-04T17:53:32.781 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.781 DEBUG:boto:Data: 2024-03-04T17:53:32.781 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.781 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.781 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.782 DEBUG:boto:Params: {} 2024-03-04T17:53:32.782 DEBUG:boto:Token: None 2024-03-04T17:53:32.782 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.782 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000916e3523ef0f291f-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.787 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.787 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.787 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.787 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.787 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.788 DEBUG:boto:Method: GET 2024-03-04T17:53:32.788 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.788 DEBUG:boto:Data: 2024-03-04T17:53:32.788 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.788 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.788 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.788 DEBUG:boto:Params: {} 2024-03-04T17:53:32.788 DEBUG:boto:Token: None 2024-03-04T17:53:32.788 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.789 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000960b2c9b1711603-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.840 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:32.840 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.841 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.841 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.841 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:32.841 DEBUG:boto:Data: 2024-03-04T17:53:32.841 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.841 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.841 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.841 DEBUG:boto:Params: {} 2024-03-04T17:53:32.841 DEBUG:boto:Token: None 2024-03-04T17:53:32.842 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj 2024-03-04T17:53:32.842 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI= 2024-03-04T17:53:32.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI=', 'Content-Length': '0'} 2024-03-04T17:53:32.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b76011f2aaf75287-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.848 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.848 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.848 DEBUG:boto:Method: HEAD 2024-03-04T17:53:32.849 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:32.849 DEBUG:boto:Data: 2024-03-04T17:53:32.849 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.849 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.849 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.849 DEBUG:boto:Params: {} 2024-03-04T17:53:32.849 DEBUG:boto:Token: None 2024-03-04T17:53:32.849 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj 2024-03-04T17:53:32.849 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI= 2024-03-04T17:53:32.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:o0R15phl7Cacvm5xbKQ3fd3UDUI=', 'Content-Length': '0'} 2024-03-04T17:53:32.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7b7162d8b491f68-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:32.856 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.856 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.856 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.856 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.856 DEBUG:boto:Method: GET 2024-03-04T17:53:32.856 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.856 DEBUG:boto:Data: 2024-03-04T17:53:32.857 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.857 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.857 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.857 DEBUG:boto:Params: {} 2024-03-04T17:53:32.857 DEBUG:boto:Token: None 2024-03-04T17:53:32.857 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.857 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI= 2024-03-04T17:53:32.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI=', 'Content-Length': '0'} 2024-03-04T17:53:32.863 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7f139a1bc3baac2-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.908 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:32.908 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:32.908 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.908 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.908 DEBUG:boto:Method: GET 2024-03-04T17:53:32.909 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.909 DEBUG:boto:Data: 2024-03-04T17:53:32.909 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.909 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:32.909 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.909 DEBUG:boto:Params: {} 2024-03-04T17:53:32.909 DEBUG:boto:Token: None 2024-03-04T17:53:32.909 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:32.909 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI= 2024-03-04T17:53:32.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FC5zz6Tf4fHSeFJm5IwFNwX2yZI=', 'Content-Length': '0'} 2024-03-04T17:53:32.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096567f0bb8064338-0065e60a9c-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:32.956 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:32.956 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:32.956 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.956 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:32.956 DEBUG:boto:Method: GET 2024-03-04T17:53:32.957 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:32.957 DEBUG:boto:Data: 2024-03-04T17:53:32.957 DEBUG:boto:Headers: {} 2024-03-04T17:53:32.957 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:32.957 DEBUG:boto:Port: 8000 2024-03-04T17:53:32.957 DEBUG:boto:Params: {} 2024-03-04T17:53:32.957 DEBUG:boto:Token: None 2024-03-04T17:53:32.957 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:32 GMT /ppfahn-22/ 2024-03-04T17:53:32.957 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI= 2024-03-04T17:53:32.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mMfW3dBP3yJgKuUzUxAZgkMJqzI=', 'Content-Length': '0'} 2024-03-04T17:53:32.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d1893fbdd0d749-0065e60a9c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.004 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.004 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.005 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.005 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.005 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.005 DEBUG:boto:Method: GET 2024-03-04T17:53:33.005 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.005 DEBUG:boto:Data: 2024-03-04T17:53:33.005 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.005 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.005 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.006 DEBUG:boto:Params: {} 2024-03-04T17:53:33.006 DEBUG:boto:Token: None 2024-03-04T17:53:33.006 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.006 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006515328f1104cfb-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.052 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:33.052 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.052 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.053 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.053 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.053 DEBUG:boto:Method: GET 2024-03-04T17:53:33.053 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.053 DEBUG:boto:Data: 2024-03-04T17:53:33.053 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.053 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.053 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.053 DEBUG:boto:Params: {} 2024-03-04T17:53:33.053 DEBUG:boto:Token: None 2024-03-04T17:53:33.054 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.054 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:asb4MxZv1+2Tk1lXLgRj58rvT9g= 2024-03-04T17:53:33.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:asb4MxZv1+2Tk1lXLgRj58rvT9g=', 'Content-Length': '0'} 2024-03-04T17:53:33.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3f48d2f88f3da30-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.057 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.057 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.057 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.057 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.057 DEBUG:boto:Method: GET 2024-03-04T17:53:33.057 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.058 DEBUG:boto:Data: 2024-03-04T17:53:33.058 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.058 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.058 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.058 DEBUG:boto:Params: {} 2024-03-04T17:53:33.058 DEBUG:boto:Token: None 2024-03-04T17:53:33.058 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.058 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:asb4MxZv1+2Tk1lXLgRj58rvT9g= 2024-03-04T17:53:33.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:asb4MxZv1+2Tk1lXLgRj58rvT9g=', 'Content-Length': '0'} 2024-03-04T17:53:33.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1c0d3542de46982-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.104 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.104 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.104 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.104 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.105 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.105 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.105 DEBUG:boto:Data: 2024-03-04T17:53:33.105 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.105 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.105 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.105 DEBUG:boto:Params: {} 2024-03-04T17:53:33.105 DEBUG:boto:Token: None 2024-03-04T17:53:33.105 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.105 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cALo+UyPf+tfgdiHzKoVSj5qBUw= 2024-03-04T17:53:33.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cALo+UyPf+tfgdiHzKoVSj5qBUw=', 'Content-Length': '0'} 2024-03-04T17:53:33.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc481e7314715a54-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.108 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.109 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.109 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.109 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.109 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.109 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.109 DEBUG:boto:Data: 2024-03-04T17:53:33.109 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.109 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.109 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.109 DEBUG:boto:Params: {} 2024-03-04T17:53:33.110 DEBUG:boto:Token: None 2024-03-04T17:53:33.110 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.110 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cALo+UyPf+tfgdiHzKoVSj5qBUw= 2024-03-04T17:53:33.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cALo+UyPf+tfgdiHzKoVSj5qBUw=', 'Content-Length': '0'} 2024-03-04T17:53:33.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000155d2f34dc181be2-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:33.113 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.113 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.113 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.114 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.114 DEBUG:boto:Method: GET 2024-03-04T17:53:33.114 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.114 DEBUG:boto:Data: 2024-03-04T17:53:33.114 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.114 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.114 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.114 DEBUG:boto:Params: {} 2024-03-04T17:53:33.114 DEBUG:boto:Token: None 2024-03-04T17:53:33.114 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.114 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:asb4MxZv1+2Tk1lXLgRj58rvT9g= 2024-03-04T17:53:33.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:asb4MxZv1+2Tk1lXLgRj58rvT9g=', 'Content-Length': '0'} 2024-03-04T17:53:33.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c59edf016c0d9621-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.160 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.160 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.160 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.160 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.160 DEBUG:boto:Method: GET 2024-03-04T17:53:33.161 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.161 DEBUG:boto:Data: 2024-03-04T17:53:33.161 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.161 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.161 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.161 DEBUG:boto:Params: {} 2024-03-04T17:53:33.161 DEBUG:boto:Token: None 2024-03-04T17:53:33.161 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:33.161 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:asb4MxZv1+2Tk1lXLgRj58rvT9g= 2024-03-04T17:53:33.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:asb4MxZv1+2Tk1lXLgRj58rvT9g=', 'Content-Length': '0'} 2024-03-04T17:53:33.164 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e4d38722634f745-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.208 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.208 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.208 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.208 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.209 DEBUG:boto:Method: GET 2024-03-04T17:53:33.209 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.209 DEBUG:boto:Data: 2024-03-04T17:53:33.209 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.209 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.209 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.209 DEBUG:boto:Params: {} 2024-03-04T17:53:33.209 DEBUG:boto:Token: None 2024-03-04T17:53:33.209 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.209 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006452da47e5a75e50-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.214 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.215 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.215 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.215 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.215 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.215 DEBUG:boto:Method: GET 2024-03-04T17:53:33.215 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.215 DEBUG:boto:Data: 2024-03-04T17:53:33.215 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.215 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.216 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.216 DEBUG:boto:Params: {} 2024-03-04T17:53:33.216 DEBUG:boto:Token: None 2024-03-04T17:53:33.216 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.216 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003363772bc0bcaeba-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.221 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.221 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.221 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.222 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.222 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:33.222 DEBUG:boto:Data: 2024-03-04T17:53:33.222 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.222 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.222 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.222 DEBUG:boto:Params: {} 2024-03-04T17:53:33.222 DEBUG:boto:Token: None 2024-03-04T17:53:33.222 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj 2024-03-04T17:53:33.222 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ= 2024-03-04T17:53:33.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ=', 'Content-Length': '0'} 2024-03-04T17:53:33.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ea1054c2042c337-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.228 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.229 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.229 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.229 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:33.229 DEBUG:boto:Data: 2024-03-04T17:53:33.229 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.229 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.229 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.229 DEBUG:boto:Params: {} 2024-03-04T17:53:33.230 DEBUG:boto:Token: None 2024-03-04T17:53:33.230 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj 2024-03-04T17:53:33.230 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ= 2024-03-04T17:53:33.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ=', 'Content-Length': '0'} 2024-03-04T17:53:33.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031996cc35902312f-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:33.236 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.236 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.236 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.236 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.237 DEBUG:boto:Method: GET 2024-03-04T17:53:33.237 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.237 DEBUG:boto:Data: 2024-03-04T17:53:33.237 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.237 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.237 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.237 DEBUG:boto:Params: {} 2024-03-04T17:53:33.237 DEBUG:boto:Token: None 2024-03-04T17:53:33.237 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.237 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA= 2024-03-04T17:53:33.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA=', 'Content-Length': '0'} 2024-03-04T17:53:33.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edb8c2cf431c02cf-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.284 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.284 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.284 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.284 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.284 DEBUG:boto:Method: GET 2024-03-04T17:53:33.284 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.285 DEBUG:boto:Data: 2024-03-04T17:53:33.285 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.285 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.285 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.285 DEBUG:boto:Params: {} 2024-03-04T17:53:33.285 DEBUG:boto:Token: None 2024-03-04T17:53:33.285 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.285 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA= 2024-03-04T17:53:33.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA=', 'Content-Length': '0'} 2024-03-04T17:53:33.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e7ff88b4d2fa8f2-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.332 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.332 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.333 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.333 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.333 DEBUG:boto:Method: GET 2024-03-04T17:53:33.333 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.333 DEBUG:boto:Data: 2024-03-04T17:53:33.333 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.333 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.333 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.333 DEBUG:boto:Params: {} 2024-03-04T17:53:33.333 DEBUG:boto:Token: None 2024-03-04T17:53:33.334 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.334 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d6947d78032bcab-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.339 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.339 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.340 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.340 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.340 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.340 DEBUG:boto:Method: GET 2024-03-04T17:53:33.340 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.340 DEBUG:boto:Data: 2024-03-04T17:53:33.340 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.340 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.340 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.341 DEBUG:boto:Params: {} 2024-03-04T17:53:33.341 DEBUG:boto:Token: None 2024-03-04T17:53:33.341 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.341 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099249a09ff6a0d84-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.347 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:33.347 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.348 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.348 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.348 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.348 DEBUG:boto:Method: GET 2024-03-04T17:53:33.348 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.348 DEBUG:boto:Data: 2024-03-04T17:53:33.348 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.348 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.348 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.348 DEBUG:boto:Params: {} 2024-03-04T17:53:33.349 DEBUG:boto:Token: None 2024-03-04T17:53:33.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.349 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+7Mr6rI8nYjeu794ZEndiQUFnQI= 2024-03-04T17:53:33.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+7Mr6rI8nYjeu794ZEndiQUFnQI=', 'Content-Length': '0'} 2024-03-04T17:53:33.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059d7f26bf97ab9f8-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.352 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.352 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.352 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.353 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.353 DEBUG:boto:Method: GET 2024-03-04T17:53:33.353 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.353 DEBUG:boto:Data: 2024-03-04T17:53:33.353 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.353 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.353 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.353 DEBUG:boto:Params: {} 2024-03-04T17:53:33.353 DEBUG:boto:Token: None 2024-03-04T17:53:33.353 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.354 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+7Mr6rI8nYjeu794ZEndiQUFnQI= 2024-03-04T17:53:33.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+7Mr6rI8nYjeu794ZEndiQUFnQI=', 'Content-Length': '0'} 2024-03-04T17:53:33.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007012d1418686bca2-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.400 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.400 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.400 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.400 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.401 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.401 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.401 DEBUG:boto:Data: 2024-03-04T17:53:33.401 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.401 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.401 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.401 DEBUG:boto:Params: {} 2024-03-04T17:53:33.401 DEBUG:boto:Token: None 2024-03-04T17:53:33.401 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.401 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vx9xuRVp3iilz7SKiFLa6JtmqTk= 2024-03-04T17:53:33.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vx9xuRVp3iilz7SKiFLa6JtmqTk=', 'Content-Length': '0'} 2024-03-04T17:53:33.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003522c0c4a4b10019-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.405 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.405 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.405 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.405 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.405 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.405 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.405 DEBUG:boto:Data: 2024-03-04T17:53:33.406 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.406 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.406 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.406 DEBUG:boto:Params: {} 2024-03-04T17:53:33.406 DEBUG:boto:Token: None 2024-03-04T17:53:33.406 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.406 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vx9xuRVp3iilz7SKiFLa6JtmqTk= 2024-03-04T17:53:33.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vx9xuRVp3iilz7SKiFLa6JtmqTk=', 'Content-Length': '0'} 2024-03-04T17:53:33.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e2cf11f786194ad-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:33.409 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.410 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.410 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.410 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.410 DEBUG:boto:Method: GET 2024-03-04T17:53:33.410 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.410 DEBUG:boto:Data: 2024-03-04T17:53:33.410 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.410 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.410 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.410 DEBUG:boto:Params: {} 2024-03-04T17:53:33.410 DEBUG:boto:Token: None 2024-03-04T17:53:33.411 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.411 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+7Mr6rI8nYjeu794ZEndiQUFnQI= 2024-03-04T17:53:33.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+7Mr6rI8nYjeu794ZEndiQUFnQI=', 'Content-Length': '0'} 2024-03-04T17:53:33.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005748b278ee58e21a-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.414 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.414 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.414 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.415 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.415 DEBUG:boto:Method: GET 2024-03-04T17:53:33.415 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.415 DEBUG:boto:Data: 2024-03-04T17:53:33.415 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.415 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.415 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.415 DEBUG:boto:Params: {} 2024-03-04T17:53:33.415 DEBUG:boto:Token: None 2024-03-04T17:53:33.416 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:33.416 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+7Mr6rI8nYjeu794ZEndiQUFnQI= 2024-03-04T17:53:33.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+7Mr6rI8nYjeu794ZEndiQUFnQI=', 'Content-Length': '0'} 2024-03-04T17:53:33.419 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000732462be670dc4ea-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.460 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.460 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.460 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.460 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.461 DEBUG:boto:Method: GET 2024-03-04T17:53:33.461 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.461 DEBUG:boto:Data: 2024-03-04T17:53:33.461 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.461 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.461 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.461 DEBUG:boto:Params: {} 2024-03-04T17:53:33.461 DEBUG:boto:Token: None 2024-03-04T17:53:33.461 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.461 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cebaca1a576d5fb3-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.466 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.467 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.467 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.467 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.467 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.467 DEBUG:boto:Method: GET 2024-03-04T17:53:33.467 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.467 DEBUG:boto:Data: 2024-03-04T17:53:33.467 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.468 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.468 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.468 DEBUG:boto:Params: {} 2024-03-04T17:53:33.468 DEBUG:boto:Token: None 2024-03-04T17:53:33.468 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.468 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ffccbf0bb6fb989-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.474 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.474 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.474 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.474 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.474 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:33.475 DEBUG:boto:Data: 2024-03-04T17:53:33.475 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.475 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.475 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.475 DEBUG:boto:Params: {} 2024-03-04T17:53:33.475 DEBUG:boto:Token: None 2024-03-04T17:53:33.475 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj 2024-03-04T17:53:33.475 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ= 2024-03-04T17:53:33.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ=', 'Content-Length': '0'} 2024-03-04T17:53:33.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058ae15cdb77c191b-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.481 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.481 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.481 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.481 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:33.482 DEBUG:boto:Data: 2024-03-04T17:53:33.482 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.482 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.482 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.482 DEBUG:boto:Params: {} 2024-03-04T17:53:33.482 DEBUG:boto:Token: None 2024-03-04T17:53:33.482 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj 2024-03-04T17:53:33.482 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ= 2024-03-04T17:53:33.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ=', 'Content-Length': '0'} 2024-03-04T17:53:33.488 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce2608320d335865-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:33.488 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.488 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.488 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.489 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.489 DEBUG:boto:Method: GET 2024-03-04T17:53:33.489 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.489 DEBUG:boto:Data: 2024-03-04T17:53:33.489 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.489 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.489 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.489 DEBUG:boto:Params: {} 2024-03-04T17:53:33.489 DEBUG:boto:Token: None 2024-03-04T17:53:33.489 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.490 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA= 2024-03-04T17:53:33.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA=', 'Content-Length': '0'} 2024-03-04T17:53:33.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000719bc5f78d569cd1-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.536 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.536 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.536 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.536 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.536 DEBUG:boto:Method: GET 2024-03-04T17:53:33.537 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.537 DEBUG:boto:Data: 2024-03-04T17:53:33.537 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.537 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.537 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.537 DEBUG:boto:Params: {} 2024-03-04T17:53:33.537 DEBUG:boto:Token: None 2024-03-04T17:53:33.537 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.537 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA= 2024-03-04T17:53:33.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA=', 'Content-Length': '0'} 2024-03-04T17:53:33.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091562ebf1cd7a0d4-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.584 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.584 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.584 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.584 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.584 DEBUG:boto:Method: GET 2024-03-04T17:53:33.584 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.585 DEBUG:boto:Data: 2024-03-04T17:53:33.585 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.585 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.585 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.585 DEBUG:boto:Params: {} 2024-03-04T17:53:33.585 DEBUG:boto:Token: None 2024-03-04T17:53:33.585 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.585 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068f56d2dedd1cd33-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.590 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.591 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.591 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.591 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.591 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.591 DEBUG:boto:Method: GET 2024-03-04T17:53:33.591 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.591 DEBUG:boto:Data: 2024-03-04T17:53:33.591 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.592 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.592 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.592 DEBUG:boto:Params: {} 2024-03-04T17:53:33.592 DEBUG:boto:Token: None 2024-03-04T17:53:33.592 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.592 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abcfd8f0861a8471-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.597 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:33.597 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.598 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.598 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.598 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.598 DEBUG:boto:Method: GET 2024-03-04T17:53:33.598 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.598 DEBUG:boto:Data: 2024-03-04T17:53:33.598 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.598 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.598 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.598 DEBUG:boto:Params: {} 2024-03-04T17:53:33.599 DEBUG:boto:Token: None 2024-03-04T17:53:33.599 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.599 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:54BcKlfOiLMP1Yyokd5via8fC24= 2024-03-04T17:53:33.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:54BcKlfOiLMP1Yyokd5via8fC24=', 'Content-Length': '0'} 2024-03-04T17:53:33.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035bcf2173880bbb6-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.602 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.603 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.603 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.603 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.603 DEBUG:boto:Method: GET 2024-03-04T17:53:33.603 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.603 DEBUG:boto:Data: 2024-03-04T17:53:33.603 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.603 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.603 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.603 DEBUG:boto:Params: {} 2024-03-04T17:53:33.604 DEBUG:boto:Token: None 2024-03-04T17:53:33.604 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.604 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:54BcKlfOiLMP1Yyokd5via8fC24= 2024-03-04T17:53:33.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:54BcKlfOiLMP1Yyokd5via8fC24=', 'Content-Length': '0'} 2024-03-04T17:53:33.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025f5364e73ac886c-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.648 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.648 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.648 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.648 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.648 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.648 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.649 DEBUG:boto:Data: 2024-03-04T17:53:33.649 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.649 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.649 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.649 DEBUG:boto:Params: {} 2024-03-04T17:53:33.649 DEBUG:boto:Token: None 2024-03-04T17:53:33.649 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.649 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZYShfxYjf7wujgho5qgj/9RT0BU= 2024-03-04T17:53:33.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZYShfxYjf7wujgho5qgj/9RT0BU=', 'Content-Length': '0'} 2024-03-04T17:53:33.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007be0454592489f8b-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.652 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.653 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.653 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.653 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.653 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.653 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.653 DEBUG:boto:Data: 2024-03-04T17:53:33.653 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.653 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.653 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.653 DEBUG:boto:Params: {} 2024-03-04T17:53:33.654 DEBUG:boto:Token: None 2024-03-04T17:53:33.654 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.654 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZYShfxYjf7wujgho5qgj/9RT0BU= 2024-03-04T17:53:33.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZYShfxYjf7wujgho5qgj/9RT0BU=', 'Content-Length': '0'} 2024-03-04T17:53:33.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044f31c1e55aac7fd-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:33.657 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.657 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.657 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.657 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.658 DEBUG:boto:Method: GET 2024-03-04T17:53:33.658 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.658 DEBUG:boto:Data: 2024-03-04T17:53:33.658 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.658 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.658 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.658 DEBUG:boto:Params: {} 2024-03-04T17:53:33.658 DEBUG:boto:Token: None 2024-03-04T17:53:33.658 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.659 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:54BcKlfOiLMP1Yyokd5via8fC24= 2024-03-04T17:53:33.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:54BcKlfOiLMP1Yyokd5via8fC24=', 'Content-Length': '0'} 2024-03-04T17:53:33.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057b608c1011aab5a-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.662 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.662 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.662 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.662 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.662 DEBUG:boto:Method: GET 2024-03-04T17:53:33.662 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.662 DEBUG:boto:Data: 2024-03-04T17:53:33.662 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.662 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.662 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.662 DEBUG:boto:Params: {} 2024-03-04T17:53:33.663 DEBUG:boto:Token: None 2024-03-04T17:53:33.663 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:33.663 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:54BcKlfOiLMP1Yyokd5via8fC24= 2024-03-04T17:53:33.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:54BcKlfOiLMP1Yyokd5via8fC24=', 'Content-Length': '0'} 2024-03-04T17:53:33.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092e1efd11309a79e-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.708 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.708 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.708 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.708 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.708 DEBUG:boto:Method: GET 2024-03-04T17:53:33.708 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.709 DEBUG:boto:Data: 2024-03-04T17:53:33.709 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.709 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.709 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.709 DEBUG:boto:Params: {} 2024-03-04T17:53:33.709 DEBUG:boto:Token: None 2024-03-04T17:53:33.709 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.709 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c2eacf1e7ff28b1-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.714 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.715 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.715 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.715 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.715 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.715 DEBUG:boto:Method: GET 2024-03-04T17:53:33.715 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.715 DEBUG:boto:Data: 2024-03-04T17:53:33.715 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.716 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.716 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.716 DEBUG:boto:Params: {} 2024-03-04T17:53:33.716 DEBUG:boto:Token: None 2024-03-04T17:53:33.716 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.716 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ce4d59a53762a39-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.721 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.722 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.722 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.722 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.722 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:33.722 DEBUG:boto:Data: 2024-03-04T17:53:33.722 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.722 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.722 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.722 DEBUG:boto:Params: {} 2024-03-04T17:53:33.722 DEBUG:boto:Token: None 2024-03-04T17:53:33.723 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj 2024-03-04T17:53:33.723 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ= 2024-03-04T17:53:33.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ=', 'Content-Length': '0'} 2024-03-04T17:53:33.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000305b94ccd7b844-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.729 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.729 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.729 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.729 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:33.729 DEBUG:boto:Data: 2024-03-04T17:53:33.729 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.729 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.729 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.729 DEBUG:boto:Params: {} 2024-03-04T17:53:33.730 DEBUG:boto:Token: None 2024-03-04T17:53:33.730 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj 2024-03-04T17:53:33.730 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ= 2024-03-04T17:53:33.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ=', 'Content-Length': '0'} 2024-03-04T17:53:33.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000721d763410028453-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:33.736 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.736 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.736 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.736 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.737 DEBUG:boto:Method: GET 2024-03-04T17:53:33.737 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.737 DEBUG:boto:Data: 2024-03-04T17:53:33.737 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.737 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.737 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.737 DEBUG:boto:Params: {} 2024-03-04T17:53:33.737 DEBUG:boto:Token: None 2024-03-04T17:53:33.737 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.738 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA= 2024-03-04T17:53:33.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA=', 'Content-Length': '0'} 2024-03-04T17:53:33.741 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a69374ab9b563420-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.784 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.784 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.784 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.784 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.784 DEBUG:boto:Method: GET 2024-03-04T17:53:33.785 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.785 DEBUG:boto:Data: 2024-03-04T17:53:33.785 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.785 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.785 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.785 DEBUG:boto:Params: {} 2024-03-04T17:53:33.785 DEBUG:boto:Token: None 2024-03-04T17:53:33.785 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.785 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA= 2024-03-04T17:53:33.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA=', 'Content-Length': '0'} 2024-03-04T17:53:33.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecbddb9116406fe2-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.832 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.832 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.832 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.832 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.833 DEBUG:boto:Method: GET 2024-03-04T17:53:33.833 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.833 DEBUG:boto:Data: 2024-03-04T17:53:33.833 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.833 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.833 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.833 DEBUG:boto:Params: {} 2024-03-04T17:53:33.833 DEBUG:boto:Token: None 2024-03-04T17:53:33.833 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.834 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ee83420c1ce6314-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.838 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.839 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.839 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.839 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.839 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.839 DEBUG:boto:Method: GET 2024-03-04T17:53:33.839 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.839 DEBUG:boto:Data: 2024-03-04T17:53:33.839 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.840 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.840 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.840 DEBUG:boto:Params: {} 2024-03-04T17:53:33.840 DEBUG:boto:Token: None 2024-03-04T17:53:33.840 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.840 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efee74a5e0a2723a-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.845 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:33.846 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.846 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.846 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.846 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.846 DEBUG:boto:Method: GET 2024-03-04T17:53:33.846 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.846 DEBUG:boto:Data: 2024-03-04T17:53:33.846 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.846 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.846 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.847 DEBUG:boto:Params: {} 2024-03-04T17:53:33.847 DEBUG:boto:Token: None 2024-03-04T17:53:33.847 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.847 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MmtcWul+rF0e2uKSU/pSSSq/QyE= 2024-03-04T17:53:33.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MmtcWul+rF0e2uKSU/pSSSq/QyE=', 'Content-Length': '0'} 2024-03-04T17:53:33.850 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000478f28e971529ffe-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.850 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.851 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.851 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.851 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.851 DEBUG:boto:Method: GET 2024-03-04T17:53:33.851 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.851 DEBUG:boto:Data: 2024-03-04T17:53:33.851 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.851 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.851 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.851 DEBUG:boto:Params: {} 2024-03-04T17:53:33.851 DEBUG:boto:Token: None 2024-03-04T17:53:33.852 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.852 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MmtcWul+rF0e2uKSU/pSSSq/QyE= 2024-03-04T17:53:33.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MmtcWul+rF0e2uKSU/pSSSq/QyE=', 'Content-Length': '0'} 2024-03-04T17:53:33.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bebaf5594d9a8933-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.855 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.855 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.855 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.855 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.855 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.855 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.856 DEBUG:boto:Data: 2024-03-04T17:53:33.856 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.856 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.856 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.856 DEBUG:boto:Params: {} 2024-03-04T17:53:33.856 DEBUG:boto:Token: None 2024-03-04T17:53:33.856 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.856 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CAQPwRHU960UkwQb4FfivGhokDI= 2024-03-04T17:53:33.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CAQPwRHU960UkwQb4FfivGhokDI=', 'Content-Length': '0'} 2024-03-04T17:53:33.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfb69df9dda36a1a-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.860 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.860 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.860 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.860 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.860 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.860 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.860 DEBUG:boto:Data: 2024-03-04T17:53:33.861 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.861 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.861 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.861 DEBUG:boto:Params: {} 2024-03-04T17:53:33.861 DEBUG:boto:Token: None 2024-03-04T17:53:33.861 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.861 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CAQPwRHU960UkwQb4FfivGhokDI= 2024-03-04T17:53:33.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CAQPwRHU960UkwQb4FfivGhokDI=', 'Content-Length': '0'} 2024-03-04T17:53:33.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a9cc93b2c4d63b6-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:33.865 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.865 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.865 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.865 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.865 DEBUG:boto:Method: GET 2024-03-04T17:53:33.865 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.865 DEBUG:boto:Data: 2024-03-04T17:53:33.865 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.865 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.866 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.866 DEBUG:boto:Params: {} 2024-03-04T17:53:33.866 DEBUG:boto:Token: None 2024-03-04T17:53:33.866 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.866 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MmtcWul+rF0e2uKSU/pSSSq/QyE= 2024-03-04T17:53:33.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MmtcWul+rF0e2uKSU/pSSSq/QyE=', 'Content-Length': '0'} 2024-03-04T17:53:33.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cc25b0f04c8c78c-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.912 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.912 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.912 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.912 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.912 DEBUG:boto:Method: GET 2024-03-04T17:53:33.913 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.913 DEBUG:boto:Data: 2024-03-04T17:53:33.913 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.913 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.913 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.913 DEBUG:boto:Params: {} 2024-03-04T17:53:33.913 DEBUG:boto:Token: None 2024-03-04T17:53:33.913 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:33.913 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MmtcWul+rF0e2uKSU/pSSSq/QyE= 2024-03-04T17:53:33.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MmtcWul+rF0e2uKSU/pSSSq/QyE=', 'Content-Length': '0'} 2024-03-04T17:53:33.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077e3fe56d30fcd5e-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.917 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.917 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.917 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.917 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.917 DEBUG:boto:Method: GET 2024-03-04T17:53:33.917 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.917 DEBUG:boto:Data: 2024-03-04T17:53:33.917 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.917 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.918 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.918 DEBUG:boto:Params: {} 2024-03-04T17:53:33.918 DEBUG:boto:Token: None 2024-03-04T17:53:33.918 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.918 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee68b5187ef3a943-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.964 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.964 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:33.964 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:33.965 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.965 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:33.965 DEBUG:boto:Method: GET 2024-03-04T17:53:33.965 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:33.965 DEBUG:boto:Data: 2024-03-04T17:53:33.965 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.965 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.965 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.965 DEBUG:boto:Params: {} 2024-03-04T17:53:33.965 DEBUG:boto:Token: None 2024-03-04T17:53:33.966 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/ 2024-03-04T17:53:33.966 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw= 2024-03-04T17:53:33.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fpWdCooZK/Ia/gwmOLvieWAGuJw=', 'Content-Length': '0'} 2024-03-04T17:53:33.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014947952ed00e1c0-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.971 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:33.972 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.972 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.972 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.972 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:33.972 DEBUG:boto:Data: 2024-03-04T17:53:33.972 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.972 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.972 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.972 DEBUG:boto:Params: {} 2024-03-04T17:53:33.972 DEBUG:boto:Token: None 2024-03-04T17:53:33.973 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj 2024-03-04T17:53:33.973 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ= 2024-03-04T17:53:33.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ=', 'Content-Length': '0'} 2024-03-04T17:53:33.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de61dae45b4fa825-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.979 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.979 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.979 DEBUG:boto:Method: HEAD 2024-03-04T17:53:33.979 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:33.979 DEBUG:boto:Data: 2024-03-04T17:53:33.979 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.979 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:33.979 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.979 DEBUG:boto:Params: {} 2024-03-04T17:53:33.979 DEBUG:boto:Token: None 2024-03-04T17:53:33.979 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj 2024-03-04T17:53:33.980 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ= 2024-03-04T17:53:33.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2mKa+P9m/++lO0d/wH+HzyrbLGQ=', 'Content-Length': '0'} 2024-03-04T17:53:33.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f238413d4da91ede-0065e60a9d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:33.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:33.986 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:33.986 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:33.986 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.986 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.986 DEBUG:boto:Method: GET 2024-03-04T17:53:33.986 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.986 DEBUG:boto:Data: 2024-03-04T17:53:33.986 DEBUG:boto:Headers: {} 2024-03-04T17:53:33.987 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:33.987 DEBUG:boto:Port: 8000 2024-03-04T17:53:33.987 DEBUG:boto:Params: {} 2024-03-04T17:53:33.987 DEBUG:boto:Token: None 2024-03-04T17:53:33.987 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:33 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:33.987 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA= 2024-03-04T17:53:33.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:33 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1AUV1ODIr+5K1gwSUDwLxJc1NdA=', 'Content-Length': '0'} 2024-03-04T17:53:33.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011d2314d35fcfb2f-0065e60a9d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.032 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.032 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.032 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.032 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.032 DEBUG:boto:Method: GET 2024-03-04T17:53:34.033 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.033 DEBUG:boto:Data: 2024-03-04T17:53:34.033 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.033 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.033 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.033 DEBUG:boto:Params: {} 2024-03-04T17:53:34.033 DEBUG:boto:Token: None 2024-03-04T17:53:34.033 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.033 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g= 2024-03-04T17:53:34.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g=', 'Content-Length': '0'} 2024-03-04T17:53:34.036 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d22e0ef3d2fa969c-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.080 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.080 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.080 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.080 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.081 DEBUG:boto:Method: GET 2024-03-04T17:53:34.081 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:34.081 DEBUG:boto:Data: 2024-03-04T17:53:34.081 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.081 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:34.081 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.081 DEBUG:boto:Params: {} 2024-03-04T17:53:34.081 DEBUG:boto:Token: None 2024-03-04T17:53:34.081 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.081 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011af1118b25ace5e-0065e60a9e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.132 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.133 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.133 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.133 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.133 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.133 DEBUG:boto:Method: GET 2024-03-04T17:53:34.133 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:34.133 DEBUG:boto:Data: 2024-03-04T17:53:34.133 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.133 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.133 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.133 DEBUG:boto:Params: {} 2024-03-04T17:53:34.134 DEBUG:boto:Token: None 2024-03-04T17:53:34.134 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.134 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007df0d3d8a4071127-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.139 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:34.139 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.139 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.140 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.140 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.140 DEBUG:boto:Method: GET 2024-03-04T17:53:34.140 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.140 DEBUG:boto:Data: 2024-03-04T17:53:34.140 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.140 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:34.140 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.140 DEBUG:boto:Params: {} 2024-03-04T17:53:34.140 DEBUG:boto:Token: None 2024-03-04T17:53:34.141 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.141 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F96AKSQfDSsW/wPl7rJlBb8MnUo= 2024-03-04T17:53:34.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F96AKSQfDSsW/wPl7rJlBb8MnUo=', 'Content-Length': '0'} 2024-03-04T17:53:34.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1246cea58463e2d-0065e60a9e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.145 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.145 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.145 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.145 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.145 DEBUG:boto:Method: GET 2024-03-04T17:53:34.145 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.145 DEBUG:boto:Data: 2024-03-04T17:53:34.146 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.146 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.146 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.146 DEBUG:boto:Params: {} 2024-03-04T17:53:34.146 DEBUG:boto:Token: None 2024-03-04T17:53:34.146 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.146 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F96AKSQfDSsW/wPl7rJlBb8MnUo= 2024-03-04T17:53:34.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F96AKSQfDSsW/wPl7rJlBb8MnUo=', 'Content-Length': '0'} 2024-03-04T17:53:34.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6c871d6e834b5d3-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.192 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.192 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.192 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.192 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.193 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.193 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.193 DEBUG:boto:Data: 2024-03-04T17:53:34.193 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.193 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:34.193 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.193 DEBUG:boto:Params: {} 2024-03-04T17:53:34.193 DEBUG:boto:Token: None 2024-03-04T17:53:34.193 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.194 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EH29eGnQ1UmyTB+Hns8ckqV+5BQ= 2024-03-04T17:53:34.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EH29eGnQ1UmyTB+Hns8ckqV+5BQ=', 'Content-Length': '0'} 2024-03-04T17:53:34.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb1940ed818a20d9-0065e60a9e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.197 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.197 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.197 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.197 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.197 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.198 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.198 DEBUG:boto:Data: 2024-03-04T17:53:34.198 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.198 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.198 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.198 DEBUG:boto:Params: {} 2024-03-04T17:53:34.198 DEBUG:boto:Token: None 2024-03-04T17:53:34.198 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.198 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EH29eGnQ1UmyTB+Hns8ckqV+5BQ= 2024-03-04T17:53:34.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EH29eGnQ1UmyTB+Hns8ckqV+5BQ=', 'Content-Length': '0'} 2024-03-04T17:53:34.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e803823adcab5487-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:34.202 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.202 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.202 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.202 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.202 DEBUG:boto:Method: GET 2024-03-04T17:53:34.202 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.202 DEBUG:boto:Data: 2024-03-04T17:53:34.202 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.203 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:34.203 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.203 DEBUG:boto:Params: {} 2024-03-04T17:53:34.203 DEBUG:boto:Token: None 2024-03-04T17:53:34.203 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.203 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F96AKSQfDSsW/wPl7rJlBb8MnUo= 2024-03-04T17:53:34.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F96AKSQfDSsW/wPl7rJlBb8MnUo=', 'Content-Length': '0'} 2024-03-04T17:53:34.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cf91820975a9e61-0065e60a9e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.248 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.248 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.248 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.248 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.249 DEBUG:boto:Method: GET 2024-03-04T17:53:34.249 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.249 DEBUG:boto:Data: 2024-03-04T17:53:34.249 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.249 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.249 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.249 DEBUG:boto:Params: {} 2024-03-04T17:53:34.249 DEBUG:boto:Token: None 2024-03-04T17:53:34.249 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:34.249 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F96AKSQfDSsW/wPl7rJlBb8MnUo= 2024-03-04T17:53:34.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F96AKSQfDSsW/wPl7rJlBb8MnUo=', 'Content-Length': '0'} 2024-03-04T17:53:34.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044b1f0cde9aa10f5-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.296 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.296 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.296 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.297 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.297 DEBUG:boto:Method: GET 2024-03-04T17:53:34.297 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:34.297 DEBUG:boto:Data: 2024-03-04T17:53:34.297 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.297 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:34.297 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.297 DEBUG:boto:Params: {} 2024-03-04T17:53:34.297 DEBUG:boto:Token: None 2024-03-04T17:53:34.297 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.298 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004deb10d9cf93c389-0065e60a9e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.346 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.347 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.347 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.347 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.347 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.348 DEBUG:boto:Method: GET 2024-03-04T17:53:34.348 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:34.348 DEBUG:boto:Data: 2024-03-04T17:53:34.348 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.348 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.348 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.349 DEBUG:boto:Params: {} 2024-03-04T17:53:34.349 DEBUG:boto:Token: None 2024-03-04T17:53:34.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.349 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093acece92d3a195e-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.355 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.355 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.356 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.356 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.356 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:34.356 DEBUG:boto:Data: 2024-03-04T17:53:34.356 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.356 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:34.356 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.356 DEBUG:boto:Params: {} 2024-03-04T17:53:34.356 DEBUG:boto:Token: None 2024-03-04T17:53:34.356 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj 2024-03-04T17:53:34.357 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:By6Zcj0NTSx4Q742VNkwI/Sw4MA= 2024-03-04T17:53:34.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:By6Zcj0NTSx4Q742VNkwI/Sw4MA=', 'Content-Length': '0'} 2024-03-04T17:53:34.363 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d5b14c8f7f8330b-0065e60a9e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.363 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.363 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.364 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.364 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:34.364 DEBUG:boto:Data: 2024-03-04T17:53:34.364 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.364 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.364 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.364 DEBUG:boto:Params: {} 2024-03-04T17:53:34.364 DEBUG:boto:Token: None 2024-03-04T17:53:34.365 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj 2024-03-04T17:53:34.365 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:By6Zcj0NTSx4Q742VNkwI/Sw4MA= 2024-03-04T17:53:34.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:By6Zcj0NTSx4Q742VNkwI/Sw4MA=', 'Content-Length': '0'} 2024-03-04T17:53:34.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052e44e734a252382-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:34.372 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.372 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.372 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.372 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.372 DEBUG:boto:Method: GET 2024-03-04T17:53:34.372 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.372 DEBUG:boto:Data: 2024-03-04T17:53:34.372 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.372 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:34.372 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.372 DEBUG:boto:Params: {} 2024-03-04T17:53:34.373 DEBUG:boto:Token: None 2024-03-04T17:53:34.373 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.373 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g= 2024-03-04T17:53:34.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g=', 'Content-Length': '0'} 2024-03-04T17:53:34.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056b130463aed6766-0065e60a9e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.420 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.420 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.421 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.421 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.421 DEBUG:boto:Method: GET 2024-03-04T17:53:34.421 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.421 DEBUG:boto:Data: 2024-03-04T17:53:34.421 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.422 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.422 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.422 DEBUG:boto:Params: {} 2024-03-04T17:53:34.422 DEBUG:boto:Token: None 2024-03-04T17:53:34.422 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.423 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g= 2024-03-04T17:53:34.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g=', 'Content-Length': '0'} 2024-03-04T17:53:34.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de453334a93860cd-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.468 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.468 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.468 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.468 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.468 DEBUG:boto:Method: GET 2024-03-04T17:53:34.469 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:34.469 DEBUG:boto:Data: 2024-03-04T17:53:34.469 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.469 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:34.469 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.469 DEBUG:boto:Params: {} 2024-03-04T17:53:34.469 DEBUG:boto:Token: None 2024-03-04T17:53:34.469 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.470 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de1891a6c5a60393-0065e60a9e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.516 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.517 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.517 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.517 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.517 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.517 DEBUG:boto:Method: GET 2024-03-04T17:53:34.517 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:34.517 DEBUG:boto:Data: 2024-03-04T17:53:34.517 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.517 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.518 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.518 DEBUG:boto:Params: {} 2024-03-04T17:53:34.518 DEBUG:boto:Token: None 2024-03-04T17:53:34.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.518 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c28945f9fa80804c-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.524 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.524 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-22 zones={test-zone2, test-zone1} 2024-03-04T17:53:34.524 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-22 zones={test-zone3, test-zone1} 2024-03-04T17:53:34.524 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.525 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.525 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.525 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:53:34.525 DEBUG:boto:Data: 2024-03-04T17:53:34.525 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.525 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.525 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.525 DEBUG:boto:Params: {} 2024-03-04T17:53:34.526 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T17:53:34.526 DEBUG:boto:Token: None 2024-03-04T17:53:34.526 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.526 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Z4sg6JjO1yLgbHL94Y4jh5jnZ0U= 2024-03-04T17:53:34.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Z4sg6JjO1yLgbHL94Y4jh5jnZ0U=', 'Content-Length': '0'} 2024-03-04T17:53:34.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009eb6d827f17961e7-0065e60a9e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.532 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.532 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.532 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.532 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:53:34.533 DEBUG:boto:Data: 2024-03-04T17:53:34.533 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.533 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.533 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.533 DEBUG:boto:Params: {} 2024-03-04T17:53:34.533 DEBUG:boto:Token: None 2024-03-04T17:53:34.533 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.533 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Z4sg6JjO1yLgbHL94Y4jh5jnZ0U= 2024-03-04T17:53:34.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Z4sg6JjO1yLgbHL94Y4jh5jnZ0U=', 'Content-Length': '0'} 2024-03-04T17:53:34.537 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c4042286c70349c-0065e60a9e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.537 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:53:34.537 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.538 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.538 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:34.538 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:34.538 DEBUG:boto:Method: GET 2024-03-04T17:53:34.538 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:34.538 DEBUG:boto:Data: 2024-03-04T17:53:34.538 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.538 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.539 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.539 DEBUG:boto:Params: {} 2024-03-04T17:53:34.539 DEBUG:boto:Token: None 2024-03-04T17:53:34.539 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/?versions 2024-03-04T17:53:34.539 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MZMUw4/Hxy2oZPAXvlXKswphnKg= 2024-03-04T17:53:34.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MZMUw4/Hxy2oZPAXvlXKswphnKg=', 'Content-Length': '0'} 2024-03-04T17:53:34.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b44ec1d86c0acd4-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.545 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG6tTJPArSLLFYQGx5Z9J94Qp3btVq8wtrue2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQDkS502cdQ0sD3MSSr4AJ.nNDwrboFEfalse2024-03-04T17:51:31.005Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfexW91D920ZLyPSn0KVa8L-0Jf11s3qfalse2024-03-04T17:51:31.057Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4aDzoElev9M4kJpBvZIFgGfNSrnTu1hfalse2024-03-04T17:51:30.946Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrBR7sHkOppK-w8aON9rPlfIadTwy8LKfalse2024-03-04T17:51:30.974Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtp1YBJlmM1cmYYj2xRDAuLHN-EIcD23false2024-03-04T17:51:30.916Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxsfalse2024-03-04T17:51:30.859Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIgnYdWgPuuz09tkj9qWq7P8qzgBSCIyfalse2024-03-04T17:51:30.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZw7l4YYB37p1xa7jtLR7kDu1d5Xjl6Gfalse2024-03-04T17:51:30.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10Vjj0fS8rYkr3OIS1pBSR-3fCH6svyfalse2024-03-04T17:51:30.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXak5SqkkJ7eVzuGI.8.u99lZqCC3tNwfalse2024-03-04T17:51:30.749Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzyjvmlB6z-cVoQlri1b6gVvpnPVUCjFfalse2024-03-04T17:51:30.721Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6BBhxVS-A79T2MI6AHE.THWDO21iftffalse2024-03-04T17:51:30.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7kfalse2024-03-04T17:51:30.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji.eNnYbM.4yY91CfDbCxKIqa-Uec-fxfalse2024-03-04T17:51:30.610Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkUfalse2024-03-04T17:51:30.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje9K3WlwCsFUmkYneucDfMv9rNQO6cdpfalse2024-03-04T17:51:30.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRTfalse2024-03-04T17:51:30.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgRxtOPcYyW7cm-IBYtAu0zyYNScRyoTfalse2024-03-04T17:51:30.416Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjme6.y9cJ8bkVcZ04PNMrwIXjDQUGzXRfalse2024-03-04T17:51:30.386Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoHs8pGuS1bc-GYUbjIaqPQPwQGDukkcfalse2024-03-04T17:51:30.469Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj160IY3UeXb5qH4Rh4JB09PNigxsTa5gfalse2024-03-04T17:51:30.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9WE9mNbplYdU449Vkwh8lwrICYxU6v7false2024-03-04T17:51:30.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZJZPq8HwR1LmdjGaT37eTcVO7gvndKgfalse2024-03-04T17:51:30.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lwBSkQlMAA54TFAsh7J7MCnzo0gTfafalse2024-03-04T17:51:30.157Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq4h-uWPuCpU7FK9SfN-LHLv-vtuKwuEfalse2024-03-04T17:51:30.239Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzKZVhMV59rWXQPf05XlhVIimSaZTr40false2024-03-04T17:51:30.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAcrzzw6Ded5-7UcCqpx-w2RGP5BSbOUfalse2024-03-04T17:51:30.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdVzfZSb65W93TGSPX8waGA5tm4N9lW6false2024-03-04T17:51:30.037Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlwBZf1YBiU2Wqy250Elw-2.f6FuQF0efalse2024-03-04T17:51:30.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.547 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:53:34.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.550 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:53:34.550 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.550 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.550 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:34.551 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:34.551 DEBUG:boto:Method: GET 2024-03-04T17:53:34.551 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:34.551 DEBUG:boto:Data: 2024-03-04T17:53:34.551 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.551 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.551 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.551 DEBUG:boto:Params: {} 2024-03-04T17:53:34.551 DEBUG:boto:Token: None 2024-03-04T17:53:34.551 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/?versions 2024-03-04T17:53:34.552 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MZMUw4/Hxy2oZPAXvlXKswphnKg= 2024-03-04T17:53:34.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MZMUw4/Hxy2oZPAXvlXKswphnKg=', 'Content-Length': '0'} 2024-03-04T17:53:34.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de1746a99be6df70-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.557 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG6tTJPArSLLFYQGx5Z9J94Qp3btVq8wtrue2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQDkS502cdQ0sD3MSSr4AJ.nNDwrboFEfalse2024-03-04T17:51:31.005Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfexW91D920ZLyPSn0KVa8L-0Jf11s3qfalse2024-03-04T17:51:31.057Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4aDzoElev9M4kJpBvZIFgGfNSrnTu1hfalse2024-03-04T17:51:30.946Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrBR7sHkOppK-w8aON9rPlfIadTwy8LKfalse2024-03-04T17:51:30.974Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtp1YBJlmM1cmYYj2xRDAuLHN-EIcD23false2024-03-04T17:51:30.916Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxsfalse2024-03-04T17:51:30.859Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIgnYdWgPuuz09tkj9qWq7P8qzgBSCIyfalse2024-03-04T17:51:30.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZw7l4YYB37p1xa7jtLR7kDu1d5Xjl6Gfalse2024-03-04T17:51:30.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10Vjj0fS8rYkr3OIS1pBSR-3fCH6svyfalse2024-03-04T17:51:30.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXak5SqkkJ7eVzuGI.8.u99lZqCC3tNwfalse2024-03-04T17:51:30.749Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzyjvmlB6z-cVoQlri1b6gVvpnPVUCjFfalse2024-03-04T17:51:30.721Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6BBhxVS-A79T2MI6AHE.THWDO21iftffalse2024-03-04T17:51:30.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7kfalse2024-03-04T17:51:30.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji.eNnYbM.4yY91CfDbCxKIqa-Uec-fxfalse2024-03-04T17:51:30.610Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkUfalse2024-03-04T17:51:30.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje9K3WlwCsFUmkYneucDfMv9rNQO6cdpfalse2024-03-04T17:51:30.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRTfalse2024-03-04T17:51:30.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgRxtOPcYyW7cm-IBYtAu0zyYNScRyoTfalse2024-03-04T17:51:30.416Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjme6.y9cJ8bkVcZ04PNMrwIXjDQUGzXRfalse2024-03-04T17:51:30.386Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoHs8pGuS1bc-GYUbjIaqPQPwQGDukkcfalse2024-03-04T17:51:30.469Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj160IY3UeXb5qH4Rh4JB09PNigxsTa5gfalse2024-03-04T17:51:30.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9WE9mNbplYdU449Vkwh8lwrICYxU6v7false2024-03-04T17:51:30.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZJZPq8HwR1LmdjGaT37eTcVO7gvndKgfalse2024-03-04T17:51:30.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lwBSkQlMAA54TFAsh7J7MCnzo0gTfafalse2024-03-04T17:51:30.157Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq4h-uWPuCpU7FK9SfN-LHLv-vtuKwuEfalse2024-03-04T17:51:30.239Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzKZVhMV59rWXQPf05XlhVIimSaZTr40false2024-03-04T17:51:30.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAcrzzw6Ded5-7UcCqpx-w2RGP5BSbOUfalse2024-03-04T17:51:30.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdVzfZSb65W93TGSPX8waGA5tm4N9lW6false2024-03-04T17:51:30.037Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlwBZf1YBiU2Wqy250Elw-2.f6FuQF0efalse2024-03-04T17:51:30.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.559 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:53:34.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:34.562 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.562 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.562 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:34.562 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:34.562 DEBUG:boto:Method: GET 2024-03-04T17:53:34.563 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:34.563 DEBUG:boto:Data: 2024-03-04T17:53:34.563 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.563 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.563 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.563 DEBUG:boto:Params: {} 2024-03-04T17:53:34.563 DEBUG:boto:Token: None 2024-03-04T17:53:34.563 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/?versions 2024-03-04T17:53:34.563 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MZMUw4/Hxy2oZPAXvlXKswphnKg= 2024-03-04T17:53:34.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MZMUw4/Hxy2oZPAXvlXKswphnKg=', 'Content-Length': '0'} 2024-03-04T17:53:34.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066097b90d8fa5a2c-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.570 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG6tTJPArSLLFYQGx5Z9J94Qp3btVq8wtrue2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQDkS502cdQ0sD3MSSr4AJ.nNDwrboFEfalse2024-03-04T17:51:31.005Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfexW91D920ZLyPSn0KVa8L-0Jf11s3qfalse2024-03-04T17:51:31.057Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4aDzoElev9M4kJpBvZIFgGfNSrnTu1hfalse2024-03-04T17:51:30.946Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrBR7sHkOppK-w8aON9rPlfIadTwy8LKfalse2024-03-04T17:51:30.974Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtp1YBJlmM1cmYYj2xRDAuLHN-EIcD23false2024-03-04T17:51:30.916Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxsfalse2024-03-04T17:51:30.859Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIgnYdWgPuuz09tkj9qWq7P8qzgBSCIyfalse2024-03-04T17:51:30.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZw7l4YYB37p1xa7jtLR7kDu1d5Xjl6Gfalse2024-03-04T17:51:30.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10Vjj0fS8rYkr3OIS1pBSR-3fCH6svyfalse2024-03-04T17:51:30.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXak5SqkkJ7eVzuGI.8.u99lZqCC3tNwfalse2024-03-04T17:51:30.749Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzyjvmlB6z-cVoQlri1b6gVvpnPVUCjFfalse2024-03-04T17:51:30.721Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6BBhxVS-A79T2MI6AHE.THWDO21iftffalse2024-03-04T17:51:30.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7kfalse2024-03-04T17:51:30.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji.eNnYbM.4yY91CfDbCxKIqa-Uec-fxfalse2024-03-04T17:51:30.610Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkUfalse2024-03-04T17:51:30.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje9K3WlwCsFUmkYneucDfMv9rNQO6cdpfalse2024-03-04T17:51:30.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRTfalse2024-03-04T17:51:30.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgRxtOPcYyW7cm-IBYtAu0zyYNScRyoTfalse2024-03-04T17:51:30.416Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjme6.y9cJ8bkVcZ04PNMrwIXjDQUGzXRfalse2024-03-04T17:51:30.386Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoHs8pGuS1bc-GYUbjIaqPQPwQGDukkcfalse2024-03-04T17:51:30.469Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj160IY3UeXb5qH4Rh4JB09PNigxsTa5gfalse2024-03-04T17:51:30.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9WE9mNbplYdU449Vkwh8lwrICYxU6v7false2024-03-04T17:51:30.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZJZPq8HwR1LmdjGaT37eTcVO7gvndKgfalse2024-03-04T17:51:30.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lwBSkQlMAA54TFAsh7J7MCnzo0gTfafalse2024-03-04T17:51:30.157Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq4h-uWPuCpU7FK9SfN-LHLv-vtuKwuEfalse2024-03-04T17:51:30.239Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzKZVhMV59rWXQPf05XlhVIimSaZTr40false2024-03-04T17:51:30.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAcrzzw6Ded5-7UcCqpx-w2RGP5BSbOUfalse2024-03-04T17:51:30.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdVzfZSb65W93TGSPX8waGA5tm4N9lW6false2024-03-04T17:51:30.037Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlwBZf1YBiU2Wqy250Elw-2.f6FuQF0efalse2024-03-04T17:51:30.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.572 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.572 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.572 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:34.572 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:34.572 DEBUG:boto:Method: GET 2024-03-04T17:53:34.572 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:34.572 DEBUG:boto:Data: 2024-03-04T17:53:34.572 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.572 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.573 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.573 DEBUG:boto:Params: {} 2024-03-04T17:53:34.573 DEBUG:boto:Token: None 2024-03-04T17:53:34.573 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/?versions 2024-03-04T17:53:34.573 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MZMUw4/Hxy2oZPAXvlXKswphnKg= 2024-03-04T17:53:34.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MZMUw4/Hxy2oZPAXvlXKswphnKg=', 'Content-Length': '0'} 2024-03-04T17:53:34.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca76827bab0a4a5c-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.579 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG6tTJPArSLLFYQGx5Z9J94Qp3btVq8wtrue2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQDkS502cdQ0sD3MSSr4AJ.nNDwrboFEfalse2024-03-04T17:51:31.005Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfexW91D920ZLyPSn0KVa8L-0Jf11s3qfalse2024-03-04T17:51:31.057Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4aDzoElev9M4kJpBvZIFgGfNSrnTu1hfalse2024-03-04T17:51:30.946Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrBR7sHkOppK-w8aON9rPlfIadTwy8LKfalse2024-03-04T17:51:30.974Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtp1YBJlmM1cmYYj2xRDAuLHN-EIcD23false2024-03-04T17:51:30.916Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxsfalse2024-03-04T17:51:30.859Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIgnYdWgPuuz09tkj9qWq7P8qzgBSCIyfalse2024-03-04T17:51:30.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZw7l4YYB37p1xa7jtLR7kDu1d5Xjl6Gfalse2024-03-04T17:51:30.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10Vjj0fS8rYkr3OIS1pBSR-3fCH6svyfalse2024-03-04T17:51:30.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXak5SqkkJ7eVzuGI.8.u99lZqCC3tNwfalse2024-03-04T17:51:30.749Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzyjvmlB6z-cVoQlri1b6gVvpnPVUCjFfalse2024-03-04T17:51:30.721Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6BBhxVS-A79T2MI6AHE.THWDO21iftffalse2024-03-04T17:51:30.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7kfalse2024-03-04T17:51:30.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji.eNnYbM.4yY91CfDbCxKIqa-Uec-fxfalse2024-03-04T17:51:30.610Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkUfalse2024-03-04T17:51:30.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje9K3WlwCsFUmkYneucDfMv9rNQO6cdpfalse2024-03-04T17:51:30.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRTfalse2024-03-04T17:51:30.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgRxtOPcYyW7cm-IBYtAu0zyYNScRyoTfalse2024-03-04T17:51:30.416Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjme6.y9cJ8bkVcZ04PNMrwIXjDQUGzXRfalse2024-03-04T17:51:30.386Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoHs8pGuS1bc-GYUbjIaqPQPwQGDukkcfalse2024-03-04T17:51:30.469Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj160IY3UeXb5qH4Rh4JB09PNigxsTa5gfalse2024-03-04T17:51:30.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9WE9mNbplYdU449Vkwh8lwrICYxU6v7false2024-03-04T17:51:30.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZJZPq8HwR1LmdjGaT37eTcVO7gvndKgfalse2024-03-04T17:51:30.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lwBSkQlMAA54TFAsh7J7MCnzo0gTfafalse2024-03-04T17:51:30.157Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq4h-uWPuCpU7FK9SfN-LHLv-vtuKwuEfalse2024-03-04T17:51:30.239Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzKZVhMV59rWXQPf05XlhVIimSaZTr40false2024-03-04T17:51:30.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAcrzzw6Ded5-7UcCqpx-w2RGP5BSbOUfalse2024-03-04T17:51:30.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdVzfZSb65W93TGSPX8waGA5tm4N9lW6false2024-03-04T17:51:30.037Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlwBZf1YBiU2Wqy250Elw-2.f6FuQF0efalse2024-03-04T17:51:30.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:53:34.581 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:34.581 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:34.582 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.582 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.582 DEBUG:boto:Method: GET 2024-03-04T17:53:34.582 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.582 DEBUG:boto:Data: 2024-03-04T17:53:34.582 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.582 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.582 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.582 DEBUG:boto:Params: {} 2024-03-04T17:53:34.583 DEBUG:boto:Token: None 2024-03-04T17:53:34.583 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.583 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l418AiM8GPRVE2bNNuPNnlWeJ6M= 2024-03-04T17:53:34.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l418AiM8GPRVE2bNNuPNnlWeJ6M=', 'Content-Length': '0'} 2024-03-04T17:53:34.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006a90fe5b80140e80-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.586 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:34.586 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:34.586 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.587 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.587 DEBUG:boto:Method: GET 2024-03-04T17:53:34.587 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.587 DEBUG:boto:Data: 2024-03-04T17:53:34.587 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.587 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.587 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.587 DEBUG:boto:Params: {} 2024-03-04T17:53:34.587 DEBUG:boto:Token: None 2024-03-04T17:53:34.588 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.588 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l418AiM8GPRVE2bNNuPNnlWeJ6M= 2024-03-04T17:53:34.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l418AiM8GPRVE2bNNuPNnlWeJ6M=', 'Content-Length': '0'} 2024-03-04T17:53:34.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000087b7ffc2e3130e1f-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.591 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:34.592 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:34.592 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.592 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.593 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.593 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.593 DEBUG:boto:Data: 2024-03-04T17:53:34.593 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.593 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.593 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.593 DEBUG:boto:Params: {} 2024-03-04T17:53:34.593 DEBUG:boto:Token: None 2024-03-04T17:53:34.594 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.594 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:feN87/0efs8l1Rerkt/36ftxd9w= 2024-03-04T17:53:34.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:feN87/0efs8l1Rerkt/36ftxd9w=', 'Content-Length': '0'} 2024-03-04T17:53:34.597 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e1b5bf08bf4e4b9c-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.597 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:34.597 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:34.597 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.597 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.597 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.597 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.598 DEBUG:boto:Data: 2024-03-04T17:53:34.598 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.598 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.598 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.598 DEBUG:boto:Params: {} 2024-03-04T17:53:34.598 DEBUG:boto:Token: None 2024-03-04T17:53:34.598 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.598 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:feN87/0efs8l1Rerkt/36ftxd9w= 2024-03-04T17:53:34.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:feN87/0efs8l1Rerkt/36ftxd9w=', 'Content-Length': '0'} 2024-03-04T17:53:34.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d91f5d9bd020c2f-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:53:34.601 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:34.602 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:34.602 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.602 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.602 DEBUG:boto:Method: GET 2024-03-04T17:53:34.602 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.602 DEBUG:boto:Data: 2024-03-04T17:53:34.602 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.602 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.602 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.602 DEBUG:boto:Params: {} 2024-03-04T17:53:34.603 DEBUG:boto:Token: None 2024-03-04T17:53:34.603 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.603 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l418AiM8GPRVE2bNNuPNnlWeJ6M= 2024-03-04T17:53:34.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l418AiM8GPRVE2bNNuPNnlWeJ6M=', 'Content-Length': '0'} 2024-03-04T17:53:34.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005413342e2141fc21-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.606 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:34.606 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:34.606 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.607 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.607 DEBUG:boto:Method: GET 2024-03-04T17:53:34.607 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.607 DEBUG:boto:Data: 2024-03-04T17:53:34.607 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.607 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.607 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.607 DEBUG:boto:Params: {} 2024-03-04T17:53:34.607 DEBUG:boto:Token: None 2024-03-04T17:53:34.607 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.608 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l418AiM8GPRVE2bNNuPNnlWeJ6M= 2024-03-04T17:53:34.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l418AiM8GPRVE2bNNuPNnlWeJ6M=', 'Content-Length': '0'} 2024-03-04T17:53:34.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b5fa76ddc540d86-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.611 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.611 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.611 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:34.611 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:34.611 DEBUG:boto:Method: GET 2024-03-04T17:53:34.611 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:53:34.611 DEBUG:boto:Data: 2024-03-04T17:53:34.611 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.611 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.611 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.612 DEBUG:boto:Params: {} 2024-03-04T17:53:34.612 DEBUG:boto:Token: None 2024-03-04T17:53:34.612 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.612 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002561e462e62b159c-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.660 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.660 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.661 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.661 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:34.661 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:34.661 DEBUG:boto:Method: GET 2024-03-04T17:53:34.661 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:53:34.661 DEBUG:boto:Data: 2024-03-04T17:53:34.661 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.661 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.661 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.661 DEBUG:boto:Params: {} 2024-03-04T17:53:34.662 DEBUG:boto:Token: None 2024-03-04T17:53:34.662 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.662 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a5bdace7f863e0f-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.666 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.667 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:34.667 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:34.667 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.667 DEBUG:boto:Path: /ppfahn-22/dummy 2024-03-04T17:53:34.667 DEBUG:boto:Data: 2024-03-04T17:53:34.667 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.667 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.667 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.668 DEBUG:boto:Params: {} 2024-03-04T17:53:34.668 DEBUG:boto:Token: None 2024-03-04T17:53:34.668 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/dummy 2024-03-04T17:53:34.668 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mjIlzz/ihtYk0uULVNflbbo6Klw= 2024-03-04T17:53:34.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mjIlzz/ihtYk0uULVNflbbo6Klw=', 'Content-Length': '0'} 2024-03-04T17:53:34.672 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000375b5d61c132e068-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.672 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:34.672 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:34.672 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.673 DEBUG:boto:Path: /ppfahn-22/dummy 2024-03-04T17:53:34.673 DEBUG:boto:Data: 2024-03-04T17:53:34.673 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.673 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.673 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.673 DEBUG:boto:Params: {} 2024-03-04T17:53:34.673 DEBUG:boto:Token: None 2024-03-04T17:53:34.673 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/dummy 2024-03-04T17:53:34.673 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mjIlzz/ihtYk0uULVNflbbo6Klw= 2024-03-04T17:53:34.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mjIlzz/ihtYk0uULVNflbbo6Klw=', 'Content-Length': '0'} 2024-03-04T17:53:34.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000804b1ba284dc739c-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:53:34.678 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:34.678 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:34.678 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.678 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.678 DEBUG:boto:Method: GET 2024-03-04T17:53:34.678 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.678 DEBUG:boto:Data: 2024-03-04T17:53:34.678 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.678 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.679 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.679 DEBUG:boto:Params: {} 2024-03-04T17:53:34.679 DEBUG:boto:Token: None 2024-03-04T17:53:34.679 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.679 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l418AiM8GPRVE2bNNuPNnlWeJ6M= 2024-03-04T17:53:34.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l418AiM8GPRVE2bNNuPNnlWeJ6M=', 'Content-Length': '0'} 2024-03-04T17:53:34.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c5c546791e20d37-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.682 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:34.683 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:34.683 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.683 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.683 DEBUG:boto:Method: GET 2024-03-04T17:53:34.683 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.683 DEBUG:boto:Data: 2024-03-04T17:53:34.683 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.683 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.683 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.683 DEBUG:boto:Params: {} 2024-03-04T17:53:34.684 DEBUG:boto:Token: None 2024-03-04T17:53:34.684 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:34.684 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l418AiM8GPRVE2bNNuPNnlWeJ6M= 2024-03-04T17:53:34.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l418AiM8GPRVE2bNNuPNnlWeJ6M=', 'Content-Length': '0'} 2024-03-04T17:53:34.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060d76185352aeb05-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.687 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.687 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.687 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:34.687 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:34.688 DEBUG:boto:Method: GET 2024-03-04T17:53:34.688 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:53:34.688 DEBUG:boto:Data: 2024-03-04T17:53:34.688 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.688 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.688 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.688 DEBUG:boto:Params: {} 2024-03-04T17:53:34.688 DEBUG:boto:Token: None 2024-03-04T17:53:34.688 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.689 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ac92f3b115531f4-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.693 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.693 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.694 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.694 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:34.694 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:34.694 DEBUG:boto:Method: GET 2024-03-04T17:53:34.694 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:53:34.694 DEBUG:boto:Data: 2024-03-04T17:53:34.694 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.694 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.694 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.694 DEBUG:boto:Params: {} 2024-03-04T17:53:34.695 DEBUG:boto:Token: None 2024-03-04T17:53:34.695 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.695 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfde74d36c196558-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.700 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:34.700 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.700 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.700 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.700 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.700 DEBUG:boto:Method: GET 2024-03-04T17:53:34.701 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.701 DEBUG:boto:Data: 2024-03-04T17:53:34.701 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.701 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.701 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.701 DEBUG:boto:Params: {} 2024-03-04T17:53:34.701 DEBUG:boto:Token: None 2024-03-04T17:53:34.701 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.701 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g= 2024-03-04T17:53:34.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g=', 'Content-Length': '0'} 2024-03-04T17:53:34.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000444b058c21878b1d-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.748 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.748 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.748 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.748 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.748 DEBUG:boto:Method: GET 2024-03-04T17:53:34.748 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.749 DEBUG:boto:Data: 2024-03-04T17:53:34.749 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.749 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.749 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.749 DEBUG:boto:Params: {} 2024-03-04T17:53:34.749 DEBUG:boto:Token: None 2024-03-04T17:53:34.749 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.749 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g= 2024-03-04T17:53:34.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g=', 'Content-Length': '0'} 2024-03-04T17:53:34.752 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb4b7fe84649c003-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.753 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.753 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.753 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.753 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.753 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.753 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.753 DEBUG:boto:Data: 2024-03-04T17:53:34.753 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.753 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.753 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.753 DEBUG:boto:Params: {} 2024-03-04T17:53:34.754 DEBUG:boto:Token: None 2024-03-04T17:53:34.754 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.754 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+Wzc6QUYY82TwpRQwXZrVbGkjJo= 2024-03-04T17:53:34.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+Wzc6QUYY82TwpRQwXZrVbGkjJo=', 'Content-Length': '0'} 2024-03-04T17:53:34.757 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b370c482933146f-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.757 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.757 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.757 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.757 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.757 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.757 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.758 DEBUG:boto:Data: 2024-03-04T17:53:34.758 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.758 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.758 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.758 DEBUG:boto:Params: {} 2024-03-04T17:53:34.758 DEBUG:boto:Token: None 2024-03-04T17:53:34.758 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.758 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+Wzc6QUYY82TwpRQwXZrVbGkjJo= 2024-03-04T17:53:34.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+Wzc6QUYY82TwpRQwXZrVbGkjJo=', 'Content-Length': '0'} 2024-03-04T17:53:34.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037e25f2dfe24c9ea-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:34.762 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.762 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.762 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.762 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.762 DEBUG:boto:Method: GET 2024-03-04T17:53:34.762 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.762 DEBUG:boto:Data: 2024-03-04T17:53:34.762 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.762 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.762 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.763 DEBUG:boto:Params: {} 2024-03-04T17:53:34.763 DEBUG:boto:Token: None 2024-03-04T17:53:34.763 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.763 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g= 2024-03-04T17:53:34.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g=', 'Content-Length': '0'} 2024-03-04T17:53:34.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f79c9064b14ce17-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.766 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.766 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.766 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.766 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.766 DEBUG:boto:Method: GET 2024-03-04T17:53:34.767 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.767 DEBUG:boto:Data: 2024-03-04T17:53:34.767 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.767 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.767 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.767 DEBUG:boto:Params: {} 2024-03-04T17:53:34.767 DEBUG:boto:Token: None 2024-03-04T17:53:34.767 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.767 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g= 2024-03-04T17:53:34.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g=', 'Content-Length': '0'} 2024-03-04T17:53:34.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e624d6a6e080fdd4-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.812 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.812 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.812 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.812 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.812 DEBUG:boto:Method: GET 2024-03-04T17:53:34.813 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:34.813 DEBUG:boto:Data: 2024-03-04T17:53:34.813 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.813 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.813 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.813 DEBUG:boto:Params: {} 2024-03-04T17:53:34.813 DEBUG:boto:Token: None 2024-03-04T17:53:34.813 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.813 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e0c222324fbd24c-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.819 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.819 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.819 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.819 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.819 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.819 DEBUG:boto:Method: GET 2024-03-04T17:53:34.820 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:34.820 DEBUG:boto:Data: 2024-03-04T17:53:34.820 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.820 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.820 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.820 DEBUG:boto:Params: {} 2024-03-04T17:53:34.820 DEBUG:boto:Token: None 2024-03-04T17:53:34.820 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.820 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000138054234e26d26a-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.825 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.826 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.826 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.826 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.826 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:34.826 DEBUG:boto:Data: 2024-03-04T17:53:34.826 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.826 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.826 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.826 DEBUG:boto:Params: {} 2024-03-04T17:53:34.826 DEBUG:boto:Token: None 2024-03-04T17:53:34.827 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj 2024-03-04T17:53:34.827 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:By6Zcj0NTSx4Q742VNkwI/Sw4MA= 2024-03-04T17:53:34.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:By6Zcj0NTSx4Q742VNkwI/Sw4MA=', 'Content-Length': '0'} 2024-03-04T17:53:34.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ff6be8d40d748e7-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.833 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.833 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.833 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.833 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:34.833 DEBUG:boto:Data: 2024-03-04T17:53:34.833 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.833 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.834 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.834 DEBUG:boto:Params: {} 2024-03-04T17:53:34.834 DEBUG:boto:Token: None 2024-03-04T17:53:34.834 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj 2024-03-04T17:53:34.834 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:By6Zcj0NTSx4Q742VNkwI/Sw4MA= 2024-03-04T17:53:34.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:By6Zcj0NTSx4Q742VNkwI/Sw4MA=', 'Content-Length': '0'} 2024-03-04T17:53:34.840 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6f8bedfad96959f-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:34.840 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.840 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.840 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.840 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.841 DEBUG:boto:Method: GET 2024-03-04T17:53:34.841 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.841 DEBUG:boto:Data: 2024-03-04T17:53:34.841 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.841 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.841 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.841 DEBUG:boto:Params: {} 2024-03-04T17:53:34.841 DEBUG:boto:Token: None 2024-03-04T17:53:34.841 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.841 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g= 2024-03-04T17:53:34.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g=', 'Content-Length': '0'} 2024-03-04T17:53:34.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c51f90a222260f83-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.888 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.888 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.888 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.888 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.888 DEBUG:boto:Method: GET 2024-03-04T17:53:34.889 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.889 DEBUG:boto:Data: 2024-03-04T17:53:34.889 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.889 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.889 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.889 DEBUG:boto:Params: {} 2024-03-04T17:53:34.889 DEBUG:boto:Token: None 2024-03-04T17:53:34.889 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:34.890 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g= 2024-03-04T17:53:34.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0EpRPdKeY3ohCgLjBy/JwHDUy3g=', 'Content-Length': '0'} 2024-03-04T17:53:34.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d90405e9ca50180-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.936 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.936 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.936 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.936 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.937 DEBUG:boto:Method: GET 2024-03-04T17:53:34.937 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:34.937 DEBUG:boto:Data: 2024-03-04T17:53:34.937 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.937 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.937 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.937 DEBUG:boto:Params: {} 2024-03-04T17:53:34.937 DEBUG:boto:Token: None 2024-03-04T17:53:34.938 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.938 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9e146a8e249a34b-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.943 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.943 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:34.943 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:34.944 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.944 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:34.944 DEBUG:boto:Method: GET 2024-03-04T17:53:34.944 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:34.944 DEBUG:boto:Data: 2024-03-04T17:53:34.944 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.944 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.944 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.944 DEBUG:boto:Params: {} 2024-03-04T17:53:34.944 DEBUG:boto:Token: None 2024-03-04T17:53:34.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/ 2024-03-04T17:53:34.945 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc= 2024-03-04T17:53:34.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDuVPfpywTWql5qQYqBiEk/FsTc=', 'Content-Length': '0'} 2024-03-04T17:53:34.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a6395aed66290c6-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.950 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:34.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:34.951 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.951 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.951 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.951 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.951 DEBUG:boto:Method: GET 2024-03-04T17:53:34.951 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.951 DEBUG:boto:Data: 2024-03-04T17:53:34.951 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.951 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.952 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.952 DEBUG:boto:Params: {} 2024-03-04T17:53:34.952 DEBUG:boto:Token: None 2024-03-04T17:53:34.952 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.952 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1D3UkPUvYcnlefWvyg3Qt2CTsPY= 2024-03-04T17:53:34.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1D3UkPUvYcnlefWvyg3Qt2CTsPY=', 'Content-Length': '0'} 2024-03-04T17:53:34.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d3d179236ba2e15-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.956 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.956 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.956 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.956 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.956 DEBUG:boto:Method: GET 2024-03-04T17:53:34.956 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.956 DEBUG:boto:Data: 2024-03-04T17:53:34.956 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.956 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.957 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.957 DEBUG:boto:Params: {} 2024-03-04T17:53:34.957 DEBUG:boto:Token: None 2024-03-04T17:53:34.957 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.957 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1D3UkPUvYcnlefWvyg3Qt2CTsPY= 2024-03-04T17:53:34.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1D3UkPUvYcnlefWvyg3Qt2CTsPY=', 'Content-Length': '0'} 2024-03-04T17:53:34.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6e35a6974ff8be7-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.960 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.960 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.960 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.961 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.961 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.961 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.961 DEBUG:boto:Data: 2024-03-04T17:53:34.961 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.961 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.961 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.961 DEBUG:boto:Params: {} 2024-03-04T17:53:34.961 DEBUG:boto:Token: None 2024-03-04T17:53:34.961 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.962 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QfyyQ3ZYFvOEQxqw2DoCwX/9ni8= 2024-03-04T17:53:34.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QfyyQ3ZYFvOEQxqw2DoCwX/9ni8=', 'Content-Length': '0'} 2024-03-04T17:53:34.964 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089b8b167d22d9739-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.965 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.965 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.965 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.965 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.965 DEBUG:boto:Method: HEAD 2024-03-04T17:53:34.965 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.965 DEBUG:boto:Data: 2024-03-04T17:53:34.965 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.965 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.965 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.966 DEBUG:boto:Params: {} 2024-03-04T17:53:34.966 DEBUG:boto:Token: None 2024-03-04T17:53:34.966 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.966 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QfyyQ3ZYFvOEQxqw2DoCwX/9ni8= 2024-03-04T17:53:34.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QfyyQ3ZYFvOEQxqw2DoCwX/9ni8=', 'Content-Length': '0'} 2024-03-04T17:53:34.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2895978247fad21-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:34.969 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.969 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.969 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.969 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.969 DEBUG:boto:Method: GET 2024-03-04T17:53:34.969 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.969 DEBUG:boto:Data: 2024-03-04T17:53:34.970 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.970 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:34.970 DEBUG:boto:Port: 8001 2024-03-04T17:53:34.970 DEBUG:boto:Params: {} 2024-03-04T17:53:34.970 DEBUG:boto:Token: None 2024-03-04T17:53:34.970 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.970 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1D3UkPUvYcnlefWvyg3Qt2CTsPY= 2024-03-04T17:53:34.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1D3UkPUvYcnlefWvyg3Qt2CTsPY=', 'Content-Length': '0'} 2024-03-04T17:53:34.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000536abb76bac9718f-0065e60a9e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:34.974 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:34.974 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:34.974 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.974 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.974 DEBUG:boto:Method: GET 2024-03-04T17:53:34.974 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.974 DEBUG:boto:Data: 2024-03-04T17:53:34.974 DEBUG:boto:Headers: {} 2024-03-04T17:53:34.974 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:34.975 DEBUG:boto:Port: 8000 2024-03-04T17:53:34.975 DEBUG:boto:Params: {} 2024-03-04T17:53:34.975 DEBUG:boto:Token: None 2024-03-04T17:53:34.975 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:34 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:34.975 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1D3UkPUvYcnlefWvyg3Qt2CTsPY= 2024-03-04T17:53:34.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:34 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1D3UkPUvYcnlefWvyg3Qt2CTsPY=', 'Content-Length': '0'} 2024-03-04T17:53:34.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa6146ba07525bb8-0065e60a9e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.020 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.020 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.020 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.020 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.020 DEBUG:boto:Method: GET 2024-03-04T17:53:35.021 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.021 DEBUG:boto:Data: 2024-03-04T17:53:35.021 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.021 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.021 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.021 DEBUG:boto:Params: {} 2024-03-04T17:53:35.021 DEBUG:boto:Token: None 2024-03-04T17:53:35.021 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.021 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ed15c824add4f8c-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.027 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.028 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.028 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.028 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.028 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.028 DEBUG:boto:Method: GET 2024-03-04T17:53:35.028 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.028 DEBUG:boto:Data: 2024-03-04T17:53:35.028 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.028 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.028 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.029 DEBUG:boto:Params: {} 2024-03-04T17:53:35.029 DEBUG:boto:Token: None 2024-03-04T17:53:35.029 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.029 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000919280ce8c45480d-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.034 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.034 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.034 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.034 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.034 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:35.035 DEBUG:boto:Data: 2024-03-04T17:53:35.035 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.035 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.035 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.035 DEBUG:boto:Params: {} 2024-03-04T17:53:35.035 DEBUG:boto:Token: None 2024-03-04T17:53:35.035 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj 2024-03-04T17:53:35.035 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI= 2024-03-04T17:53:35.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI=', 'Content-Length': '0'} 2024-03-04T17:53:35.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d56cda00651020d-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.041 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.041 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.041 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.042 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:35.042 DEBUG:boto:Data: 2024-03-04T17:53:35.042 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.042 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.042 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.042 DEBUG:boto:Params: {} 2024-03-04T17:53:35.042 DEBUG:boto:Token: None 2024-03-04T17:53:35.042 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj 2024-03-04T17:53:35.042 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI= 2024-03-04T17:53:35.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI=', 'Content-Length': '0'} 2024-03-04T17:53:35.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f0db7095ea11656-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:35.049 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.049 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.049 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.049 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.049 DEBUG:boto:Method: GET 2024-03-04T17:53:35.049 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.049 DEBUG:boto:Data: 2024-03-04T17:53:35.049 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.049 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.050 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.050 DEBUG:boto:Params: {} 2024-03-04T17:53:35.050 DEBUG:boto:Token: None 2024-03-04T17:53:35.050 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.050 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k= 2024-03-04T17:53:35.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k=', 'Content-Length': '0'} 2024-03-04T17:53:35.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6999a6ac26dda5c-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.100 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.100 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.100 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.100 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.100 DEBUG:boto:Method: GET 2024-03-04T17:53:35.101 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.101 DEBUG:boto:Data: 2024-03-04T17:53:35.101 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.101 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.101 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.101 DEBUG:boto:Params: {} 2024-03-04T17:53:35.101 DEBUG:boto:Token: None 2024-03-04T17:53:35.101 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.102 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k= 2024-03-04T17:53:35.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k=', 'Content-Length': '0'} 2024-03-04T17:53:35.104 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076359b7e9a126f8f-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.148 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.148 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.148 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.148 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.148 DEBUG:boto:Method: GET 2024-03-04T17:53:35.149 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.149 DEBUG:boto:Data: 2024-03-04T17:53:35.149 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.149 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.149 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.149 DEBUG:boto:Params: {} 2024-03-04T17:53:35.149 DEBUG:boto:Token: None 2024-03-04T17:53:35.149 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.149 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b0a6fc92839d0ba-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.154 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.154 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.155 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.155 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.155 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.155 DEBUG:boto:Method: GET 2024-03-04T17:53:35.155 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.155 DEBUG:boto:Data: 2024-03-04T17:53:35.155 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.155 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.155 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.155 DEBUG:boto:Params: {} 2024-03-04T17:53:35.156 DEBUG:boto:Token: None 2024-03-04T17:53:35.156 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.156 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042381b25da1d53db-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.160 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:35.161 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.161 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.161 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.161 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.161 DEBUG:boto:Method: GET 2024-03-04T17:53:35.161 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.162 DEBUG:boto:Data: 2024-03-04T17:53:35.162 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.162 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.162 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.162 DEBUG:boto:Params: {} 2024-03-04T17:53:35.162 DEBUG:boto:Token: None 2024-03-04T17:53:35.162 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.162 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4qocGnQ72ZtJhzHX/WHpKLrdYfY= 2024-03-04T17:53:35.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4qocGnQ72ZtJhzHX/WHpKLrdYfY=', 'Content-Length': '0'} 2024-03-04T17:53:35.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d63f61825d40c61-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.166 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.166 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.166 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.166 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.166 DEBUG:boto:Method: GET 2024-03-04T17:53:35.166 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.166 DEBUG:boto:Data: 2024-03-04T17:53:35.166 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.167 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.167 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.167 DEBUG:boto:Params: {} 2024-03-04T17:53:35.167 DEBUG:boto:Token: None 2024-03-04T17:53:35.167 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.167 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4qocGnQ72ZtJhzHX/WHpKLrdYfY= 2024-03-04T17:53:35.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4qocGnQ72ZtJhzHX/WHpKLrdYfY=', 'Content-Length': '0'} 2024-03-04T17:53:35.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f65a1fdf5482c0d-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.212 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.212 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.212 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.212 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.212 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.213 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.213 DEBUG:boto:Data: 2024-03-04T17:53:35.213 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.213 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.213 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.213 DEBUG:boto:Params: {} 2024-03-04T17:53:35.213 DEBUG:boto:Token: None 2024-03-04T17:53:35.213 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.213 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SlNIYJXZoLShCZCXB5gBwXGYcYU= 2024-03-04T17:53:35.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SlNIYJXZoLShCZCXB5gBwXGYcYU=', 'Content-Length': '0'} 2024-03-04T17:53:35.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000809ab31c5661ab3-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.217 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.217 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.217 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.217 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.217 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.217 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.217 DEBUG:boto:Data: 2024-03-04T17:53:35.217 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.217 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.218 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.218 DEBUG:boto:Params: {} 2024-03-04T17:53:35.218 DEBUG:boto:Token: None 2024-03-04T17:53:35.218 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.218 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SlNIYJXZoLShCZCXB5gBwXGYcYU= 2024-03-04T17:53:35.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SlNIYJXZoLShCZCXB5gBwXGYcYU=', 'Content-Length': '0'} 2024-03-04T17:53:35.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3ce2d7e5ae45c39-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:35.221 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.221 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.222 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.222 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.222 DEBUG:boto:Method: GET 2024-03-04T17:53:35.222 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.222 DEBUG:boto:Data: 2024-03-04T17:53:35.222 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.222 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.222 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.222 DEBUG:boto:Params: {} 2024-03-04T17:53:35.222 DEBUG:boto:Token: None 2024-03-04T17:53:35.223 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.223 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4qocGnQ72ZtJhzHX/WHpKLrdYfY= 2024-03-04T17:53:35.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4qocGnQ72ZtJhzHX/WHpKLrdYfY=', 'Content-Length': '0'} 2024-03-04T17:53:35.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8dbf4c31dd7e7e2-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.226 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.226 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.226 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.227 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.227 DEBUG:boto:Method: GET 2024-03-04T17:53:35.227 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.227 DEBUG:boto:Data: 2024-03-04T17:53:35.227 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.227 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.227 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.227 DEBUG:boto:Params: {} 2024-03-04T17:53:35.227 DEBUG:boto:Token: None 2024-03-04T17:53:35.227 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:35.228 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4qocGnQ72ZtJhzHX/WHpKLrdYfY= 2024-03-04T17:53:35.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4qocGnQ72ZtJhzHX/WHpKLrdYfY=', 'Content-Length': '0'} 2024-03-04T17:53:35.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077ce17179765766c-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.272 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.272 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.272 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.272 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.273 DEBUG:boto:Method: GET 2024-03-04T17:53:35.273 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.273 DEBUG:boto:Data: 2024-03-04T17:53:35.273 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.273 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.273 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.273 DEBUG:boto:Params: {} 2024-03-04T17:53:35.273 DEBUG:boto:Token: None 2024-03-04T17:53:35.273 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.273 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000300d230b1cbb76c3-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.279 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.279 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.279 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.279 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.279 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.279 DEBUG:boto:Method: GET 2024-03-04T17:53:35.280 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.280 DEBUG:boto:Data: 2024-03-04T17:53:35.280 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.280 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.280 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.280 DEBUG:boto:Params: {} 2024-03-04T17:53:35.280 DEBUG:boto:Token: None 2024-03-04T17:53:35.280 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.280 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000335c72faa79023e3-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.286 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.286 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.286 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.286 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.286 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:35.286 DEBUG:boto:Data: 2024-03-04T17:53:35.287 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.287 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.287 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.287 DEBUG:boto:Params: {} 2024-03-04T17:53:35.287 DEBUG:boto:Token: None 2024-03-04T17:53:35.287 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj 2024-03-04T17:53:35.287 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI= 2024-03-04T17:53:35.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI=', 'Content-Length': '0'} 2024-03-04T17:53:35.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7867d16dee5b716-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.293 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.293 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.294 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.294 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:35.294 DEBUG:boto:Data: 2024-03-04T17:53:35.294 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.294 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.294 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.294 DEBUG:boto:Params: {} 2024-03-04T17:53:35.295 DEBUG:boto:Token: None 2024-03-04T17:53:35.295 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj 2024-03-04T17:53:35.295 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI= 2024-03-04T17:53:35.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI=', 'Content-Length': '0'} 2024-03-04T17:53:35.300 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000030c285d1a2b86a8-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:35.301 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.301 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.301 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.301 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.301 DEBUG:boto:Method: GET 2024-03-04T17:53:35.301 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.302 DEBUG:boto:Data: 2024-03-04T17:53:35.302 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.302 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.302 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.302 DEBUG:boto:Params: {} 2024-03-04T17:53:35.302 DEBUG:boto:Token: None 2024-03-04T17:53:35.302 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.302 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k= 2024-03-04T17:53:35.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k=', 'Content-Length': '0'} 2024-03-04T17:53:35.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007aa31b446a1bb556-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.348 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.348 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.348 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.348 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.348 DEBUG:boto:Method: GET 2024-03-04T17:53:35.348 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.349 DEBUG:boto:Data: 2024-03-04T17:53:35.349 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.349 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.349 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.349 DEBUG:boto:Params: {} 2024-03-04T17:53:35.349 DEBUG:boto:Token: None 2024-03-04T17:53:35.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.349 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k= 2024-03-04T17:53:35.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k=', 'Content-Length': '0'} 2024-03-04T17:53:35.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d058bb7aa199471b-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.396 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.396 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.396 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.396 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.396 DEBUG:boto:Method: GET 2024-03-04T17:53:35.397 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.397 DEBUG:boto:Data: 2024-03-04T17:53:35.397 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.397 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.397 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.397 DEBUG:boto:Params: {} 2024-03-04T17:53:35.397 DEBUG:boto:Token: None 2024-03-04T17:53:35.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.397 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf75387173c7ca98-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.404 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.404 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.404 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.404 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.404 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.404 DEBUG:boto:Method: GET 2024-03-04T17:53:35.404 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.405 DEBUG:boto:Data: 2024-03-04T17:53:35.405 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.405 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.405 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.405 DEBUG:boto:Params: {} 2024-03-04T17:53:35.405 DEBUG:boto:Token: None 2024-03-04T17:53:35.405 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.405 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000663c28f9565cf415-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.411 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:35.411 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.411 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.411 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.411 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.411 DEBUG:boto:Method: GET 2024-03-04T17:53:35.412 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.412 DEBUG:boto:Data: 2024-03-04T17:53:35.412 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.412 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.412 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.412 DEBUG:boto:Params: {} 2024-03-04T17:53:35.412 DEBUG:boto:Token: None 2024-03-04T17:53:35.412 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.412 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OniiNp9obmr2Aj2dG1pOdFT8kog= 2024-03-04T17:53:35.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OniiNp9obmr2Aj2dG1pOdFT8kog=', 'Content-Length': '0'} 2024-03-04T17:53:35.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000177237f539a7272a-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.417 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.417 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.417 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.417 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.417 DEBUG:boto:Method: GET 2024-03-04T17:53:35.417 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.418 DEBUG:boto:Data: 2024-03-04T17:53:35.418 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.418 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.418 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.418 DEBUG:boto:Params: {} 2024-03-04T17:53:35.418 DEBUG:boto:Token: None 2024-03-04T17:53:35.418 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.418 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OniiNp9obmr2Aj2dG1pOdFT8kog= 2024-03-04T17:53:35.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OniiNp9obmr2Aj2dG1pOdFT8kog=', 'Content-Length': '0'} 2024-03-04T17:53:35.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e527a54f87242ea-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.468 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.468 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.468 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.469 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.469 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.469 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.469 DEBUG:boto:Data: 2024-03-04T17:53:35.469 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.469 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.469 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.469 DEBUG:boto:Params: {} 2024-03-04T17:53:35.469 DEBUG:boto:Token: None 2024-03-04T17:53:35.469 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.470 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nak9Y+Rkd7jtO4xuwZv2V4UPLaA= 2024-03-04T17:53:35.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nak9Y+Rkd7jtO4xuwZv2V4UPLaA=', 'Content-Length': '0'} 2024-03-04T17:53:35.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006956ba7c54811022-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.473 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.473 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.473 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.473 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.473 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.473 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.473 DEBUG:boto:Data: 2024-03-04T17:53:35.473 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.473 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.473 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.473 DEBUG:boto:Params: {} 2024-03-04T17:53:35.474 DEBUG:boto:Token: None 2024-03-04T17:53:35.474 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.474 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nak9Y+Rkd7jtO4xuwZv2V4UPLaA= 2024-03-04T17:53:35.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nak9Y+Rkd7jtO4xuwZv2V4UPLaA=', 'Content-Length': '0'} 2024-03-04T17:53:35.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d840819b04fb361c-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:35.477 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.478 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.478 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.478 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.478 DEBUG:boto:Method: GET 2024-03-04T17:53:35.478 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.478 DEBUG:boto:Data: 2024-03-04T17:53:35.478 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.478 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.478 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.478 DEBUG:boto:Params: {} 2024-03-04T17:53:35.479 DEBUG:boto:Token: None 2024-03-04T17:53:35.479 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.479 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OniiNp9obmr2Aj2dG1pOdFT8kog= 2024-03-04T17:53:35.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OniiNp9obmr2Aj2dG1pOdFT8kog=', 'Content-Length': '0'} 2024-03-04T17:53:35.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047c80e059f022e50-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.482 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.482 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.482 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.482 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.482 DEBUG:boto:Method: GET 2024-03-04T17:53:35.482 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.483 DEBUG:boto:Data: 2024-03-04T17:53:35.483 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.483 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.483 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.483 DEBUG:boto:Params: {} 2024-03-04T17:53:35.483 DEBUG:boto:Token: None 2024-03-04T17:53:35.483 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:35.483 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OniiNp9obmr2Aj2dG1pOdFT8kog= 2024-03-04T17:53:35.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OniiNp9obmr2Aj2dG1pOdFT8kog=', 'Content-Length': '0'} 2024-03-04T17:53:35.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000126f7075dac338ab-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.528 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.528 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.528 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.528 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.528 DEBUG:boto:Method: GET 2024-03-04T17:53:35.528 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.529 DEBUG:boto:Data: 2024-03-04T17:53:35.529 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.529 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.529 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.529 DEBUG:boto:Params: {} 2024-03-04T17:53:35.529 DEBUG:boto:Token: None 2024-03-04T17:53:35.529 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.529 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e4c6d0fdde81df2-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.534 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.534 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.535 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.535 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.535 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.535 DEBUG:boto:Method: GET 2024-03-04T17:53:35.535 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.535 DEBUG:boto:Data: 2024-03-04T17:53:35.535 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.535 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.535 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.535 DEBUG:boto:Params: {} 2024-03-04T17:53:35.535 DEBUG:boto:Token: None 2024-03-04T17:53:35.536 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.536 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd9c54cf6e9a15df-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.541 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.541 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.541 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.541 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.542 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:35.542 DEBUG:boto:Data: 2024-03-04T17:53:35.542 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.542 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.542 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.542 DEBUG:boto:Params: {} 2024-03-04T17:53:35.542 DEBUG:boto:Token: None 2024-03-04T17:53:35.542 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj 2024-03-04T17:53:35.542 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI= 2024-03-04T17:53:35.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI=', 'Content-Length': '0'} 2024-03-04T17:53:35.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d270134d2e496399-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.548 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.548 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.549 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.549 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:35.549 DEBUG:boto:Data: 2024-03-04T17:53:35.549 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.549 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.549 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.549 DEBUG:boto:Params: {} 2024-03-04T17:53:35.549 DEBUG:boto:Token: None 2024-03-04T17:53:35.549 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj 2024-03-04T17:53:35.550 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI= 2024-03-04T17:53:35.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI=', 'Content-Length': '0'} 2024-03-04T17:53:35.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048d5c74ff2cade6e-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:35.556 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.556 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.556 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.556 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.556 DEBUG:boto:Method: GET 2024-03-04T17:53:35.556 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.556 DEBUG:boto:Data: 2024-03-04T17:53:35.556 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.556 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.557 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.557 DEBUG:boto:Params: {} 2024-03-04T17:53:35.557 DEBUG:boto:Token: None 2024-03-04T17:53:35.557 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.557 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k= 2024-03-04T17:53:35.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k=', 'Content-Length': '0'} 2024-03-04T17:53:35.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b1470e569afc8eb-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.608 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.608 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.608 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.608 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.608 DEBUG:boto:Method: GET 2024-03-04T17:53:35.608 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.609 DEBUG:boto:Data: 2024-03-04T17:53:35.609 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.609 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.609 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.609 DEBUG:boto:Params: {} 2024-03-04T17:53:35.609 DEBUG:boto:Token: None 2024-03-04T17:53:35.609 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.609 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k= 2024-03-04T17:53:35.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k=', 'Content-Length': '0'} 2024-03-04T17:53:35.613 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003336e47ce4cabaf6-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.656 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.656 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.656 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.656 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.656 DEBUG:boto:Method: GET 2024-03-04T17:53:35.657 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.657 DEBUG:boto:Data: 2024-03-04T17:53:35.657 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.657 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.657 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.657 DEBUG:boto:Params: {} 2024-03-04T17:53:35.657 DEBUG:boto:Token: None 2024-03-04T17:53:35.657 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.657 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000152c93ebee94dd96-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.662 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.663 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.663 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.663 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.663 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.663 DEBUG:boto:Method: GET 2024-03-04T17:53:35.663 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.663 DEBUG:boto:Data: 2024-03-04T17:53:35.664 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.664 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.664 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.664 DEBUG:boto:Params: {} 2024-03-04T17:53:35.664 DEBUG:boto:Token: None 2024-03-04T17:53:35.664 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.664 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f543eb332700e374-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.669 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:35.669 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.670 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.670 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.670 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.670 DEBUG:boto:Method: GET 2024-03-04T17:53:35.670 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.670 DEBUG:boto:Data: 2024-03-04T17:53:35.670 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.670 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.670 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.670 DEBUG:boto:Params: {} 2024-03-04T17:53:35.671 DEBUG:boto:Token: None 2024-03-04T17:53:35.671 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.671 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ncVaW93tUEUydeQb0jrOe5g/70o= 2024-03-04T17:53:35.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ncVaW93tUEUydeQb0jrOe5g/70o=', 'Content-Length': '0'} 2024-03-04T17:53:35.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006a0cbc732c204f8-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.674 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.674 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.674 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.674 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.675 DEBUG:boto:Method: GET 2024-03-04T17:53:35.675 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.675 DEBUG:boto:Data: 2024-03-04T17:53:35.675 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.675 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.675 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.675 DEBUG:boto:Params: {} 2024-03-04T17:53:35.675 DEBUG:boto:Token: None 2024-03-04T17:53:35.675 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.676 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ncVaW93tUEUydeQb0jrOe5g/70o= 2024-03-04T17:53:35.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ncVaW93tUEUydeQb0jrOe5g/70o=', 'Content-Length': '0'} 2024-03-04T17:53:35.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003edf0f21513947f8-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.724 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.724 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.724 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.724 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.724 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.724 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.725 DEBUG:boto:Data: 2024-03-04T17:53:35.725 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.725 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.725 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.725 DEBUG:boto:Params: {} 2024-03-04T17:53:35.725 DEBUG:boto:Token: None 2024-03-04T17:53:35.725 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.725 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dbeYmpJnRFr3/ROqIOjurg2lxjw= 2024-03-04T17:53:35.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dbeYmpJnRFr3/ROqIOjurg2lxjw=', 'Content-Length': '0'} 2024-03-04T17:53:35.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d240536e314ccff2-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.728 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.728 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.729 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.729 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.729 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.729 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.729 DEBUG:boto:Data: 2024-03-04T17:53:35.729 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.729 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.729 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.729 DEBUG:boto:Params: {} 2024-03-04T17:53:35.729 DEBUG:boto:Token: None 2024-03-04T17:53:35.729 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.730 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dbeYmpJnRFr3/ROqIOjurg2lxjw= 2024-03-04T17:53:35.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dbeYmpJnRFr3/ROqIOjurg2lxjw=', 'Content-Length': '0'} 2024-03-04T17:53:35.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbcde786c70221ea-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:35.733 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.733 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.733 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.733 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.733 DEBUG:boto:Method: GET 2024-03-04T17:53:35.733 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.733 DEBUG:boto:Data: 2024-03-04T17:53:35.733 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.734 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.734 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.734 DEBUG:boto:Params: {} 2024-03-04T17:53:35.734 DEBUG:boto:Token: None 2024-03-04T17:53:35.734 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.734 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ncVaW93tUEUydeQb0jrOe5g/70o= 2024-03-04T17:53:35.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ncVaW93tUEUydeQb0jrOe5g/70o=', 'Content-Length': '0'} 2024-03-04T17:53:35.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff201603aa8e40fc-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.737 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.738 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.738 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.738 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.738 DEBUG:boto:Method: GET 2024-03-04T17:53:35.738 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.738 DEBUG:boto:Data: 2024-03-04T17:53:35.738 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.738 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.738 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.738 DEBUG:boto:Params: {} 2024-03-04T17:53:35.739 DEBUG:boto:Token: None 2024-03-04T17:53:35.739 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:35.739 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ncVaW93tUEUydeQb0jrOe5g/70o= 2024-03-04T17:53:35.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ncVaW93tUEUydeQb0jrOe5g/70o=', 'Content-Length': '0'} 2024-03-04T17:53:35.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aeda4bb9bfdc90a9-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.784 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.784 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.784 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.784 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.784 DEBUG:boto:Method: GET 2024-03-04T17:53:35.784 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.785 DEBUG:boto:Data: 2024-03-04T17:53:35.785 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.785 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.785 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.785 DEBUG:boto:Params: {} 2024-03-04T17:53:35.785 DEBUG:boto:Token: None 2024-03-04T17:53:35.785 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.785 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bba89a1d059c310f-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.790 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.791 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.791 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.791 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.791 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.791 DEBUG:boto:Method: GET 2024-03-04T17:53:35.791 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.791 DEBUG:boto:Data: 2024-03-04T17:53:35.792 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.792 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.792 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.792 DEBUG:boto:Params: {} 2024-03-04T17:53:35.792 DEBUG:boto:Token: None 2024-03-04T17:53:35.792 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.792 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa887c65a31ea1f3-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.797 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.797 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.797 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.798 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.798 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:35.798 DEBUG:boto:Data: 2024-03-04T17:53:35.798 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.798 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.798 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.798 DEBUG:boto:Params: {} 2024-03-04T17:53:35.798 DEBUG:boto:Token: None 2024-03-04T17:53:35.798 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj 2024-03-04T17:53:35.799 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI= 2024-03-04T17:53:35.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI=', 'Content-Length': '0'} 2024-03-04T17:53:35.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2454bca5dd2fa06-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.805 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.805 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.805 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.805 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:35.805 DEBUG:boto:Data: 2024-03-04T17:53:35.805 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.805 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.805 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.806 DEBUG:boto:Params: {} 2024-03-04T17:53:35.806 DEBUG:boto:Token: None 2024-03-04T17:53:35.806 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj 2024-03-04T17:53:35.806 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI= 2024-03-04T17:53:35.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:96N3LfzSBpqqWuZ3CJj+nlme8gI=', 'Content-Length': '0'} 2024-03-04T17:53:35.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008573f8d2ebdc3a6c-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:35.812 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.812 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.812 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.812 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.813 DEBUG:boto:Method: GET 2024-03-04T17:53:35.813 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.813 DEBUG:boto:Data: 2024-03-04T17:53:35.813 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.813 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.813 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.813 DEBUG:boto:Params: {} 2024-03-04T17:53:35.813 DEBUG:boto:Token: None 2024-03-04T17:53:35.813 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.814 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k= 2024-03-04T17:53:35.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k=', 'Content-Length': '0'} 2024-03-04T17:53:35.816 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000068c9f325a1ce2d1-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.860 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.860 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.860 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.860 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.860 DEBUG:boto:Method: GET 2024-03-04T17:53:35.861 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.861 DEBUG:boto:Data: 2024-03-04T17:53:35.861 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.861 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.861 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.861 DEBUG:boto:Params: {} 2024-03-04T17:53:35.861 DEBUG:boto:Token: None 2024-03-04T17:53:35.861 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:35.862 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k= 2024-03-04T17:53:35.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6eWIOsx//J9RpUQtwn27WV8rp+k=', 'Content-Length': '0'} 2024-03-04T17:53:35.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001279868206f5bc2d-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.908 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.908 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.908 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.908 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.908 DEBUG:boto:Method: GET 2024-03-04T17:53:35.909 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.909 DEBUG:boto:Data: 2024-03-04T17:53:35.909 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.909 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.909 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.909 DEBUG:boto:Params: {} 2024-03-04T17:53:35.909 DEBUG:boto:Token: None 2024-03-04T17:53:35.909 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.909 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076ed3275416478b2-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.915 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.915 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.915 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.915 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.915 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.915 DEBUG:boto:Method: GET 2024-03-04T17:53:35.916 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.916 DEBUG:boto:Data: 2024-03-04T17:53:35.916 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.916 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.916 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.916 DEBUG:boto:Params: {} 2024-03-04T17:53:35.916 DEBUG:boto:Token: None 2024-03-04T17:53:35.916 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.916 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:35.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031bd7aa8d68aa62b-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.921 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:35.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:35.921 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.921 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.922 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.922 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.922 DEBUG:boto:Method: GET 2024-03-04T17:53:35.922 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.922 DEBUG:boto:Data: 2024-03-04T17:53:35.922 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.922 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.922 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.922 DEBUG:boto:Params: {} 2024-03-04T17:53:35.922 DEBUG:boto:Token: None 2024-03-04T17:53:35.923 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.923 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BN0nfqVLOOcsNwsZbPYYVvBB4Ws= 2024-03-04T17:53:35.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BN0nfqVLOOcsNwsZbPYYVvBB4Ws=', 'Content-Length': '0'} 2024-03-04T17:53:35.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa1a967fe6b6d3ad-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.927 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.927 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.927 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.927 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.928 DEBUG:boto:Method: GET 2024-03-04T17:53:35.928 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.928 DEBUG:boto:Data: 2024-03-04T17:53:35.928 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.928 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.928 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.928 DEBUG:boto:Params: {} 2024-03-04T17:53:35.928 DEBUG:boto:Token: None 2024-03-04T17:53:35.928 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.928 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BN0nfqVLOOcsNwsZbPYYVvBB4Ws= 2024-03-04T17:53:35.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BN0nfqVLOOcsNwsZbPYYVvBB4Ws=', 'Content-Length': '0'} 2024-03-04T17:53:35.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9797cc1c7f9281c-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.932 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.932 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.932 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.932 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.933 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.933 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.933 DEBUG:boto:Data: 2024-03-04T17:53:35.933 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.933 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.933 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.933 DEBUG:boto:Params: {} 2024-03-04T17:53:35.933 DEBUG:boto:Token: None 2024-03-04T17:53:35.933 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.934 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YRwHSmVNjRIja8UhHI7G/xL6nk0= 2024-03-04T17:53:35.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YRwHSmVNjRIja8UhHI7G/xL6nk0=', 'Content-Length': '0'} 2024-03-04T17:53:35.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea80322ae2be441a-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.937 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.937 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.937 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.937 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.937 DEBUG:boto:Method: HEAD 2024-03-04T17:53:35.937 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.937 DEBUG:boto:Data: 2024-03-04T17:53:35.937 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.938 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.938 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.938 DEBUG:boto:Params: {} 2024-03-04T17:53:35.938 DEBUG:boto:Token: None 2024-03-04T17:53:35.938 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.938 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YRwHSmVNjRIja8UhHI7G/xL6nk0= 2024-03-04T17:53:35.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YRwHSmVNjRIja8UhHI7G/xL6nk0=', 'Content-Length': '0'} 2024-03-04T17:53:35.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c30c4351a807688-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:35.941 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.941 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.942 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.942 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.942 DEBUG:boto:Method: GET 2024-03-04T17:53:35.942 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.942 DEBUG:boto:Data: 2024-03-04T17:53:35.942 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.942 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.942 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.942 DEBUG:boto:Params: {} 2024-03-04T17:53:35.943 DEBUG:boto:Token: None 2024-03-04T17:53:35.943 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.943 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BN0nfqVLOOcsNwsZbPYYVvBB4Ws= 2024-03-04T17:53:35.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BN0nfqVLOOcsNwsZbPYYVvBB4Ws=', 'Content-Length': '0'} 2024-03-04T17:53:35.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f92e88c9d776ba1f-0065e60a9f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.988 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:35.988 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:35.988 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.988 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.988 DEBUG:boto:Method: GET 2024-03-04T17:53:35.989 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.989 DEBUG:boto:Data: 2024-03-04T17:53:35.989 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.989 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:35.989 DEBUG:boto:Port: 8000 2024-03-04T17:53:35.989 DEBUG:boto:Params: {} 2024-03-04T17:53:35.989 DEBUG:boto:Token: None 2024-03-04T17:53:35.989 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:35.989 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BN0nfqVLOOcsNwsZbPYYVvBB4Ws= 2024-03-04T17:53:35.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BN0nfqVLOOcsNwsZbPYYVvBB4Ws=', 'Content-Length': '0'} 2024-03-04T17:53:35.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e769ef7a1a34cff-0065e60a9f-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:35.993 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:35.993 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:35.993 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.993 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:35.994 DEBUG:boto:Method: GET 2024-03-04T17:53:35.994 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:35.994 DEBUG:boto:Data: 2024-03-04T17:53:35.994 DEBUG:boto:Headers: {} 2024-03-04T17:53:35.994 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:35.994 DEBUG:boto:Port: 8001 2024-03-04T17:53:35.994 DEBUG:boto:Params: {} 2024-03-04T17:53:35.994 DEBUG:boto:Token: None 2024-03-04T17:53:35.994 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:35 GMT /ppfahn-22/ 2024-03-04T17:53:35.995 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc= 2024-03-04T17:53:35.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUZkh+GRoKrG7j1DYy2uBGKeDyc=', 'Content-Length': '0'} 2024-03-04T17:53:36.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ffb55163b312c66-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.000 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.000 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.000 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.001 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.001 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.001 DEBUG:boto:Method: GET 2024-03-04T17:53:36.001 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.001 DEBUG:boto:Data: 2024-03-04T17:53:36.001 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.001 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.001 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.001 DEBUG:boto:Params: {} 2024-03-04T17:53:36.001 DEBUG:boto:Token: None 2024-03-04T17:53:36.002 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.002 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4b9953014ff6607-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.007 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.007 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.007 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.007 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.007 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:36.008 DEBUG:boto:Data: 2024-03-04T17:53:36.008 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.008 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.008 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.008 DEBUG:boto:Params: {} 2024-03-04T17:53:36.008 DEBUG:boto:Token: None 2024-03-04T17:53:36.008 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj 2024-03-04T17:53:36.008 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE= 2024-03-04T17:53:36.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE=', 'Content-Length': '0'} 2024-03-04T17:53:36.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024cd02108b51b350-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.015 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.015 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.015 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.015 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:36.015 DEBUG:boto:Data: 2024-03-04T17:53:36.015 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.015 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.015 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.015 DEBUG:boto:Params: {} 2024-03-04T17:53:36.015 DEBUG:boto:Token: None 2024-03-04T17:53:36.016 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj 2024-03-04T17:53:36.016 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE= 2024-03-04T17:53:36.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE=', 'Content-Length': '0'} 2024-03-04T17:53:36.022 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c0e0def2d32a8b1-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:36.023 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.023 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.023 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.023 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.023 DEBUG:boto:Method: GET 2024-03-04T17:53:36.023 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.023 DEBUG:boto:Data: 2024-03-04T17:53:36.023 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.023 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.023 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.024 DEBUG:boto:Params: {} 2024-03-04T17:53:36.024 DEBUG:boto:Token: None 2024-03-04T17:53:36.024 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.024 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM= 2024-03-04T17:53:36.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM=', 'Content-Length': '0'} 2024-03-04T17:53:36.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ea07275d4f5d5f3-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.068 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.068 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.068 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.068 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.068 DEBUG:boto:Method: GET 2024-03-04T17:53:36.069 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.069 DEBUG:boto:Data: 2024-03-04T17:53:36.069 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.069 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.069 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.069 DEBUG:boto:Params: {} 2024-03-04T17:53:36.069 DEBUG:boto:Token: None 2024-03-04T17:53:36.069 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.069 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM= 2024-03-04T17:53:36.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM=', 'Content-Length': '0'} 2024-03-04T17:53:36.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000121ba894e3434e4b-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.116 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.116 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.116 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.116 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.116 DEBUG:boto:Method: GET 2024-03-04T17:53:36.117 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.117 DEBUG:boto:Data: 2024-03-04T17:53:36.117 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.117 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.117 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.117 DEBUG:boto:Params: {} 2024-03-04T17:53:36.117 DEBUG:boto:Token: None 2024-03-04T17:53:36.117 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.117 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f851f64d38beabc0-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.123 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.123 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.123 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.123 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.124 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.124 DEBUG:boto:Method: GET 2024-03-04T17:53:36.124 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.124 DEBUG:boto:Data: 2024-03-04T17:53:36.124 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.124 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.124 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.124 DEBUG:boto:Params: {} 2024-03-04T17:53:36.124 DEBUG:boto:Token: None 2024-03-04T17:53:36.125 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.125 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006715913e559067e6-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.131 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:36.131 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.131 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.131 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.131 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.132 DEBUG:boto:Method: GET 2024-03-04T17:53:36.132 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.132 DEBUG:boto:Data: 2024-03-04T17:53:36.132 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.132 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.132 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.132 DEBUG:boto:Params: {} 2024-03-04T17:53:36.132 DEBUG:boto:Token: None 2024-03-04T17:53:36.132 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.133 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Z2MRRPFXge8im5yDPxc/z5DoQbI= 2024-03-04T17:53:36.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Z2MRRPFXge8im5yDPxc/z5DoQbI=', 'Content-Length': '0'} 2024-03-04T17:53:36.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009889c79598f799b8-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.136 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.136 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.136 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.136 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.136 DEBUG:boto:Method: GET 2024-03-04T17:53:36.136 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.137 DEBUG:boto:Data: 2024-03-04T17:53:36.137 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.137 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.137 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.137 DEBUG:boto:Params: {} 2024-03-04T17:53:36.137 DEBUG:boto:Token: None 2024-03-04T17:53:36.137 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.137 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Z2MRRPFXge8im5yDPxc/z5DoQbI= 2024-03-04T17:53:36.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Z2MRRPFXge8im5yDPxc/z5DoQbI=', 'Content-Length': '0'} 2024-03-04T17:53:36.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027d9ced0f9cb6796-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.188 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.188 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.188 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.188 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.188 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.188 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.189 DEBUG:boto:Data: 2024-03-04T17:53:36.189 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.189 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.189 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.189 DEBUG:boto:Params: {} 2024-03-04T17:53:36.189 DEBUG:boto:Token: None 2024-03-04T17:53:36.189 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.189 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tnU4tgju0pdKOEyJKX8MqYIhnKU= 2024-03-04T17:53:36.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tnU4tgju0pdKOEyJKX8MqYIhnKU=', 'Content-Length': '0'} 2024-03-04T17:53:36.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3dd5a180319bcbe-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.192 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.193 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.193 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.193 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.193 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.193 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.193 DEBUG:boto:Data: 2024-03-04T17:53:36.193 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.193 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.193 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.193 DEBUG:boto:Params: {} 2024-03-04T17:53:36.193 DEBUG:boto:Token: None 2024-03-04T17:53:36.194 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.194 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tnU4tgju0pdKOEyJKX8MqYIhnKU= 2024-03-04T17:53:36.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tnU4tgju0pdKOEyJKX8MqYIhnKU=', 'Content-Length': '0'} 2024-03-04T17:53:36.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008866688cd51c7590-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:36.197 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.197 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.197 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.197 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.197 DEBUG:boto:Method: GET 2024-03-04T17:53:36.197 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.197 DEBUG:boto:Data: 2024-03-04T17:53:36.198 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.198 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.198 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.198 DEBUG:boto:Params: {} 2024-03-04T17:53:36.198 DEBUG:boto:Token: None 2024-03-04T17:53:36.198 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.198 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Z2MRRPFXge8im5yDPxc/z5DoQbI= 2024-03-04T17:53:36.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Z2MRRPFXge8im5yDPxc/z5DoQbI=', 'Content-Length': '0'} 2024-03-04T17:53:36.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc0611f074f315aa-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.201 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.201 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.202 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.202 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.202 DEBUG:boto:Method: GET 2024-03-04T17:53:36.202 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.202 DEBUG:boto:Data: 2024-03-04T17:53:36.202 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.202 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.202 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.202 DEBUG:boto:Params: {} 2024-03-04T17:53:36.202 DEBUG:boto:Token: None 2024-03-04T17:53:36.203 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:36.203 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Z2MRRPFXge8im5yDPxc/z5DoQbI= 2024-03-04T17:53:36.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Z2MRRPFXge8im5yDPxc/z5DoQbI=', 'Content-Length': '0'} 2024-03-04T17:53:36.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d427c418acdaa2af-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.248 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.248 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.248 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.248 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.249 DEBUG:boto:Method: GET 2024-03-04T17:53:36.249 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.249 DEBUG:boto:Data: 2024-03-04T17:53:36.249 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.249 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.249 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.249 DEBUG:boto:Params: {} 2024-03-04T17:53:36.249 DEBUG:boto:Token: None 2024-03-04T17:53:36.249 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.250 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000484f54fd7f6ddc3-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.254 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.255 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.255 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.255 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.255 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.255 DEBUG:boto:Method: GET 2024-03-04T17:53:36.255 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.255 DEBUG:boto:Data: 2024-03-04T17:53:36.256 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.256 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.256 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.256 DEBUG:boto:Params: {} 2024-03-04T17:53:36.256 DEBUG:boto:Token: None 2024-03-04T17:53:36.256 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.256 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001305f3598b428d5-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.261 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.262 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.262 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.262 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.262 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:36.262 DEBUG:boto:Data: 2024-03-04T17:53:36.262 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.262 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.262 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.262 DEBUG:boto:Params: {} 2024-03-04T17:53:36.263 DEBUG:boto:Token: None 2024-03-04T17:53:36.263 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj 2024-03-04T17:53:36.263 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE= 2024-03-04T17:53:36.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE=', 'Content-Length': '0'} 2024-03-04T17:53:36.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb169f692820e4af-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.272 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.272 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.272 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.273 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:36.273 DEBUG:boto:Data: 2024-03-04T17:53:36.273 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.273 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.273 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.273 DEBUG:boto:Params: {} 2024-03-04T17:53:36.273 DEBUG:boto:Token: None 2024-03-04T17:53:36.273 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj 2024-03-04T17:53:36.273 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE= 2024-03-04T17:53:36.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE=', 'Content-Length': '0'} 2024-03-04T17:53:36.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b7630a06b0c8db6-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:36.279 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.279 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.280 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.280 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.280 DEBUG:boto:Method: GET 2024-03-04T17:53:36.280 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.280 DEBUG:boto:Data: 2024-03-04T17:53:36.280 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.280 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.280 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.280 DEBUG:boto:Params: {} 2024-03-04T17:53:36.280 DEBUG:boto:Token: None 2024-03-04T17:53:36.281 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.281 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM= 2024-03-04T17:53:36.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM=', 'Content-Length': '0'} 2024-03-04T17:53:36.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0d9270abdc3b35d-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.324 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.324 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.324 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.324 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.324 DEBUG:boto:Method: GET 2024-03-04T17:53:36.325 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.325 DEBUG:boto:Data: 2024-03-04T17:53:36.325 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.325 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.325 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.325 DEBUG:boto:Params: {} 2024-03-04T17:53:36.325 DEBUG:boto:Token: None 2024-03-04T17:53:36.325 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.325 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM= 2024-03-04T17:53:36.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM=', 'Content-Length': '0'} 2024-03-04T17:53:36.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000045fa3f7348d1088-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.372 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.372 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.372 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.372 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.373 DEBUG:boto:Method: GET 2024-03-04T17:53:36.373 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.373 DEBUG:boto:Data: 2024-03-04T17:53:36.373 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.373 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.373 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.373 DEBUG:boto:Params: {} 2024-03-04T17:53:36.373 DEBUG:boto:Token: None 2024-03-04T17:53:36.373 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.373 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005642489fda543d53-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.378 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.379 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.379 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.379 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.379 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.379 DEBUG:boto:Method: GET 2024-03-04T17:53:36.379 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.379 DEBUG:boto:Data: 2024-03-04T17:53:36.379 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.379 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.379 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.380 DEBUG:boto:Params: {} 2024-03-04T17:53:36.380 DEBUG:boto:Token: None 2024-03-04T17:53:36.380 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.380 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010d801d567611b24-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.385 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:36.385 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.385 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.386 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.386 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.386 DEBUG:boto:Method: GET 2024-03-04T17:53:36.386 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.386 DEBUG:boto:Data: 2024-03-04T17:53:36.386 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.386 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.386 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.386 DEBUG:boto:Params: {} 2024-03-04T17:53:36.386 DEBUG:boto:Token: None 2024-03-04T17:53:36.387 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.387 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ajOGBrhLCG8qcCGCwFidS39hdxI= 2024-03-04T17:53:36.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ajOGBrhLCG8qcCGCwFidS39hdxI=', 'Content-Length': '0'} 2024-03-04T17:53:36.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cb3bb61e4d90812-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.390 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.390 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.390 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.390 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.390 DEBUG:boto:Method: GET 2024-03-04T17:53:36.390 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.390 DEBUG:boto:Data: 2024-03-04T17:53:36.390 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.391 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.391 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.391 DEBUG:boto:Params: {} 2024-03-04T17:53:36.391 DEBUG:boto:Token: None 2024-03-04T17:53:36.391 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.391 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ajOGBrhLCG8qcCGCwFidS39hdxI= 2024-03-04T17:53:36.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ajOGBrhLCG8qcCGCwFidS39hdxI=', 'Content-Length': '0'} 2024-03-04T17:53:36.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3ea9ba64f7a9b7d-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.394 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.395 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.395 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.395 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.395 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.395 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.395 DEBUG:boto:Data: 2024-03-04T17:53:36.395 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.395 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.395 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.396 DEBUG:boto:Params: {} 2024-03-04T17:53:36.396 DEBUG:boto:Token: None 2024-03-04T17:53:36.396 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.396 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aMDJbhHoK1nVI8oV2WTzrCuW8uQ= 2024-03-04T17:53:36.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aMDJbhHoK1nVI8oV2WTzrCuW8uQ=', 'Content-Length': '0'} 2024-03-04T17:53:36.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6472d5ebccdd4af-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.399 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.399 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.399 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.400 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.400 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.400 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.400 DEBUG:boto:Data: 2024-03-04T17:53:36.400 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.400 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.400 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.400 DEBUG:boto:Params: {} 2024-03-04T17:53:36.400 DEBUG:boto:Token: None 2024-03-04T17:53:36.401 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.401 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aMDJbhHoK1nVI8oV2WTzrCuW8uQ= 2024-03-04T17:53:36.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aMDJbhHoK1nVI8oV2WTzrCuW8uQ=', 'Content-Length': '0'} 2024-03-04T17:53:36.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bde9eee69ae213c4-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:36.404 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.404 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.404 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.404 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.404 DEBUG:boto:Method: GET 2024-03-04T17:53:36.404 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.404 DEBUG:boto:Data: 2024-03-04T17:53:36.404 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.405 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.405 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.405 DEBUG:boto:Params: {} 2024-03-04T17:53:36.405 DEBUG:boto:Token: None 2024-03-04T17:53:36.405 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.405 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ajOGBrhLCG8qcCGCwFidS39hdxI= 2024-03-04T17:53:36.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ajOGBrhLCG8qcCGCwFidS39hdxI=', 'Content-Length': '0'} 2024-03-04T17:53:36.408 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000df4f5686263b9ac-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.408 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.408 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.409 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.409 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.409 DEBUG:boto:Method: GET 2024-03-04T17:53:36.409 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.409 DEBUG:boto:Data: 2024-03-04T17:53:36.409 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.409 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.409 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.409 DEBUG:boto:Params: {} 2024-03-04T17:53:36.409 DEBUG:boto:Token: None 2024-03-04T17:53:36.410 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:36.410 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ajOGBrhLCG8qcCGCwFidS39hdxI= 2024-03-04T17:53:36.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ajOGBrhLCG8qcCGCwFidS39hdxI=', 'Content-Length': '0'} 2024-03-04T17:53:36.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000914d9757b4e1e660-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.456 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.456 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.456 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.456 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.456 DEBUG:boto:Method: GET 2024-03-04T17:53:36.457 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.457 DEBUG:boto:Data: 2024-03-04T17:53:36.457 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.457 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.457 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.457 DEBUG:boto:Params: {} 2024-03-04T17:53:36.457 DEBUG:boto:Token: None 2024-03-04T17:53:36.457 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.458 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4f4ccae33838ecf-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.463 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.463 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.463 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.464 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.464 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.464 DEBUG:boto:Method: GET 2024-03-04T17:53:36.464 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.464 DEBUG:boto:Data: 2024-03-04T17:53:36.464 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.464 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.464 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.464 DEBUG:boto:Params: {} 2024-03-04T17:53:36.465 DEBUG:boto:Token: None 2024-03-04T17:53:36.465 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.465 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066d74186e0c2761b-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.470 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.471 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.471 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.471 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.471 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:36.471 DEBUG:boto:Data: 2024-03-04T17:53:36.471 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.471 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.471 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.472 DEBUG:boto:Params: {} 2024-03-04T17:53:36.472 DEBUG:boto:Token: None 2024-03-04T17:53:36.472 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj 2024-03-04T17:53:36.472 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE= 2024-03-04T17:53:36.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE=', 'Content-Length': '0'} 2024-03-04T17:53:36.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbd489192d9acdca-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.478 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.478 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.478 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.478 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:36.479 DEBUG:boto:Data: 2024-03-04T17:53:36.479 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.479 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.479 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.479 DEBUG:boto:Params: {} 2024-03-04T17:53:36.479 DEBUG:boto:Token: None 2024-03-04T17:53:36.479 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj 2024-03-04T17:53:36.479 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE= 2024-03-04T17:53:36.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE=', 'Content-Length': '0'} 2024-03-04T17:53:36.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052d57cba308a9753-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:36.486 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.486 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.486 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.486 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.486 DEBUG:boto:Method: GET 2024-03-04T17:53:36.486 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.486 DEBUG:boto:Data: 2024-03-04T17:53:36.486 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.486 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.487 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.487 DEBUG:boto:Params: {} 2024-03-04T17:53:36.487 DEBUG:boto:Token: None 2024-03-04T17:53:36.487 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.487 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM= 2024-03-04T17:53:36.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM=', 'Content-Length': '0'} 2024-03-04T17:53:36.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f869edbc0349bd49-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.532 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.532 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.532 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.532 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.533 DEBUG:boto:Method: GET 2024-03-04T17:53:36.533 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.533 DEBUG:boto:Data: 2024-03-04T17:53:36.533 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.533 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.533 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.533 DEBUG:boto:Params: {} 2024-03-04T17:53:36.533 DEBUG:boto:Token: None 2024-03-04T17:53:36.533 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.534 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM= 2024-03-04T17:53:36.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM=', 'Content-Length': '0'} 2024-03-04T17:53:36.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6e19b2b6fa17990-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.580 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.580 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.580 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.580 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.580 DEBUG:boto:Method: GET 2024-03-04T17:53:36.580 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.581 DEBUG:boto:Data: 2024-03-04T17:53:36.581 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.581 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.581 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.581 DEBUG:boto:Params: {} 2024-03-04T17:53:36.581 DEBUG:boto:Token: None 2024-03-04T17:53:36.581 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.581 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016d8341a3d28f83e-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.587 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.587 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.587 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.587 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.588 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.588 DEBUG:boto:Method: GET 2024-03-04T17:53:36.588 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.588 DEBUG:boto:Data: 2024-03-04T17:53:36.588 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.588 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.588 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.588 DEBUG:boto:Params: {} 2024-03-04T17:53:36.588 DEBUG:boto:Token: None 2024-03-04T17:53:36.589 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.589 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031eb8928754615b5-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.594 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:36.594 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.594 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.594 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.594 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.594 DEBUG:boto:Method: GET 2024-03-04T17:53:36.595 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.595 DEBUG:boto:Data: 2024-03-04T17:53:36.595 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.595 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.595 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.595 DEBUG:boto:Params: {} 2024-03-04T17:53:36.595 DEBUG:boto:Token: None 2024-03-04T17:53:36.595 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.595 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CqHdWpFAN1yXf6w0Q/6eS3oOHUk= 2024-03-04T17:53:36.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CqHdWpFAN1yXf6w0Q/6eS3oOHUk=', 'Content-Length': '0'} 2024-03-04T17:53:36.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc67a0df9d4a4142-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.599 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.599 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.599 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.599 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.599 DEBUG:boto:Method: GET 2024-03-04T17:53:36.600 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.600 DEBUG:boto:Data: 2024-03-04T17:53:36.600 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.600 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.600 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.600 DEBUG:boto:Params: {} 2024-03-04T17:53:36.600 DEBUG:boto:Token: None 2024-03-04T17:53:36.600 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.600 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CqHdWpFAN1yXf6w0Q/6eS3oOHUk= 2024-03-04T17:53:36.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CqHdWpFAN1yXf6w0Q/6eS3oOHUk=', 'Content-Length': '0'} 2024-03-04T17:53:36.603 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017f19ce58f21250c-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.644 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.644 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.644 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.644 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.644 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.644 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.645 DEBUG:boto:Data: 2024-03-04T17:53:36.645 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.645 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.645 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.645 DEBUG:boto:Params: {} 2024-03-04T17:53:36.645 DEBUG:boto:Token: None 2024-03-04T17:53:36.645 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.645 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mfX931eN1JAp3XECn2v5WCjsSBk= 2024-03-04T17:53:36.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mfX931eN1JAp3XECn2v5WCjsSBk=', 'Content-Length': '0'} 2024-03-04T17:53:36.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d3daade02456b00-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.648 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.649 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.649 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.649 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.649 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.649 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.649 DEBUG:boto:Data: 2024-03-04T17:53:36.649 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.649 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.649 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.649 DEBUG:boto:Params: {} 2024-03-04T17:53:36.650 DEBUG:boto:Token: None 2024-03-04T17:53:36.650 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.650 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mfX931eN1JAp3XECn2v5WCjsSBk= 2024-03-04T17:53:36.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mfX931eN1JAp3XECn2v5WCjsSBk=', 'Content-Length': '0'} 2024-03-04T17:53:36.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f948b4d4c3911668-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:36.653 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.653 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.653 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.654 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.654 DEBUG:boto:Method: GET 2024-03-04T17:53:36.654 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.654 DEBUG:boto:Data: 2024-03-04T17:53:36.654 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.654 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.654 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.654 DEBUG:boto:Params: {} 2024-03-04T17:53:36.654 DEBUG:boto:Token: None 2024-03-04T17:53:36.655 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.655 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CqHdWpFAN1yXf6w0Q/6eS3oOHUk= 2024-03-04T17:53:36.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CqHdWpFAN1yXf6w0Q/6eS3oOHUk=', 'Content-Length': '0'} 2024-03-04T17:53:36.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bdf10f6b429aaa3-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.658 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.658 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.658 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.658 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.658 DEBUG:boto:Method: GET 2024-03-04T17:53:36.659 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.659 DEBUG:boto:Data: 2024-03-04T17:53:36.659 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.659 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.659 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.659 DEBUG:boto:Params: {} 2024-03-04T17:53:36.659 DEBUG:boto:Token: None 2024-03-04T17:53:36.659 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:36.659 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CqHdWpFAN1yXf6w0Q/6eS3oOHUk= 2024-03-04T17:53:36.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CqHdWpFAN1yXf6w0Q/6eS3oOHUk=', 'Content-Length': '0'} 2024-03-04T17:53:36.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e462952076f452e-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.704 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.704 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.704 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.704 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.704 DEBUG:boto:Method: GET 2024-03-04T17:53:36.704 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.705 DEBUG:boto:Data: 2024-03-04T17:53:36.705 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.705 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.705 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.705 DEBUG:boto:Params: {} 2024-03-04T17:53:36.705 DEBUG:boto:Token: None 2024-03-04T17:53:36.705 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.705 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4a5ead5df24ce78-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.710 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.711 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.711 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.711 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.711 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.711 DEBUG:boto:Method: GET 2024-03-04T17:53:36.711 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.711 DEBUG:boto:Data: 2024-03-04T17:53:36.711 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.711 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.711 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.712 DEBUG:boto:Params: {} 2024-03-04T17:53:36.712 DEBUG:boto:Token: None 2024-03-04T17:53:36.712 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.712 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007a140a7b8b2b064-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.717 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.717 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.717 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.717 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.717 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:36.717 DEBUG:boto:Data: 2024-03-04T17:53:36.718 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.718 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.718 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.718 DEBUG:boto:Params: {} 2024-03-04T17:53:36.718 DEBUG:boto:Token: None 2024-03-04T17:53:36.718 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj 2024-03-04T17:53:36.718 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE= 2024-03-04T17:53:36.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE=', 'Content-Length': '0'} 2024-03-04T17:53:36.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1c7fa33d8da04a4-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.725 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.725 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.725 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.725 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:36.725 DEBUG:boto:Data: 2024-03-04T17:53:36.725 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.725 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.725 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.725 DEBUG:boto:Params: {} 2024-03-04T17:53:36.725 DEBUG:boto:Token: None 2024-03-04T17:53:36.726 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj 2024-03-04T17:53:36.726 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE= 2024-03-04T17:53:36.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE=', 'Content-Length': '0'} 2024-03-04T17:53:36.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000127f2ee6edc41865-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:36.732 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.732 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.732 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.732 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.732 DEBUG:boto:Method: GET 2024-03-04T17:53:36.733 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.733 DEBUG:boto:Data: 2024-03-04T17:53:36.733 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.733 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.733 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.733 DEBUG:boto:Params: {} 2024-03-04T17:53:36.733 DEBUG:boto:Token: None 2024-03-04T17:53:36.733 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.734 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM= 2024-03-04T17:53:36.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM=', 'Content-Length': '0'} 2024-03-04T17:53:36.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000919320285e297b-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.780 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.780 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.780 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.780 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.780 DEBUG:boto:Method: GET 2024-03-04T17:53:36.781 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.781 DEBUG:boto:Data: 2024-03-04T17:53:36.781 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.781 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.781 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.781 DEBUG:boto:Params: {} 2024-03-04T17:53:36.781 DEBUG:boto:Token: None 2024-03-04T17:53:36.781 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.781 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM= 2024-03-04T17:53:36.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM=', 'Content-Length': '0'} 2024-03-04T17:53:36.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007aea3d19e4fbbdd6-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.828 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.828 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.828 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.828 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.828 DEBUG:boto:Method: GET 2024-03-04T17:53:36.829 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.829 DEBUG:boto:Data: 2024-03-04T17:53:36.829 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.829 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.829 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.829 DEBUG:boto:Params: {} 2024-03-04T17:53:36.829 DEBUG:boto:Token: None 2024-03-04T17:53:36.829 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.829 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045c39686409162e3-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.834 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.835 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.835 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.835 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.835 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.835 DEBUG:boto:Method: GET 2024-03-04T17:53:36.835 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.835 DEBUG:boto:Data: 2024-03-04T17:53:36.835 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.835 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.836 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.836 DEBUG:boto:Params: {} 2024-03-04T17:53:36.836 DEBUG:boto:Token: None 2024-03-04T17:53:36.836 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.836 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1a6c410b08ce846-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.841 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:36.841 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.842 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.842 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.842 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.842 DEBUG:boto:Method: GET 2024-03-04T17:53:36.842 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.842 DEBUG:boto:Data: 2024-03-04T17:53:36.842 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.842 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.842 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.842 DEBUG:boto:Params: {} 2024-03-04T17:53:36.843 DEBUG:boto:Token: None 2024-03-04T17:53:36.843 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.843 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KPzQKfxoKS/fA8NUGPuhp//3/Vg= 2024-03-04T17:53:36.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KPzQKfxoKS/fA8NUGPuhp//3/Vg=', 'Content-Length': '0'} 2024-03-04T17:53:36.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007083eb258777e4fe-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.846 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.846 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.847 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.847 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.847 DEBUG:boto:Method: GET 2024-03-04T17:53:36.847 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.847 DEBUG:boto:Data: 2024-03-04T17:53:36.847 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.847 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.847 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.847 DEBUG:boto:Params: {} 2024-03-04T17:53:36.847 DEBUG:boto:Token: None 2024-03-04T17:53:36.848 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.848 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KPzQKfxoKS/fA8NUGPuhp//3/Vg= 2024-03-04T17:53:36.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KPzQKfxoKS/fA8NUGPuhp//3/Vg=', 'Content-Length': '0'} 2024-03-04T17:53:36.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c9360686cf01923-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.892 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.892 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.892 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.892 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.892 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.892 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.893 DEBUG:boto:Data: 2024-03-04T17:53:36.893 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.893 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.893 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.893 DEBUG:boto:Params: {} 2024-03-04T17:53:36.893 DEBUG:boto:Token: None 2024-03-04T17:53:36.893 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.893 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8xuSCMx1JzuE3PgnhGSbpogYKG4= 2024-03-04T17:53:36.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8xuSCMx1JzuE3PgnhGSbpogYKG4=', 'Content-Length': '0'} 2024-03-04T17:53:36.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a45ad865ff0ab09-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.897 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.898 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.898 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.898 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.898 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.898 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.898 DEBUG:boto:Data: 2024-03-04T17:53:36.898 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.898 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.898 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.898 DEBUG:boto:Params: {} 2024-03-04T17:53:36.898 DEBUG:boto:Token: None 2024-03-04T17:53:36.899 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.899 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8xuSCMx1JzuE3PgnhGSbpogYKG4= 2024-03-04T17:53:36.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8xuSCMx1JzuE3PgnhGSbpogYKG4=', 'Content-Length': '0'} 2024-03-04T17:53:36.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044d58fefae6476ee-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:36.902 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.902 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.902 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.902 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.902 DEBUG:boto:Method: GET 2024-03-04T17:53:36.902 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.903 DEBUG:boto:Data: 2024-03-04T17:53:36.903 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.903 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.903 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.903 DEBUG:boto:Params: {} 2024-03-04T17:53:36.903 DEBUG:boto:Token: None 2024-03-04T17:53:36.903 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.903 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KPzQKfxoKS/fA8NUGPuhp//3/Vg= 2024-03-04T17:53:36.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KPzQKfxoKS/fA8NUGPuhp//3/Vg=', 'Content-Length': '0'} 2024-03-04T17:53:36.906 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc5985bfe1554b99-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.907 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.907 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.907 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.907 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.907 DEBUG:boto:Method: GET 2024-03-04T17:53:36.907 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.907 DEBUG:boto:Data: 2024-03-04T17:53:36.907 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.907 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.907 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.907 DEBUG:boto:Params: {} 2024-03-04T17:53:36.908 DEBUG:boto:Token: None 2024-03-04T17:53:36.908 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:36.908 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KPzQKfxoKS/fA8NUGPuhp//3/Vg= 2024-03-04T17:53:36.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KPzQKfxoKS/fA8NUGPuhp//3/Vg=', 'Content-Length': '0'} 2024-03-04T17:53:36.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000972b1e23c2e58902-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.952 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.952 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.952 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.952 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.952 DEBUG:boto:Method: GET 2024-03-04T17:53:36.953 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.953 DEBUG:boto:Data: 2024-03-04T17:53:36.953 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.953 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.953 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.953 DEBUG:boto:Params: {} 2024-03-04T17:53:36.953 DEBUG:boto:Token: None 2024-03-04T17:53:36.953 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.953 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfd788ccbf51443a-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.958 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.958 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:36.958 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:36.959 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.959 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:36.959 DEBUG:boto:Method: GET 2024-03-04T17:53:36.959 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:36.959 DEBUG:boto:Data: 2024-03-04T17:53:36.959 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.959 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.959 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.959 DEBUG:boto:Params: {} 2024-03-04T17:53:36.959 DEBUG:boto:Token: None 2024-03-04T17:53:36.960 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/ 2024-03-04T17:53:36.960 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM= 2024-03-04T17:53:36.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A://t0P7QJ33EC2HptTP1Oe8XeRgM=', 'Content-Length': '0'} 2024-03-04T17:53:36.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ede7ec27f2e681ba-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.965 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:36.965 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.965 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.965 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.965 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:36.966 DEBUG:boto:Data: 2024-03-04T17:53:36.966 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.966 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.966 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.966 DEBUG:boto:Params: {} 2024-03-04T17:53:36.966 DEBUG:boto:Token: None 2024-03-04T17:53:36.966 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj 2024-03-04T17:53:36.966 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE= 2024-03-04T17:53:36.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE=', 'Content-Length': '0'} 2024-03-04T17:53:36.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc8bd2700d1df98e-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.973 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.973 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.973 DEBUG:boto:Method: HEAD 2024-03-04T17:53:36.973 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:36.973 DEBUG:boto:Data: 2024-03-04T17:53:36.973 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.973 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:36.973 DEBUG:boto:Port: 8000 2024-03-04T17:53:36.973 DEBUG:boto:Params: {} 2024-03-04T17:53:36.973 DEBUG:boto:Token: None 2024-03-04T17:53:36.974 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj 2024-03-04T17:53:36.974 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE= 2024-03-04T17:53:36.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7edAWEfsBOuW+vQQ2LrteEJu2OE=', 'Content-Length': '0'} 2024-03-04T17:53:36.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ea0a5862f252473-0065e60aa0-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:36.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:36.980 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:36.980 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:36.980 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.980 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.980 DEBUG:boto:Method: GET 2024-03-04T17:53:36.980 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.980 DEBUG:boto:Data: 2024-03-04T17:53:36.981 DEBUG:boto:Headers: {} 2024-03-04T17:53:36.981 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:36.981 DEBUG:boto:Port: 8001 2024-03-04T17:53:36.981 DEBUG:boto:Params: {} 2024-03-04T17:53:36.981 DEBUG:boto:Token: None 2024-03-04T17:53:36.981 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:36 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:36.981 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM= 2024-03-04T17:53:36.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vCxgFz9Cgp9WBgJ9+ZXozCzPvOM=', 'Content-Length': '0'} 2024-03-04T17:53:36.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8ec95a004f7f1b7-0065e60aa0-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.028 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.028 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.028 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.028 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.029 DEBUG:boto:Method: GET 2024-03-04T17:53:37.029 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.029 DEBUG:boto:Data: 2024-03-04T17:53:37.029 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.029 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.029 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.029 DEBUG:boto:Params: {} 2024-03-04T17:53:37.029 DEBUG:boto:Token: None 2024-03-04T17:53:37.029 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.029 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY= 2024-03-04T17:53:37.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY=', 'Content-Length': '0'} 2024-03-04T17:53:37.032 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c3fd1fa52e84db9-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.076 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.076 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.076 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.076 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.076 DEBUG:boto:Method: GET 2024-03-04T17:53:37.077 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.077 DEBUG:boto:Data: 2024-03-04T17:53:37.077 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.077 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.077 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.077 DEBUG:boto:Params: {} 2024-03-04T17:53:37.077 DEBUG:boto:Token: None 2024-03-04T17:53:37.077 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.077 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb7b8914e8df3329-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.083 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.083 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.083 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.083 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.083 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.083 DEBUG:boto:Method: GET 2024-03-04T17:53:37.084 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.084 DEBUG:boto:Data: 2024-03-04T17:53:37.084 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.084 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.084 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.084 DEBUG:boto:Params: {} 2024-03-04T17:53:37.084 DEBUG:boto:Token: None 2024-03-04T17:53:37.084 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.084 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a6a0563e3691e3e-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.090 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:37.090 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.090 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.090 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.090 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.091 DEBUG:boto:Method: GET 2024-03-04T17:53:37.091 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.091 DEBUG:boto:Data: 2024-03-04T17:53:37.091 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.091 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.091 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.091 DEBUG:boto:Params: {} 2024-03-04T17:53:37.091 DEBUG:boto:Token: None 2024-03-04T17:53:37.091 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.091 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A081h8AhjbSm+9oJlBoIUPxniV8= 2024-03-04T17:53:37.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A081h8AhjbSm+9oJlBoIUPxniV8=', 'Content-Length': '0'} 2024-03-04T17:53:37.095 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edb001ac45bf45a8-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.095 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.095 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.095 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.095 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.095 DEBUG:boto:Method: GET 2024-03-04T17:53:37.096 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.096 DEBUG:boto:Data: 2024-03-04T17:53:37.096 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.096 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.096 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.096 DEBUG:boto:Params: {} 2024-03-04T17:53:37.096 DEBUG:boto:Token: None 2024-03-04T17:53:37.096 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.097 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A081h8AhjbSm+9oJlBoIUPxniV8= 2024-03-04T17:53:37.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A081h8AhjbSm+9oJlBoIUPxniV8=', 'Content-Length': '0'} 2024-03-04T17:53:37.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064e98482aa50c4f9-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.140 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.140 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.140 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.140 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.140 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.141 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.141 DEBUG:boto:Data: 2024-03-04T17:53:37.141 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.141 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.141 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.141 DEBUG:boto:Params: {} 2024-03-04T17:53:37.141 DEBUG:boto:Token: None 2024-03-04T17:53:37.141 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.141 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/CaBaXAJ8pbZ9wXDg57cUN0PTDY= 2024-03-04T17:53:37.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/CaBaXAJ8pbZ9wXDg57cUN0PTDY=', 'Content-Length': '0'} 2024-03-04T17:53:37.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032f1093891654b61-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.144 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.145 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.145 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.145 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.145 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.145 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.145 DEBUG:boto:Data: 2024-03-04T17:53:37.145 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.145 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.145 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.145 DEBUG:boto:Params: {} 2024-03-04T17:53:37.145 DEBUG:boto:Token: None 2024-03-04T17:53:37.146 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.146 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/CaBaXAJ8pbZ9wXDg57cUN0PTDY= 2024-03-04T17:53:37.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/CaBaXAJ8pbZ9wXDg57cUN0PTDY=', 'Content-Length': '0'} 2024-03-04T17:53:37.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d59fcd3b863a314f-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:37.149 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.149 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.149 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.149 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.149 DEBUG:boto:Method: GET 2024-03-04T17:53:37.150 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.150 DEBUG:boto:Data: 2024-03-04T17:53:37.150 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.150 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.150 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.150 DEBUG:boto:Params: {} 2024-03-04T17:53:37.150 DEBUG:boto:Token: None 2024-03-04T17:53:37.150 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.150 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A081h8AhjbSm+9oJlBoIUPxniV8= 2024-03-04T17:53:37.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A081h8AhjbSm+9oJlBoIUPxniV8=', 'Content-Length': '0'} 2024-03-04T17:53:37.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd77c4d3ce5ad4f0-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.154 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.154 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.154 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.154 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.154 DEBUG:boto:Method: GET 2024-03-04T17:53:37.154 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.154 DEBUG:boto:Data: 2024-03-04T17:53:37.154 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.154 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.155 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.155 DEBUG:boto:Params: {} 2024-03-04T17:53:37.155 DEBUG:boto:Token: None 2024-03-04T17:53:37.155 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:37.155 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A081h8AhjbSm+9oJlBoIUPxniV8= 2024-03-04T17:53:37.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A081h8AhjbSm+9oJlBoIUPxniV8=', 'Content-Length': '0'} 2024-03-04T17:53:37.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000300995deca772408-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.200 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.200 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.200 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.200 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.200 DEBUG:boto:Method: GET 2024-03-04T17:53:37.200 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.201 DEBUG:boto:Data: 2024-03-04T17:53:37.201 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.201 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.201 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.201 DEBUG:boto:Params: {} 2024-03-04T17:53:37.201 DEBUG:boto:Token: None 2024-03-04T17:53:37.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.201 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f712f8f420ff8f2-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.207 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.207 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.207 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.207 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.207 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.207 DEBUG:boto:Method: GET 2024-03-04T17:53:37.208 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.208 DEBUG:boto:Data: 2024-03-04T17:53:37.208 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.208 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.208 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.208 DEBUG:boto:Params: {} 2024-03-04T17:53:37.208 DEBUG:boto:Token: None 2024-03-04T17:53:37.208 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.208 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d2a10d874ca14d5-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.213 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.214 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.214 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.214 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.214 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:37.214 DEBUG:boto:Data: 2024-03-04T17:53:37.214 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.214 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.214 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.214 DEBUG:boto:Params: {} 2024-03-04T17:53:37.215 DEBUG:boto:Token: None 2024-03-04T17:53:37.215 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj 2024-03-04T17:53:37.215 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU= 2024-03-04T17:53:37.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU=', 'Content-Length': '0'} 2024-03-04T17:53:37.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb572e4c43e754f7-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.221 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.221 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.221 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.221 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:37.221 DEBUG:boto:Data: 2024-03-04T17:53:37.222 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.222 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.222 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.222 DEBUG:boto:Params: {} 2024-03-04T17:53:37.222 DEBUG:boto:Token: None 2024-03-04T17:53:37.222 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj 2024-03-04T17:53:37.222 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU= 2024-03-04T17:53:37.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU=', 'Content-Length': '0'} 2024-03-04T17:53:37.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001daff6779ffd06ca-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:37.229 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.229 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.229 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.229 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.229 DEBUG:boto:Method: GET 2024-03-04T17:53:37.229 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.229 DEBUG:boto:Data: 2024-03-04T17:53:37.229 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.230 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.230 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.230 DEBUG:boto:Params: {} 2024-03-04T17:53:37.230 DEBUG:boto:Token: None 2024-03-04T17:53:37.230 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.230 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY= 2024-03-04T17:53:37.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY=', 'Content-Length': '0'} 2024-03-04T17:53:37.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004744ef0b604109e0-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.280 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.280 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.280 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.280 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.280 DEBUG:boto:Method: GET 2024-03-04T17:53:37.281 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.281 DEBUG:boto:Data: 2024-03-04T17:53:37.281 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.281 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.281 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.281 DEBUG:boto:Params: {} 2024-03-04T17:53:37.281 DEBUG:boto:Token: None 2024-03-04T17:53:37.281 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.281 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY= 2024-03-04T17:53:37.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY=', 'Content-Length': '0'} 2024-03-04T17:53:37.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000821365d686ba8478-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.328 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.328 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.328 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.328 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.328 DEBUG:boto:Method: GET 2024-03-04T17:53:37.329 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.329 DEBUG:boto:Data: 2024-03-04T17:53:37.329 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.329 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.329 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.329 DEBUG:boto:Params: {} 2024-03-04T17:53:37.329 DEBUG:boto:Token: None 2024-03-04T17:53:37.329 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.329 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003db19e2b3948bda7-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.335 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.335 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.336 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.336 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.336 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.336 DEBUG:boto:Method: GET 2024-03-04T17:53:37.336 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.336 DEBUG:boto:Data: 2024-03-04T17:53:37.336 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.336 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.336 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.336 DEBUG:boto:Params: {} 2024-03-04T17:53:37.337 DEBUG:boto:Token: None 2024-03-04T17:53:37.337 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.337 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006149e5926fd7be3c-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.342 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:37.343 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.343 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.343 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.343 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.343 DEBUG:boto:Method: GET 2024-03-04T17:53:37.343 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.343 DEBUG:boto:Data: 2024-03-04T17:53:37.343 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.344 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.344 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.344 DEBUG:boto:Params: {} 2024-03-04T17:53:37.344 DEBUG:boto:Token: None 2024-03-04T17:53:37.344 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.344 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qgDXLL6Dce6kidpuRJU/00m5YCk= 2024-03-04T17:53:37.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qgDXLL6Dce6kidpuRJU/00m5YCk=', 'Content-Length': '0'} 2024-03-04T17:53:37.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d16eb0f22e94078-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.347 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.348 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.348 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.348 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.348 DEBUG:boto:Method: GET 2024-03-04T17:53:37.348 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.348 DEBUG:boto:Data: 2024-03-04T17:53:37.348 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.348 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.348 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.348 DEBUG:boto:Params: {} 2024-03-04T17:53:37.348 DEBUG:boto:Token: None 2024-03-04T17:53:37.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.349 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qgDXLL6Dce6kidpuRJU/00m5YCk= 2024-03-04T17:53:37.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qgDXLL6Dce6kidpuRJU/00m5YCk=', 'Content-Length': '0'} 2024-03-04T17:53:37.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e341a7ec4a0c178-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.352 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.352 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.352 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.353 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.353 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.353 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.353 DEBUG:boto:Data: 2024-03-04T17:53:37.353 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.353 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.353 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.353 DEBUG:boto:Params: {} 2024-03-04T17:53:37.353 DEBUG:boto:Token: None 2024-03-04T17:53:37.353 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.354 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:L9LuzG8OVVgcio2xE8Mmm6eAuR0= 2024-03-04T17:53:37.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:L9LuzG8OVVgcio2xE8Mmm6eAuR0=', 'Content-Length': '0'} 2024-03-04T17:53:37.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f418eb454fa61db-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.356 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.357 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.357 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.357 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.357 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.357 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.357 DEBUG:boto:Data: 2024-03-04T17:53:37.357 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.357 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.357 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.357 DEBUG:boto:Params: {} 2024-03-04T17:53:37.358 DEBUG:boto:Token: None 2024-03-04T17:53:37.358 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.358 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:L9LuzG8OVVgcio2xE8Mmm6eAuR0= 2024-03-04T17:53:37.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:L9LuzG8OVVgcio2xE8Mmm6eAuR0=', 'Content-Length': '0'} 2024-03-04T17:53:37.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018d69010f9beb30a-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:37.362 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.362 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.362 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.363 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.363 DEBUG:boto:Method: GET 2024-03-04T17:53:37.363 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.363 DEBUG:boto:Data: 2024-03-04T17:53:37.363 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.363 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.363 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.363 DEBUG:boto:Params: {} 2024-03-04T17:53:37.363 DEBUG:boto:Token: None 2024-03-04T17:53:37.363 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.364 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qgDXLL6Dce6kidpuRJU/00m5YCk= 2024-03-04T17:53:37.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qgDXLL6Dce6kidpuRJU/00m5YCk=', 'Content-Length': '0'} 2024-03-04T17:53:37.366 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087c0b029d7da7f48-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.367 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.367 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.367 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.367 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.367 DEBUG:boto:Method: GET 2024-03-04T17:53:37.367 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.367 DEBUG:boto:Data: 2024-03-04T17:53:37.367 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.368 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.368 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.368 DEBUG:boto:Params: {} 2024-03-04T17:53:37.368 DEBUG:boto:Token: None 2024-03-04T17:53:37.368 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:37.368 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qgDXLL6Dce6kidpuRJU/00m5YCk= 2024-03-04T17:53:37.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qgDXLL6Dce6kidpuRJU/00m5YCk=', 'Content-Length': '0'} 2024-03-04T17:53:37.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081cfbfecd545d08e-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.412 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.412 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.412 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.412 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.412 DEBUG:boto:Method: GET 2024-03-04T17:53:37.413 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.413 DEBUG:boto:Data: 2024-03-04T17:53:37.413 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.413 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.413 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.413 DEBUG:boto:Params: {} 2024-03-04T17:53:37.413 DEBUG:boto:Token: None 2024-03-04T17:53:37.413 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.413 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000196971144d588924-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.418 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.418 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.419 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.419 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.419 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.419 DEBUG:boto:Method: GET 2024-03-04T17:53:37.419 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.419 DEBUG:boto:Data: 2024-03-04T17:53:37.419 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.419 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.419 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.419 DEBUG:boto:Params: {} 2024-03-04T17:53:37.420 DEBUG:boto:Token: None 2024-03-04T17:53:37.420 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.420 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acb22c67183c491a-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.426 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.427 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.427 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.427 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.427 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:37.427 DEBUG:boto:Data: 2024-03-04T17:53:37.427 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.427 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.427 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.428 DEBUG:boto:Params: {} 2024-03-04T17:53:37.428 DEBUG:boto:Token: None 2024-03-04T17:53:37.428 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj 2024-03-04T17:53:37.428 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU= 2024-03-04T17:53:37.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU=', 'Content-Length': '0'} 2024-03-04T17:53:37.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab838192df546a9c-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.434 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.434 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.434 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.434 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:37.434 DEBUG:boto:Data: 2024-03-04T17:53:37.434 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.434 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.434 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.434 DEBUG:boto:Params: {} 2024-03-04T17:53:37.435 DEBUG:boto:Token: None 2024-03-04T17:53:37.435 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj 2024-03-04T17:53:37.435 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU= 2024-03-04T17:53:37.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU=', 'Content-Length': '0'} 2024-03-04T17:53:37.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e624d5095facedcd-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:37.442 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.442 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.442 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.442 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.442 DEBUG:boto:Method: GET 2024-03-04T17:53:37.442 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.442 DEBUG:boto:Data: 2024-03-04T17:53:37.442 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.442 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.443 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.443 DEBUG:boto:Params: {} 2024-03-04T17:53:37.443 DEBUG:boto:Token: None 2024-03-04T17:53:37.443 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.443 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY= 2024-03-04T17:53:37.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY=', 'Content-Length': '0'} 2024-03-04T17:53:37.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b3229ca67e3012b-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.488 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.488 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.488 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.488 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.488 DEBUG:boto:Method: GET 2024-03-04T17:53:37.489 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.489 DEBUG:boto:Data: 2024-03-04T17:53:37.489 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.489 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.489 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.489 DEBUG:boto:Params: {} 2024-03-04T17:53:37.489 DEBUG:boto:Token: None 2024-03-04T17:53:37.489 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.490 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY= 2024-03-04T17:53:37.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY=', 'Content-Length': '0'} 2024-03-04T17:53:37.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2c5a486d0566f2e-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.540 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.540 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.540 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.540 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.541 DEBUG:boto:Method: GET 2024-03-04T17:53:37.541 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.541 DEBUG:boto:Data: 2024-03-04T17:53:37.541 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.541 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.541 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.541 DEBUG:boto:Params: {} 2024-03-04T17:53:37.541 DEBUG:boto:Token: None 2024-03-04T17:53:37.541 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.541 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a49862b98e942c24-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.547 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.547 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.547 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.547 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.548 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.548 DEBUG:boto:Method: GET 2024-03-04T17:53:37.548 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.548 DEBUG:boto:Data: 2024-03-04T17:53:37.548 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.548 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.548 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.548 DEBUG:boto:Params: {} 2024-03-04T17:53:37.548 DEBUG:boto:Token: None 2024-03-04T17:53:37.548 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.549 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d86627a453d41182-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.554 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:37.554 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.554 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.554 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.554 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.555 DEBUG:boto:Method: GET 2024-03-04T17:53:37.555 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.555 DEBUG:boto:Data: 2024-03-04T17:53:37.555 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.555 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.555 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.555 DEBUG:boto:Params: {} 2024-03-04T17:53:37.555 DEBUG:boto:Token: None 2024-03-04T17:53:37.555 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.556 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OfTHPCHnWmokUST2y/zIXBEfvQE= 2024-03-04T17:53:37.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OfTHPCHnWmokUST2y/zIXBEfvQE=', 'Content-Length': '0'} 2024-03-04T17:53:37.559 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006086355894196055-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.559 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.559 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.559 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.559 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.559 DEBUG:boto:Method: GET 2024-03-04T17:53:37.559 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.560 DEBUG:boto:Data: 2024-03-04T17:53:37.560 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.560 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.560 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.560 DEBUG:boto:Params: {} 2024-03-04T17:53:37.560 DEBUG:boto:Token: None 2024-03-04T17:53:37.560 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.560 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OfTHPCHnWmokUST2y/zIXBEfvQE= 2024-03-04T17:53:37.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OfTHPCHnWmokUST2y/zIXBEfvQE=', 'Content-Length': '0'} 2024-03-04T17:53:37.563 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f24161cb7c1060a-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.604 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.604 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.604 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.604 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.604 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.604 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.605 DEBUG:boto:Data: 2024-03-04T17:53:37.605 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.605 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.605 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.605 DEBUG:boto:Params: {} 2024-03-04T17:53:37.605 DEBUG:boto:Token: None 2024-03-04T17:53:37.605 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.605 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3EwFu0lDp++49WSQQY9qUMBJYIc= 2024-03-04T17:53:37.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3EwFu0lDp++49WSQQY9qUMBJYIc=', 'Content-Length': '0'} 2024-03-04T17:53:37.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9fde6b5b0769e94-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.609 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.609 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.609 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.609 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.609 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.609 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.609 DEBUG:boto:Data: 2024-03-04T17:53:37.609 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.609 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.609 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.609 DEBUG:boto:Params: {} 2024-03-04T17:53:37.610 DEBUG:boto:Token: None 2024-03-04T17:53:37.610 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.610 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3EwFu0lDp++49WSQQY9qUMBJYIc= 2024-03-04T17:53:37.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3EwFu0lDp++49WSQQY9qUMBJYIc=', 'Content-Length': '0'} 2024-03-04T17:53:37.613 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a784ace750f110e-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:37.613 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.613 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.613 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.613 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.614 DEBUG:boto:Method: GET 2024-03-04T17:53:37.614 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.614 DEBUG:boto:Data: 2024-03-04T17:53:37.614 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.614 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.614 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.614 DEBUG:boto:Params: {} 2024-03-04T17:53:37.614 DEBUG:boto:Token: None 2024-03-04T17:53:37.614 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.615 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OfTHPCHnWmokUST2y/zIXBEfvQE= 2024-03-04T17:53:37.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OfTHPCHnWmokUST2y/zIXBEfvQE=', 'Content-Length': '0'} 2024-03-04T17:53:37.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000916f3b5a7588d861-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.618 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.618 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.618 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.618 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.618 DEBUG:boto:Method: GET 2024-03-04T17:53:37.618 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.618 DEBUG:boto:Data: 2024-03-04T17:53:37.618 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.619 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.619 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.619 DEBUG:boto:Params: {} 2024-03-04T17:53:37.619 DEBUG:boto:Token: None 2024-03-04T17:53:37.619 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:37.619 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OfTHPCHnWmokUST2y/zIXBEfvQE= 2024-03-04T17:53:37.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OfTHPCHnWmokUST2y/zIXBEfvQE=', 'Content-Length': '0'} 2024-03-04T17:53:37.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cae55c17046fc441-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.664 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.664 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.664 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.664 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.664 DEBUG:boto:Method: GET 2024-03-04T17:53:37.664 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.665 DEBUG:boto:Data: 2024-03-04T17:53:37.665 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.665 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.665 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.665 DEBUG:boto:Params: {} 2024-03-04T17:53:37.665 DEBUG:boto:Token: None 2024-03-04T17:53:37.665 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.665 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca7eb3fa5ebc961b-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.671 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.671 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.671 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.671 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.671 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.671 DEBUG:boto:Method: GET 2024-03-04T17:53:37.671 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.672 DEBUG:boto:Data: 2024-03-04T17:53:37.672 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.672 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.672 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.672 DEBUG:boto:Params: {} 2024-03-04T17:53:37.672 DEBUG:boto:Token: None 2024-03-04T17:53:37.672 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.672 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a7f05f808d0f4f-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.678 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.678 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.678 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.678 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.678 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:37.678 DEBUG:boto:Data: 2024-03-04T17:53:37.678 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.678 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.679 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.679 DEBUG:boto:Params: {} 2024-03-04T17:53:37.679 DEBUG:boto:Token: None 2024-03-04T17:53:37.679 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj 2024-03-04T17:53:37.679 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU= 2024-03-04T17:53:37.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU=', 'Content-Length': '0'} 2024-03-04T17:53:37.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f5dfe10aa2f8c55-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.685 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.685 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.685 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.685 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:37.685 DEBUG:boto:Data: 2024-03-04T17:53:37.685 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.686 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.686 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.686 DEBUG:boto:Params: {} 2024-03-04T17:53:37.686 DEBUG:boto:Token: None 2024-03-04T17:53:37.686 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj 2024-03-04T17:53:37.686 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU= 2024-03-04T17:53:37.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU=', 'Content-Length': '0'} 2024-03-04T17:53:37.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001590115cf487f3a6-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:37.692 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.692 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.692 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.693 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.693 DEBUG:boto:Method: GET 2024-03-04T17:53:37.693 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.693 DEBUG:boto:Data: 2024-03-04T17:53:37.693 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.693 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.693 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.693 DEBUG:boto:Params: {} 2024-03-04T17:53:37.693 DEBUG:boto:Token: None 2024-03-04T17:53:37.693 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.694 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY= 2024-03-04T17:53:37.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY=', 'Content-Length': '0'} 2024-03-04T17:53:37.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008d6d3bdfc73b9a3-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.740 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.740 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.740 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.740 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.740 DEBUG:boto:Method: GET 2024-03-04T17:53:37.741 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.741 DEBUG:boto:Data: 2024-03-04T17:53:37.741 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.741 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.741 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.741 DEBUG:boto:Params: {} 2024-03-04T17:53:37.741 DEBUG:boto:Token: None 2024-03-04T17:53:37.741 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.741 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY= 2024-03-04T17:53:37.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY=', 'Content-Length': '0'} 2024-03-04T17:53:37.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035fcf6be0d634798-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.792 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.792 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.792 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.792 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.792 DEBUG:boto:Method: GET 2024-03-04T17:53:37.793 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.793 DEBUG:boto:Data: 2024-03-04T17:53:37.793 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.793 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.793 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.793 DEBUG:boto:Params: {} 2024-03-04T17:53:37.793 DEBUG:boto:Token: None 2024-03-04T17:53:37.793 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.793 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a869a488bdbe22e4-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.799 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.799 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.799 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.799 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.799 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.800 DEBUG:boto:Method: GET 2024-03-04T17:53:37.800 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.800 DEBUG:boto:Data: 2024-03-04T17:53:37.800 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.800 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.800 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.800 DEBUG:boto:Params: {} 2024-03-04T17:53:37.800 DEBUG:boto:Token: None 2024-03-04T17:53:37.800 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.801 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a31d1cae0c1a08cc-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.806 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:37.806 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.806 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.806 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.806 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.807 DEBUG:boto:Method: GET 2024-03-04T17:53:37.807 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.807 DEBUG:boto:Data: 2024-03-04T17:53:37.807 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.807 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.807 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.807 DEBUG:boto:Params: {} 2024-03-04T17:53:37.807 DEBUG:boto:Token: None 2024-03-04T17:53:37.807 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.807 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BVR+aRRDSgofjLpitFWGWWJNA+4= 2024-03-04T17:53:37.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BVR+aRRDSgofjLpitFWGWWJNA+4=', 'Content-Length': '0'} 2024-03-04T17:53:37.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e48e10e843bb57a2-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.811 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.811 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.811 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.811 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.811 DEBUG:boto:Method: GET 2024-03-04T17:53:37.811 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.811 DEBUG:boto:Data: 2024-03-04T17:53:37.811 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.811 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.812 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.812 DEBUG:boto:Params: {} 2024-03-04T17:53:37.812 DEBUG:boto:Token: None 2024-03-04T17:53:37.812 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.812 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BVR+aRRDSgofjLpitFWGWWJNA+4= 2024-03-04T17:53:37.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BVR+aRRDSgofjLpitFWGWWJNA+4=', 'Content-Length': '0'} 2024-03-04T17:53:37.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f3e5457ad2f3433-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.856 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.856 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.856 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.856 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.857 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.857 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.857 DEBUG:boto:Data: 2024-03-04T17:53:37.857 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.857 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.857 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.857 DEBUG:boto:Params: {} 2024-03-04T17:53:37.857 DEBUG:boto:Token: None 2024-03-04T17:53:37.857 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.857 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kbXFgdC3WtyPOFfSRLnEsXfBB1A= 2024-03-04T17:53:37.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kbXFgdC3WtyPOFfSRLnEsXfBB1A=', 'Content-Length': '0'} 2024-03-04T17:53:37.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1919d9dd7c33398-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.861 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.861 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.861 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.861 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.862 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.862 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.862 DEBUG:boto:Data: 2024-03-04T17:53:37.862 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.862 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.862 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.862 DEBUG:boto:Params: {} 2024-03-04T17:53:37.862 DEBUG:boto:Token: None 2024-03-04T17:53:37.862 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.862 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kbXFgdC3WtyPOFfSRLnEsXfBB1A= 2024-03-04T17:53:37.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kbXFgdC3WtyPOFfSRLnEsXfBB1A=', 'Content-Length': '0'} 2024-03-04T17:53:37.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000518d1b20c7ad14c9-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:37.866 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.866 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.866 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.866 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.866 DEBUG:boto:Method: GET 2024-03-04T17:53:37.866 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.866 DEBUG:boto:Data: 2024-03-04T17:53:37.867 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.867 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.867 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.867 DEBUG:boto:Params: {} 2024-03-04T17:53:37.867 DEBUG:boto:Token: None 2024-03-04T17:53:37.867 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.867 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BVR+aRRDSgofjLpitFWGWWJNA+4= 2024-03-04T17:53:37.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BVR+aRRDSgofjLpitFWGWWJNA+4=', 'Content-Length': '0'} 2024-03-04T17:53:37.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000298e1b68bb1b13a0-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.871 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.871 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.871 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.871 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.871 DEBUG:boto:Method: GET 2024-03-04T17:53:37.871 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.871 DEBUG:boto:Data: 2024-03-04T17:53:37.871 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.871 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.871 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.872 DEBUG:boto:Params: {} 2024-03-04T17:53:37.872 DEBUG:boto:Token: None 2024-03-04T17:53:37.872 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:37.872 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BVR+aRRDSgofjLpitFWGWWJNA+4= 2024-03-04T17:53:37.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BVR+aRRDSgofjLpitFWGWWJNA+4=', 'Content-Length': '0'} 2024-03-04T17:53:37.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fd58605b74f9b1a-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.916 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.916 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.916 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.916 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.916 DEBUG:boto:Method: GET 2024-03-04T17:53:37.916 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.917 DEBUG:boto:Data: 2024-03-04T17:53:37.917 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.917 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.917 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.917 DEBUG:boto:Params: {} 2024-03-04T17:53:37.917 DEBUG:boto:Token: None 2024-03-04T17:53:37.917 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.917 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cc921009e071a68-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.922 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.922 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:37.923 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:37.923 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.923 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:37.923 DEBUG:boto:Method: GET 2024-03-04T17:53:37.923 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:37.923 DEBUG:boto:Data: 2024-03-04T17:53:37.923 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.923 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.923 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.923 DEBUG:boto:Params: {} 2024-03-04T17:53:37.924 DEBUG:boto:Token: None 2024-03-04T17:53:37.924 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/ 2024-03-04T17:53:37.924 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg= 2024-03-04T17:53:37.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:H4XttPSkHqycSJChQ9pn7IbQGBg=', 'Content-Length': '0'} 2024-03-04T17:53:37.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042f18a6b73a80f30-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.928 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:37.929 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.929 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.929 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.929 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:37.929 DEBUG:boto:Data: 2024-03-04T17:53:37.929 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.929 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.929 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.929 DEBUG:boto:Params: {} 2024-03-04T17:53:37.930 DEBUG:boto:Token: None 2024-03-04T17:53:37.930 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj 2024-03-04T17:53:37.930 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU= 2024-03-04T17:53:37.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU=', 'Content-Length': '0'} 2024-03-04T17:53:37.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004115a30362acf138-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.935 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.936 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.936 DEBUG:boto:Method: HEAD 2024-03-04T17:53:37.936 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:37.936 DEBUG:boto:Data: 2024-03-04T17:53:37.936 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.936 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.936 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.936 DEBUG:boto:Params: {} 2024-03-04T17:53:37.936 DEBUG:boto:Token: None 2024-03-04T17:53:37.937 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj 2024-03-04T17:53:37.937 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU= 2024-03-04T17:53:37.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eND2Axz3pRRv+OJ8y7LJH2SFjnU=', 'Content-Length': '0'} 2024-03-04T17:53:37.942 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5b9cbaefaf224c4-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:37.943 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.943 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.943 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.943 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.943 DEBUG:boto:Method: GET 2024-03-04T17:53:37.943 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.943 DEBUG:boto:Data: 2024-03-04T17:53:37.943 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.944 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:37.944 DEBUG:boto:Port: 8001 2024-03-04T17:53:37.944 DEBUG:boto:Params: {} 2024-03-04T17:53:37.944 DEBUG:boto:Token: None 2024-03-04T17:53:37.944 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.944 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY= 2024-03-04T17:53:37.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY=', 'Content-Length': '0'} 2024-03-04T17:53:37.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000704983aa13a1c88f-0065e60aa1-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:37.988 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:37.988 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:37.988 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.988 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.988 DEBUG:boto:Method: GET 2024-03-04T17:53:37.989 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.989 DEBUG:boto:Data: 2024-03-04T17:53:37.989 DEBUG:boto:Headers: {} 2024-03-04T17:53:37.989 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:37.989 DEBUG:boto:Port: 8000 2024-03-04T17:53:37.989 DEBUG:boto:Params: {} 2024-03-04T17:53:37.989 DEBUG:boto:Token: None 2024-03-04T17:53:37.989 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:37 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:37.990 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY= 2024-03-04T17:53:37.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:37 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+Mc2yzof9+whq+nPgDY3+3/gqwY=', 'Content-Length': '0'} 2024-03-04T17:53:37.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d952b0fc229bdbf-0065e60aa1-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.036 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.036 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.036 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.036 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.037 DEBUG:boto:Method: GET 2024-03-04T17:53:38.037 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.037 DEBUG:boto:Data: 2024-03-04T17:53:38.037 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.037 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.037 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.037 DEBUG:boto:Params: {} 2024-03-04T17:53:38.037 DEBUG:boto:Token: None 2024-03-04T17:53:38.037 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.038 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000901c24e80f0a114-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.043 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.043 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.043 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.043 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.044 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.044 DEBUG:boto:Method: GET 2024-03-04T17:53:38.044 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.044 DEBUG:boto:Data: 2024-03-04T17:53:38.044 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.044 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.044 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.044 DEBUG:boto:Params: {} 2024-03-04T17:53:38.044 DEBUG:boto:Token: None 2024-03-04T17:53:38.045 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.045 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000ccdcf01046efd8-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.050 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:38.050 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.050 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.050 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.051 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.051 DEBUG:boto:Method: GET 2024-03-04T17:53:38.051 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.051 DEBUG:boto:Data: 2024-03-04T17:53:38.051 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.051 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.051 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.051 DEBUG:boto:Params: {} 2024-03-04T17:53:38.051 DEBUG:boto:Token: None 2024-03-04T17:53:38.052 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.052 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kB3Q4AB+rEIdkJ41+MAsG4Kzcpo= 2024-03-04T17:53:38.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kB3Q4AB+rEIdkJ41+MAsG4Kzcpo=', 'Content-Length': '0'} 2024-03-04T17:53:38.054 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098607de3ec5277bf-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.055 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.055 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.055 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.055 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.055 DEBUG:boto:Method: GET 2024-03-04T17:53:38.055 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.055 DEBUG:boto:Data: 2024-03-04T17:53:38.056 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.056 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.056 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.056 DEBUG:boto:Params: {} 2024-03-04T17:53:38.056 DEBUG:boto:Token: None 2024-03-04T17:53:38.056 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.056 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kB3Q4AB+rEIdkJ41+MAsG4Kzcpo= 2024-03-04T17:53:38.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kB3Q4AB+rEIdkJ41+MAsG4Kzcpo=', 'Content-Length': '0'} 2024-03-04T17:53:38.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007dac12bb615525d9-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.060 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.060 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.060 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.060 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.060 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.060 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.060 DEBUG:boto:Data: 2024-03-04T17:53:38.060 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.060 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.060 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.061 DEBUG:boto:Params: {} 2024-03-04T17:53:38.061 DEBUG:boto:Token: None 2024-03-04T17:53:38.061 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.061 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IUTEM7R5V2ec9/EUAjwv/6/inEY= 2024-03-04T17:53:38.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IUTEM7R5V2ec9/EUAjwv/6/inEY=', 'Content-Length': '0'} 2024-03-04T17:53:38.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000384a72bcde3c9eab-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.064 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.065 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.065 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.065 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.065 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.065 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.065 DEBUG:boto:Data: 2024-03-04T17:53:38.065 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.065 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.065 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.065 DEBUG:boto:Params: {} 2024-03-04T17:53:38.066 DEBUG:boto:Token: None 2024-03-04T17:53:38.066 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.066 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IUTEM7R5V2ec9/EUAjwv/6/inEY= 2024-03-04T17:53:38.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IUTEM7R5V2ec9/EUAjwv/6/inEY=', 'Content-Length': '0'} 2024-03-04T17:53:38.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f69859273dcf27e-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:38.069 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.069 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.069 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.070 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.070 DEBUG:boto:Method: GET 2024-03-04T17:53:38.070 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.070 DEBUG:boto:Data: 2024-03-04T17:53:38.070 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.070 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.070 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.070 DEBUG:boto:Params: {} 2024-03-04T17:53:38.070 DEBUG:boto:Token: None 2024-03-04T17:53:38.070 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.071 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kB3Q4AB+rEIdkJ41+MAsG4Kzcpo= 2024-03-04T17:53:38.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kB3Q4AB+rEIdkJ41+MAsG4Kzcpo=', 'Content-Length': '0'} 2024-03-04T17:53:38.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009da60ece0b966071-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.116 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.116 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.116 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.116 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.116 DEBUG:boto:Method: GET 2024-03-04T17:53:38.116 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.117 DEBUG:boto:Data: 2024-03-04T17:53:38.117 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.117 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.117 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.117 DEBUG:boto:Params: {} 2024-03-04T17:53:38.117 DEBUG:boto:Token: None 2024-03-04T17:53:38.117 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:38.117 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kB3Q4AB+rEIdkJ41+MAsG4Kzcpo= 2024-03-04T17:53:38.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kB3Q4AB+rEIdkJ41+MAsG4Kzcpo=', 'Content-Length': '0'} 2024-03-04T17:53:38.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000468f3fa5b2fb654f-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.121 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.121 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.121 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.121 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.121 DEBUG:boto:Method: GET 2024-03-04T17:53:38.121 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.121 DEBUG:boto:Data: 2024-03-04T17:53:38.121 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.121 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.121 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.121 DEBUG:boto:Params: {} 2024-03-04T17:53:38.122 DEBUG:boto:Token: None 2024-03-04T17:53:38.122 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.122 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d71d4acfabc37d44-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.127 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.127 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.128 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.128 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.128 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.128 DEBUG:boto:Method: GET 2024-03-04T17:53:38.128 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.128 DEBUG:boto:Data: 2024-03-04T17:53:38.128 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.128 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.128 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.128 DEBUG:boto:Params: {} 2024-03-04T17:53:38.129 DEBUG:boto:Token: None 2024-03-04T17:53:38.129 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.129 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b549f0293d476fa4-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.176 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.176 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.176 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.176 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.177 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:38.177 DEBUG:boto:Data: 2024-03-04T17:53:38.177 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.177 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.177 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.177 DEBUG:boto:Params: {} 2024-03-04T17:53:38.177 DEBUG:boto:Token: None 2024-03-04T17:53:38.177 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj 2024-03-04T17:53:38.177 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Nc5HB8gDLEwtdUeiJfqh1e/ds= 2024-03-04T17:53:38.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Nc5HB8gDLEwtdUeiJfqh1e/ds=', 'Content-Length': '0'} 2024-03-04T17:53:38.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c4b0c09b4db07f9-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.184 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.184 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.184 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.184 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:38.184 DEBUG:boto:Data: 2024-03-04T17:53:38.184 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.184 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.184 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.184 DEBUG:boto:Params: {} 2024-03-04T17:53:38.185 DEBUG:boto:Token: None 2024-03-04T17:53:38.185 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj 2024-03-04T17:53:38.185 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Nc5HB8gDLEwtdUeiJfqh1e/ds= 2024-03-04T17:53:38.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Nc5HB8gDLEwtdUeiJfqh1e/ds=', 'Content-Length': '0'} 2024-03-04T17:53:38.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d13aeacfa1337595-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:38.191 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.191 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.191 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.191 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.191 DEBUG:boto:Method: GET 2024-03-04T17:53:38.191 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.191 DEBUG:boto:Data: 2024-03-04T17:53:38.191 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.192 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.192 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.192 DEBUG:boto:Params: {} 2024-03-04T17:53:38.192 DEBUG:boto:Token: None 2024-03-04T17:53:38.192 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.192 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jgukpGb9Y2sFlLs+H9A1tzmJA5M= 2024-03-04T17:53:38.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jgukpGb9Y2sFlLs+H9A1tzmJA5M=', 'Content-Length': '0'} 2024-03-04T17:53:38.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008770abf0ce901b7c-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.236 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.236 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.236 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.236 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.236 DEBUG:boto:Method: GET 2024-03-04T17:53:38.236 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.236 DEBUG:boto:Data: 2024-03-04T17:53:38.237 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.237 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.237 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.237 DEBUG:boto:Params: {} 2024-03-04T17:53:38.237 DEBUG:boto:Token: None 2024-03-04T17:53:38.237 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.237 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jgukpGb9Y2sFlLs+H9A1tzmJA5M= 2024-03-04T17:53:38.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jgukpGb9Y2sFlLs+H9A1tzmJA5M=', 'Content-Length': '0'} 2024-03-04T17:53:38.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c01d208670134e47-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.284 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.284 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.284 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.284 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.284 DEBUG:boto:Method: GET 2024-03-04T17:53:38.284 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.285 DEBUG:boto:Data: 2024-03-04T17:53:38.285 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.285 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.285 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.285 DEBUG:boto:Params: {} 2024-03-04T17:53:38.285 DEBUG:boto:Token: None 2024-03-04T17:53:38.285 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.285 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000828430bfdf2a6745-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.291 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.291 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.291 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.291 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.291 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.291 DEBUG:boto:Method: GET 2024-03-04T17:53:38.292 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.292 DEBUG:boto:Data: 2024-03-04T17:53:38.292 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.292 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.292 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.292 DEBUG:boto:Params: {} 2024-03-04T17:53:38.292 DEBUG:boto:Token: None 2024-03-04T17:53:38.292 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.292 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c44dbde086bdf78-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.340 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:38.340 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.341 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.341 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.341 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.341 DEBUG:boto:Method: GET 2024-03-04T17:53:38.341 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.341 DEBUG:boto:Data: 2024-03-04T17:53:38.341 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.341 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.341 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.341 DEBUG:boto:Params: {} 2024-03-04T17:53:38.341 DEBUG:boto:Token: None 2024-03-04T17:53:38.341 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.342 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5mUJzKZgK3Xruuq7Odnye4YMhuM= 2024-03-04T17:53:38.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5mUJzKZgK3Xruuq7Odnye4YMhuM=', 'Content-Length': '0'} 2024-03-04T17:53:38.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c60a91f527c1eb72-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.388 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.388 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.388 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.388 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.388 DEBUG:boto:Method: GET 2024-03-04T17:53:38.389 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.389 DEBUG:boto:Data: 2024-03-04T17:53:38.389 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.389 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.389 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.389 DEBUG:boto:Params: {} 2024-03-04T17:53:38.389 DEBUG:boto:Token: None 2024-03-04T17:53:38.389 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.389 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5mUJzKZgK3Xruuq7Odnye4YMhuM= 2024-03-04T17:53:38.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5mUJzKZgK3Xruuq7Odnye4YMhuM=', 'Content-Length': '0'} 2024-03-04T17:53:38.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000379a822da347e7a1-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.392 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.393 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.393 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.393 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.393 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.393 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.393 DEBUG:boto:Data: 2024-03-04T17:53:38.393 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.393 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.393 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.393 DEBUG:boto:Params: {} 2024-03-04T17:53:38.393 DEBUG:boto:Token: None 2024-03-04T17:53:38.394 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.394 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SDhh23Qde3UsqDJjwfKEKqCpF7k= 2024-03-04T17:53:38.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SDhh23Qde3UsqDJjwfKEKqCpF7k=', 'Content-Length': '0'} 2024-03-04T17:53:38.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000516f47ab7bd8d433-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.396 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.397 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.397 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.397 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.397 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.397 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.397 DEBUG:boto:Data: 2024-03-04T17:53:38.397 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.397 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.397 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.397 DEBUG:boto:Params: {} 2024-03-04T17:53:38.397 DEBUG:boto:Token: None 2024-03-04T17:53:38.398 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.398 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SDhh23Qde3UsqDJjwfKEKqCpF7k= 2024-03-04T17:53:38.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SDhh23Qde3UsqDJjwfKEKqCpF7k=', 'Content-Length': '0'} 2024-03-04T17:53:38.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025d1bc0941d99fb0-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:38.401 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.401 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.401 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.402 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.402 DEBUG:boto:Method: GET 2024-03-04T17:53:38.402 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.402 DEBUG:boto:Data: 2024-03-04T17:53:38.402 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.402 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.402 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.402 DEBUG:boto:Params: {} 2024-03-04T17:53:38.402 DEBUG:boto:Token: None 2024-03-04T17:53:38.402 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.402 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5mUJzKZgK3Xruuq7Odnye4YMhuM= 2024-03-04T17:53:38.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5mUJzKZgK3Xruuq7Odnye4YMhuM=', 'Content-Length': '0'} 2024-03-04T17:53:38.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f9a79cf8efec352-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.448 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.448 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.448 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.448 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.448 DEBUG:boto:Method: GET 2024-03-04T17:53:38.449 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.449 DEBUG:boto:Data: 2024-03-04T17:53:38.449 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.449 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.449 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.449 DEBUG:boto:Params: {} 2024-03-04T17:53:38.449 DEBUG:boto:Token: None 2024-03-04T17:53:38.449 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:38.449 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5mUJzKZgK3Xruuq7Odnye4YMhuM= 2024-03-04T17:53:38.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5mUJzKZgK3Xruuq7Odnye4YMhuM=', 'Content-Length': '0'} 2024-03-04T17:53:38.452 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029913549523e7008-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.452 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.453 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.453 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.453 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.453 DEBUG:boto:Method: GET 2024-03-04T17:53:38.453 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.453 DEBUG:boto:Data: 2024-03-04T17:53:38.453 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.453 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.453 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.453 DEBUG:boto:Params: {} 2024-03-04T17:53:38.453 DEBUG:boto:Token: None 2024-03-04T17:53:38.454 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.454 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e69c04027f35776-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.500 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.500 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.500 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.501 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.501 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.501 DEBUG:boto:Method: GET 2024-03-04T17:53:38.501 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.501 DEBUG:boto:Data: 2024-03-04T17:53:38.501 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.501 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.501 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.501 DEBUG:boto:Params: {} 2024-03-04T17:53:38.501 DEBUG:boto:Token: None 2024-03-04T17:53:38.502 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.502 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000383d1b40869c9f20-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.506 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.507 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.507 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.507 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.507 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:38.507 DEBUG:boto:Data: 2024-03-04T17:53:38.507 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.507 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.507 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.507 DEBUG:boto:Params: {} 2024-03-04T17:53:38.508 DEBUG:boto:Token: None 2024-03-04T17:53:38.508 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj 2024-03-04T17:53:38.508 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Nc5HB8gDLEwtdUeiJfqh1e/ds= 2024-03-04T17:53:38.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Nc5HB8gDLEwtdUeiJfqh1e/ds=', 'Content-Length': '0'} 2024-03-04T17:53:38.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e0c96c9c0ce6bde-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.514 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.514 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.514 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.514 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:38.514 DEBUG:boto:Data: 2024-03-04T17:53:38.514 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.514 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.514 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.514 DEBUG:boto:Params: {} 2024-03-04T17:53:38.514 DEBUG:boto:Token: None 2024-03-04T17:53:38.515 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj 2024-03-04T17:53:38.515 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Nc5HB8gDLEwtdUeiJfqh1e/ds= 2024-03-04T17:53:38.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Nc5HB8gDLEwtdUeiJfqh1e/ds=', 'Content-Length': '0'} 2024-03-04T17:53:38.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b02a2679fde24f27-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:38.521 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.521 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.521 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.521 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.521 DEBUG:boto:Method: GET 2024-03-04T17:53:38.521 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.521 DEBUG:boto:Data: 2024-03-04T17:53:38.521 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.521 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.521 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.522 DEBUG:boto:Params: {} 2024-03-04T17:53:38.522 DEBUG:boto:Token: None 2024-03-04T17:53:38.522 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.522 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jgukpGb9Y2sFlLs+H9A1tzmJA5M= 2024-03-04T17:53:38.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jgukpGb9Y2sFlLs+H9A1tzmJA5M=', 'Content-Length': '0'} 2024-03-04T17:53:38.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfa5af296e8a5463-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.525 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.525 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.525 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.525 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.525 DEBUG:boto:Method: GET 2024-03-04T17:53:38.526 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.526 DEBUG:boto:Data: 2024-03-04T17:53:38.526 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.526 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.526 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.526 DEBUG:boto:Params: {} 2024-03-04T17:53:38.526 DEBUG:boto:Token: None 2024-03-04T17:53:38.526 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.526 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jgukpGb9Y2sFlLs+H9A1tzmJA5M= 2024-03-04T17:53:38.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jgukpGb9Y2sFlLs+H9A1tzmJA5M=', 'Content-Length': '0'} 2024-03-04T17:53:38.529 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034cd5b6759231be0-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.576 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.576 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.576 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.576 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.576 DEBUG:boto:Method: GET 2024-03-04T17:53:38.576 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.577 DEBUG:boto:Data: 2024-03-04T17:53:38.577 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.577 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.577 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.577 DEBUG:boto:Params: {} 2024-03-04T17:53:38.577 DEBUG:boto:Token: None 2024-03-04T17:53:38.577 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.577 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0ff829869b28c53-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.582 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.583 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.583 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.583 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.583 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.583 DEBUG:boto:Method: GET 2024-03-04T17:53:38.583 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.583 DEBUG:boto:Data: 2024-03-04T17:53:38.583 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.584 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.584 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.584 DEBUG:boto:Params: {} 2024-03-04T17:53:38.584 DEBUG:boto:Token: None 2024-03-04T17:53:38.584 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.584 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce2368cdb4ead81-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.590 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:38.590 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.591 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.591 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.591 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.591 DEBUG:boto:Method: GET 2024-03-04T17:53:38.591 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.591 DEBUG:boto:Data: 2024-03-04T17:53:38.591 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.591 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.591 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.591 DEBUG:boto:Params: {} 2024-03-04T17:53:38.591 DEBUG:boto:Token: None 2024-03-04T17:53:38.592 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.592 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TTEd6L0ZjSZnResq+aR2uGPM5mA= 2024-03-04T17:53:38.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TTEd6L0ZjSZnResq+aR2uGPM5mA=', 'Content-Length': '0'} 2024-03-04T17:53:38.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001844657083ae3e8f-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.636 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.636 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.636 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.636 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.636 DEBUG:boto:Method: GET 2024-03-04T17:53:38.636 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.637 DEBUG:boto:Data: 2024-03-04T17:53:38.637 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.637 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.637 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.637 DEBUG:boto:Params: {} 2024-03-04T17:53:38.637 DEBUG:boto:Token: None 2024-03-04T17:53:38.637 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.637 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TTEd6L0ZjSZnResq+aR2uGPM5mA= 2024-03-04T17:53:38.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TTEd6L0ZjSZnResq+aR2uGPM5mA=', 'Content-Length': '0'} 2024-03-04T17:53:38.640 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab8d062dc0efcece-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.684 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.684 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.684 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.684 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.684 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.685 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.685 DEBUG:boto:Data: 2024-03-04T17:53:38.685 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.685 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.685 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.685 DEBUG:boto:Params: {} 2024-03-04T17:53:38.685 DEBUG:boto:Token: None 2024-03-04T17:53:38.685 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.685 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:x9capU5CRkzQBY3kMqTQ1LaI92Q= 2024-03-04T17:53:38.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:x9capU5CRkzQBY3kMqTQ1LaI92Q=', 'Content-Length': '0'} 2024-03-04T17:53:38.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b88ca60851a12272-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.689 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.689 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.689 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.689 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.689 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.689 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.689 DEBUG:boto:Data: 2024-03-04T17:53:38.689 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.689 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.689 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.689 DEBUG:boto:Params: {} 2024-03-04T17:53:38.690 DEBUG:boto:Token: None 2024-03-04T17:53:38.690 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.690 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:x9capU5CRkzQBY3kMqTQ1LaI92Q= 2024-03-04T17:53:38.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:x9capU5CRkzQBY3kMqTQ1LaI92Q=', 'Content-Length': '0'} 2024-03-04T17:53:38.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c5286d7aadd4a34-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:38.693 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.693 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.693 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.693 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.693 DEBUG:boto:Method: GET 2024-03-04T17:53:38.693 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.694 DEBUG:boto:Data: 2024-03-04T17:53:38.694 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.694 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.694 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.694 DEBUG:boto:Params: {} 2024-03-04T17:53:38.694 DEBUG:boto:Token: None 2024-03-04T17:53:38.694 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.694 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TTEd6L0ZjSZnResq+aR2uGPM5mA= 2024-03-04T17:53:38.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TTEd6L0ZjSZnResq+aR2uGPM5mA=', 'Content-Length': '0'} 2024-03-04T17:53:38.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cec4c9a1f8139368-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.740 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.740 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.740 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.740 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.740 DEBUG:boto:Method: GET 2024-03-04T17:53:38.741 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.741 DEBUG:boto:Data: 2024-03-04T17:53:38.741 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.741 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.741 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.741 DEBUG:boto:Params: {} 2024-03-04T17:53:38.741 DEBUG:boto:Token: None 2024-03-04T17:53:38.741 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:38.741 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TTEd6L0ZjSZnResq+aR2uGPM5mA= 2024-03-04T17:53:38.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TTEd6L0ZjSZnResq+aR2uGPM5mA=', 'Content-Length': '0'} 2024-03-04T17:53:38.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034a1da3009705b28-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.788 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.788 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.788 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.788 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.788 DEBUG:boto:Method: GET 2024-03-04T17:53:38.789 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.789 DEBUG:boto:Data: 2024-03-04T17:53:38.789 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.789 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.789 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.789 DEBUG:boto:Params: {} 2024-03-04T17:53:38.789 DEBUG:boto:Token: None 2024-03-04T17:53:38.789 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.789 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f44eba0367f55c3e-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.794 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.795 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.795 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.795 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.795 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.795 DEBUG:boto:Method: GET 2024-03-04T17:53:38.795 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.795 DEBUG:boto:Data: 2024-03-04T17:53:38.795 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.795 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.795 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.796 DEBUG:boto:Params: {} 2024-03-04T17:53:38.796 DEBUG:boto:Token: None 2024-03-04T17:53:38.796 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.796 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073a3bbc9366e239b-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.801 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.801 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.801 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.801 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.801 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:38.801 DEBUG:boto:Data: 2024-03-04T17:53:38.802 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.802 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.802 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.802 DEBUG:boto:Params: {} 2024-03-04T17:53:38.802 DEBUG:boto:Token: None 2024-03-04T17:53:38.802 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj 2024-03-04T17:53:38.802 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Nc5HB8gDLEwtdUeiJfqh1e/ds= 2024-03-04T17:53:38.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Nc5HB8gDLEwtdUeiJfqh1e/ds=', 'Content-Length': '0'} 2024-03-04T17:53:38.808 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff746845e905ce66-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.808 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.808 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.808 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.809 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:38.809 DEBUG:boto:Data: 2024-03-04T17:53:38.809 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.809 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.809 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.809 DEBUG:boto:Params: {} 2024-03-04T17:53:38.809 DEBUG:boto:Token: None 2024-03-04T17:53:38.809 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj 2024-03-04T17:53:38.809 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:m8Nc5HB8gDLEwtdUeiJfqh1e/ds= 2024-03-04T17:53:38.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m8Nc5HB8gDLEwtdUeiJfqh1e/ds=', 'Content-Length': '0'} 2024-03-04T17:53:38.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c1ada323d6bfd1c-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:38.815 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.815 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.815 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.816 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.816 DEBUG:boto:Method: GET 2024-03-04T17:53:38.816 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.816 DEBUG:boto:Data: 2024-03-04T17:53:38.816 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.816 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.816 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.816 DEBUG:boto:Params: {} 2024-03-04T17:53:38.816 DEBUG:boto:Token: None 2024-03-04T17:53:38.817 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.817 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jgukpGb9Y2sFlLs+H9A1tzmJA5M= 2024-03-04T17:53:38.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jgukpGb9Y2sFlLs+H9A1tzmJA5M=', 'Content-Length': '0'} 2024-03-04T17:53:38.819 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000469eca5aab741cb1-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.860 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.860 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.860 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.860 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.860 DEBUG:boto:Method: GET 2024-03-04T17:53:38.861 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.861 DEBUG:boto:Data: 2024-03-04T17:53:38.861 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.861 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.861 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.861 DEBUG:boto:Params: {} 2024-03-04T17:53:38.861 DEBUG:boto:Token: None 2024-03-04T17:53:38.861 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:38.861 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jgukpGb9Y2sFlLs+H9A1tzmJA5M= 2024-03-04T17:53:38.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jgukpGb9Y2sFlLs+H9A1tzmJA5M=', 'Content-Length': '0'} 2024-03-04T17:53:38.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077a366a232c2956f-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.912 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.912 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.912 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.912 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.912 DEBUG:boto:Method: GET 2024-03-04T17:53:38.913 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.913 DEBUG:boto:Data: 2024-03-04T17:53:38.913 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.913 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.913 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.913 DEBUG:boto:Params: {} 2024-03-04T17:53:38.913 DEBUG:boto:Token: None 2024-03-04T17:53:38.913 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.913 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5e32d5973e343d7-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.918 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.918 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:38.918 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:38.919 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.919 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:38.919 DEBUG:boto:Method: GET 2024-03-04T17:53:38.919 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:38.919 DEBUG:boto:Data: 2024-03-04T17:53:38.919 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.919 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.919 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.919 DEBUG:boto:Params: {} 2024-03-04T17:53:38.919 DEBUG:boto:Token: None 2024-03-04T17:53:38.920 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/ 2024-03-04T17:53:38.920 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc= 2024-03-04T17:53:38.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dIWUq9mAlYZ12Gu/7BuzPwFRDoc=', 'Content-Length': '0'} 2024-03-04T17:53:38.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edf18db6635b2075-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.924 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:38.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:38.925 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.925 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.925 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.925 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.925 DEBUG:boto:Method: GET 2024-03-04T17:53:38.925 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.925 DEBUG:boto:Data: 2024-03-04T17:53:38.925 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.925 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.926 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.926 DEBUG:boto:Params: {} 2024-03-04T17:53:38.926 DEBUG:boto:Token: None 2024-03-04T17:53:38.926 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.926 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:75JV2T3DPVSuzIjpGaPZg4V9DwY= 2024-03-04T17:53:38.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:75JV2T3DPVSuzIjpGaPZg4V9DwY=', 'Content-Length': '0'} 2024-03-04T17:53:38.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006df36d28593b003b-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.929 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.929 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.929 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.929 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.929 DEBUG:boto:Method: GET 2024-03-04T17:53:38.930 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.930 DEBUG:boto:Data: 2024-03-04T17:53:38.930 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.930 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.930 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.930 DEBUG:boto:Params: {} 2024-03-04T17:53:38.930 DEBUG:boto:Token: None 2024-03-04T17:53:38.930 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.930 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:75JV2T3DPVSuzIjpGaPZg4V9DwY= 2024-03-04T17:53:38.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:75JV2T3DPVSuzIjpGaPZg4V9DwY=', 'Content-Length': '0'} 2024-03-04T17:53:38.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea3fd1ca5ed10673-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.976 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.976 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.976 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.976 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.976 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.976 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.977 DEBUG:boto:Data: 2024-03-04T17:53:38.977 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.977 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.977 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.977 DEBUG:boto:Params: {} 2024-03-04T17:53:38.977 DEBUG:boto:Token: None 2024-03-04T17:53:38.977 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.977 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:njQaPMQ1fAkK7wESs5AcB/36OTY= 2024-03-04T17:53:38.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:njQaPMQ1fAkK7wESs5AcB/36OTY=', 'Content-Length': '0'} 2024-03-04T17:53:38.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003a6dfaf72b3bca0-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.980 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.980 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.980 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.980 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.981 DEBUG:boto:Method: HEAD 2024-03-04T17:53:38.981 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.981 DEBUG:boto:Data: 2024-03-04T17:53:38.981 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.981 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.981 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.981 DEBUG:boto:Params: {} 2024-03-04T17:53:38.981 DEBUG:boto:Token: None 2024-03-04T17:53:38.981 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.981 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:njQaPMQ1fAkK7wESs5AcB/36OTY= 2024-03-04T17:53:38.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:njQaPMQ1fAkK7wESs5AcB/36OTY=', 'Content-Length': '0'} 2024-03-04T17:53:38.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fda5eeff3b5aa4a-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:38.984 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.985 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.985 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.985 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.985 DEBUG:boto:Method: GET 2024-03-04T17:53:38.985 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.985 DEBUG:boto:Data: 2024-03-04T17:53:38.985 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.985 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:38.985 DEBUG:boto:Port: 8001 2024-03-04T17:53:38.985 DEBUG:boto:Params: {} 2024-03-04T17:53:38.985 DEBUG:boto:Token: None 2024-03-04T17:53:38.986 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.986 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:75JV2T3DPVSuzIjpGaPZg4V9DwY= 2024-03-04T17:53:38.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:75JV2T3DPVSuzIjpGaPZg4V9DwY=', 'Content-Length': '0'} 2024-03-04T17:53:38.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004555b9e7e8ea8121-0065e60aa2-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:38.989 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:38.989 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:38.989 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.990 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.990 DEBUG:boto:Method: GET 2024-03-04T17:53:38.990 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.990 DEBUG:boto:Data: 2024-03-04T17:53:38.990 DEBUG:boto:Headers: {} 2024-03-04T17:53:38.990 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:38.990 DEBUG:boto:Port: 8000 2024-03-04T17:53:38.990 DEBUG:boto:Params: {} 2024-03-04T17:53:38.990 DEBUG:boto:Token: None 2024-03-04T17:53:38.990 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:38 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:38.990 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:75JV2T3DPVSuzIjpGaPZg4V9DwY= 2024-03-04T17:53:38.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:75JV2T3DPVSuzIjpGaPZg4V9DwY=', 'Content-Length': '0'} 2024-03-04T17:53:38.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037326f98c3b04924-0065e60aa2-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.036 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.036 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.036 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.036 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.036 DEBUG:boto:Method: GET 2024-03-04T17:53:39.037 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.037 DEBUG:boto:Data: 2024-03-04T17:53:39.037 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.037 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.037 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.037 DEBUG:boto:Params: {} 2024-03-04T17:53:39.037 DEBUG:boto:Token: None 2024-03-04T17:53:39.037 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.037 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f36976e7547c04b-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.042 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.042 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.043 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.043 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.043 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.043 DEBUG:boto:Method: GET 2024-03-04T17:53:39.043 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.043 DEBUG:boto:Data: 2024-03-04T17:53:39.043 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.043 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.043 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.043 DEBUG:boto:Params: {} 2024-03-04T17:53:39.043 DEBUG:boto:Token: None 2024-03-04T17:53:39.044 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.044 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2a2eaa2612c736e-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.048 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.049 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.049 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.049 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.049 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:39.049 DEBUG:boto:Data: 2024-03-04T17:53:39.049 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.049 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.049 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.049 DEBUG:boto:Params: {} 2024-03-04T17:53:39.050 DEBUG:boto:Token: None 2024-03-04T17:53:39.050 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj 2024-03-04T17:53:39.050 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ= 2024-03-04T17:53:39.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ=', 'Content-Length': '0'} 2024-03-04T17:53:39.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000babd72ceec85e992-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.056 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.056 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.056 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.056 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:39.056 DEBUG:boto:Data: 2024-03-04T17:53:39.056 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.056 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.056 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.056 DEBUG:boto:Params: {} 2024-03-04T17:53:39.057 DEBUG:boto:Token: None 2024-03-04T17:53:39.057 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj 2024-03-04T17:53:39.057 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ= 2024-03-04T17:53:39.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ=', 'Content-Length': '0'} 2024-03-04T17:53:39.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f301d872305414f-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:39.063 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.063 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.063 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.063 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.063 DEBUG:boto:Method: GET 2024-03-04T17:53:39.063 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.063 DEBUG:boto:Data: 2024-03-04T17:53:39.064 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.064 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.064 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.064 DEBUG:boto:Params: {} 2024-03-04T17:53:39.064 DEBUG:boto:Token: None 2024-03-04T17:53:39.064 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.064 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk= 2024-03-04T17:53:39.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk=', 'Content-Length': '0'} 2024-03-04T17:53:39.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082140bcbdbd6dc4a-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.108 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.108 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.108 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.108 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.108 DEBUG:boto:Method: GET 2024-03-04T17:53:39.108 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.109 DEBUG:boto:Data: 2024-03-04T17:53:39.109 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.109 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.109 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.109 DEBUG:boto:Params: {} 2024-03-04T17:53:39.109 DEBUG:boto:Token: None 2024-03-04T17:53:39.109 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.109 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk= 2024-03-04T17:53:39.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk=', 'Content-Length': '0'} 2024-03-04T17:53:39.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034c7bd6b77ab7d02-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.156 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.156 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.156 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.156 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.156 DEBUG:boto:Method: GET 2024-03-04T17:53:39.156 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.157 DEBUG:boto:Data: 2024-03-04T17:53:39.157 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.157 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.157 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.157 DEBUG:boto:Params: {} 2024-03-04T17:53:39.157 DEBUG:boto:Token: None 2024-03-04T17:53:39.157 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.157 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a2c81401701db77-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.162 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.162 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.162 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.163 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.163 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.163 DEBUG:boto:Method: GET 2024-03-04T17:53:39.163 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.163 DEBUG:boto:Data: 2024-03-04T17:53:39.163 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.163 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.163 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.163 DEBUG:boto:Params: {} 2024-03-04T17:53:39.163 DEBUG:boto:Token: None 2024-03-04T17:53:39.163 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.164 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083b15dda544e01a3-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.168 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:39.169 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.169 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.169 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.169 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.169 DEBUG:boto:Method: GET 2024-03-04T17:53:39.169 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.169 DEBUG:boto:Data: 2024-03-04T17:53:39.169 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.170 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.170 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.170 DEBUG:boto:Params: {} 2024-03-04T17:53:39.170 DEBUG:boto:Token: None 2024-03-04T17:53:39.170 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.170 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LorhVBeN2RS7i+f/LNmZcfaGgXs= 2024-03-04T17:53:39.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LorhVBeN2RS7i+f/LNmZcfaGgXs=', 'Content-Length': '0'} 2024-03-04T17:53:39.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f449a7c7ce29a486-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.173 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.173 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.173 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.174 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.174 DEBUG:boto:Method: GET 2024-03-04T17:53:39.174 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.174 DEBUG:boto:Data: 2024-03-04T17:53:39.174 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.174 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.174 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.174 DEBUG:boto:Params: {} 2024-03-04T17:53:39.174 DEBUG:boto:Token: None 2024-03-04T17:53:39.175 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.175 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LorhVBeN2RS7i+f/LNmZcfaGgXs= 2024-03-04T17:53:39.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LorhVBeN2RS7i+f/LNmZcfaGgXs=', 'Content-Length': '0'} 2024-03-04T17:53:39.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e98f84867450e416-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.220 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.220 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.220 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.220 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.220 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.221 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.221 DEBUG:boto:Data: 2024-03-04T17:53:39.221 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.221 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.221 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.221 DEBUG:boto:Params: {} 2024-03-04T17:53:39.221 DEBUG:boto:Token: None 2024-03-04T17:53:39.221 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.221 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TuB8UhTFoSzUakNt+IzXo5UL+GY= 2024-03-04T17:53:39.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TuB8UhTFoSzUakNt+IzXo5UL+GY=', 'Content-Length': '0'} 2024-03-04T17:53:39.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a98885b9de096a7-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.224 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.225 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.225 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.225 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.225 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.225 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.225 DEBUG:boto:Data: 2024-03-04T17:53:39.225 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.225 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.225 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.225 DEBUG:boto:Params: {} 2024-03-04T17:53:39.226 DEBUG:boto:Token: None 2024-03-04T17:53:39.226 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.226 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TuB8UhTFoSzUakNt+IzXo5UL+GY= 2024-03-04T17:53:39.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TuB8UhTFoSzUakNt+IzXo5UL+GY=', 'Content-Length': '0'} 2024-03-04T17:53:39.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6320a002598a241-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:39.229 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.229 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.229 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.229 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.230 DEBUG:boto:Method: GET 2024-03-04T17:53:39.230 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.230 DEBUG:boto:Data: 2024-03-04T17:53:39.230 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.230 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.230 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.230 DEBUG:boto:Params: {} 2024-03-04T17:53:39.230 DEBUG:boto:Token: None 2024-03-04T17:53:39.230 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.230 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LorhVBeN2RS7i+f/LNmZcfaGgXs= 2024-03-04T17:53:39.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LorhVBeN2RS7i+f/LNmZcfaGgXs=', 'Content-Length': '0'} 2024-03-04T17:53:39.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005185667353fe121a-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.280 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.280 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.280 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.280 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.280 DEBUG:boto:Method: GET 2024-03-04T17:53:39.280 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.281 DEBUG:boto:Data: 2024-03-04T17:53:39.281 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.281 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.281 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.281 DEBUG:boto:Params: {} 2024-03-04T17:53:39.281 DEBUG:boto:Token: None 2024-03-04T17:53:39.281 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:39.281 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LorhVBeN2RS7i+f/LNmZcfaGgXs= 2024-03-04T17:53:39.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LorhVBeN2RS7i+f/LNmZcfaGgXs=', 'Content-Length': '0'} 2024-03-04T17:53:39.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000443416e15ec50c97-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.332 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.332 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.332 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.332 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.332 DEBUG:boto:Method: GET 2024-03-04T17:53:39.332 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.333 DEBUG:boto:Data: 2024-03-04T17:53:39.333 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.333 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.333 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.333 DEBUG:boto:Params: {} 2024-03-04T17:53:39.333 DEBUG:boto:Token: None 2024-03-04T17:53:39.333 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.333 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7905faef0b605a8-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.380 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.381 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.381 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.381 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.381 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.381 DEBUG:boto:Method: GET 2024-03-04T17:53:39.381 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.381 DEBUG:boto:Data: 2024-03-04T17:53:39.381 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.381 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.381 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.381 DEBUG:boto:Params: {} 2024-03-04T17:53:39.382 DEBUG:boto:Token: None 2024-03-04T17:53:39.382 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.382 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee4978b27a4086a-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.387 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.388 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.388 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.388 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.388 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:39.388 DEBUG:boto:Data: 2024-03-04T17:53:39.388 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.388 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.388 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.389 DEBUG:boto:Params: {} 2024-03-04T17:53:39.389 DEBUG:boto:Token: None 2024-03-04T17:53:39.389 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj 2024-03-04T17:53:39.389 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ= 2024-03-04T17:53:39.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ=', 'Content-Length': '0'} 2024-03-04T17:53:39.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000492089ff4a06b250-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.395 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.395 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.395 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.395 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:39.395 DEBUG:boto:Data: 2024-03-04T17:53:39.396 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.396 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.396 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.396 DEBUG:boto:Params: {} 2024-03-04T17:53:39.396 DEBUG:boto:Token: None 2024-03-04T17:53:39.396 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj 2024-03-04T17:53:39.396 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ= 2024-03-04T17:53:39.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ=', 'Content-Length': '0'} 2024-03-04T17:53:39.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ae0f8eac46a4d47-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:39.402 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.402 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.402 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.403 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.403 DEBUG:boto:Method: GET 2024-03-04T17:53:39.403 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.403 DEBUG:boto:Data: 2024-03-04T17:53:39.403 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.403 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.403 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.403 DEBUG:boto:Params: {} 2024-03-04T17:53:39.403 DEBUG:boto:Token: None 2024-03-04T17:53:39.403 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.404 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk= 2024-03-04T17:53:39.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk=', 'Content-Length': '0'} 2024-03-04T17:53:39.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029a28ca7b6f70ceb-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.406 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.407 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.407 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.407 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.407 DEBUG:boto:Method: GET 2024-03-04T17:53:39.407 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.407 DEBUG:boto:Data: 2024-03-04T17:53:39.407 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.407 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.407 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.407 DEBUG:boto:Params: {} 2024-03-04T17:53:39.407 DEBUG:boto:Token: None 2024-03-04T17:53:39.408 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.408 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk= 2024-03-04T17:53:39.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk=', 'Content-Length': '0'} 2024-03-04T17:53:39.411 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7dd3f617d66942e-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.452 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.452 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.452 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.452 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.452 DEBUG:boto:Method: GET 2024-03-04T17:53:39.453 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.453 DEBUG:boto:Data: 2024-03-04T17:53:39.453 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.453 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.453 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.453 DEBUG:boto:Params: {} 2024-03-04T17:53:39.453 DEBUG:boto:Token: None 2024-03-04T17:53:39.453 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.453 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000460f329a874afe9e-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.459 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.459 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.459 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.459 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.459 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.460 DEBUG:boto:Method: GET 2024-03-04T17:53:39.460 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.460 DEBUG:boto:Data: 2024-03-04T17:53:39.460 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.460 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.460 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.460 DEBUG:boto:Params: {} 2024-03-04T17:53:39.460 DEBUG:boto:Token: None 2024-03-04T17:53:39.460 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.460 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011b10291b17b865a-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.466 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:39.466 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.466 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.466 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.466 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.466 DEBUG:boto:Method: GET 2024-03-04T17:53:39.466 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.467 DEBUG:boto:Data: 2024-03-04T17:53:39.467 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.467 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.467 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.467 DEBUG:boto:Params: {} 2024-03-04T17:53:39.467 DEBUG:boto:Token: None 2024-03-04T17:53:39.467 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.467 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WivvWpNlRnBgqzfBNG7HlL0VseI= 2024-03-04T17:53:39.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WivvWpNlRnBgqzfBNG7HlL0VseI=', 'Content-Length': '0'} 2024-03-04T17:53:39.470 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e09d843f8841296-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.512 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.512 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.512 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.512 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.512 DEBUG:boto:Method: GET 2024-03-04T17:53:39.512 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.512 DEBUG:boto:Data: 2024-03-04T17:53:39.513 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.513 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.513 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.513 DEBUG:boto:Params: {} 2024-03-04T17:53:39.513 DEBUG:boto:Token: None 2024-03-04T17:53:39.513 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.513 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WivvWpNlRnBgqzfBNG7HlL0VseI= 2024-03-04T17:53:39.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WivvWpNlRnBgqzfBNG7HlL0VseI=', 'Content-Length': '0'} 2024-03-04T17:53:39.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032d878cbf864066b-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.517 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.517 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.517 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.517 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.517 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.517 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.518 DEBUG:boto:Data: 2024-03-04T17:53:39.518 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.518 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.518 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.518 DEBUG:boto:Params: {} 2024-03-04T17:53:39.518 DEBUG:boto:Token: None 2024-03-04T17:53:39.518 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.518 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vBUq4qKfJ4poKVG0bazRrxeWHOo= 2024-03-04T17:53:39.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vBUq4qKfJ4poKVG0bazRrxeWHOo=', 'Content-Length': '0'} 2024-03-04T17:53:39.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000399a61eba6bcd09a-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.521 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.522 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.522 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.522 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.522 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.522 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.522 DEBUG:boto:Data: 2024-03-04T17:53:39.522 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.522 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.522 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.522 DEBUG:boto:Params: {} 2024-03-04T17:53:39.522 DEBUG:boto:Token: None 2024-03-04T17:53:39.523 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.523 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vBUq4qKfJ4poKVG0bazRrxeWHOo= 2024-03-04T17:53:39.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vBUq4qKfJ4poKVG0bazRrxeWHOo=', 'Content-Length': '0'} 2024-03-04T17:53:39.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dccaf08e2bc5e7a8-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:39.526 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.526 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.526 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.526 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.526 DEBUG:boto:Method: GET 2024-03-04T17:53:39.527 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.527 DEBUG:boto:Data: 2024-03-04T17:53:39.527 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.527 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.527 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.527 DEBUG:boto:Params: {} 2024-03-04T17:53:39.527 DEBUG:boto:Token: None 2024-03-04T17:53:39.527 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.527 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WivvWpNlRnBgqzfBNG7HlL0VseI= 2024-03-04T17:53:39.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WivvWpNlRnBgqzfBNG7HlL0VseI=', 'Content-Length': '0'} 2024-03-04T17:53:39.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000329e46491fb86351-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.572 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.572 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.572 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.572 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.572 DEBUG:boto:Method: GET 2024-03-04T17:53:39.572 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.573 DEBUG:boto:Data: 2024-03-04T17:53:39.573 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.573 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.573 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.573 DEBUG:boto:Params: {} 2024-03-04T17:53:39.573 DEBUG:boto:Token: None 2024-03-04T17:53:39.573 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:39.573 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WivvWpNlRnBgqzfBNG7HlL0VseI= 2024-03-04T17:53:39.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WivvWpNlRnBgqzfBNG7HlL0VseI=', 'Content-Length': '0'} 2024-03-04T17:53:39.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053ddb53db1d67a8b-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.577 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.577 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.577 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.577 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.577 DEBUG:boto:Method: GET 2024-03-04T17:53:39.577 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.577 DEBUG:boto:Data: 2024-03-04T17:53:39.577 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.577 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.578 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.578 DEBUG:boto:Params: {} 2024-03-04T17:53:39.578 DEBUG:boto:Token: None 2024-03-04T17:53:39.578 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.578 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000736172ff3805baf3-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.624 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.624 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.624 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.625 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.625 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.625 DEBUG:boto:Method: GET 2024-03-04T17:53:39.625 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.625 DEBUG:boto:Data: 2024-03-04T17:53:39.625 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.625 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.625 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.625 DEBUG:boto:Params: {} 2024-03-04T17:53:39.625 DEBUG:boto:Token: None 2024-03-04T17:53:39.626 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.626 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e4ea6e1a2235b12-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.630 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.630 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.630 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.630 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.631 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:39.631 DEBUG:boto:Data: 2024-03-04T17:53:39.631 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.631 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.631 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.631 DEBUG:boto:Params: {} 2024-03-04T17:53:39.631 DEBUG:boto:Token: None 2024-03-04T17:53:39.631 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj 2024-03-04T17:53:39.631 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ= 2024-03-04T17:53:39.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ=', 'Content-Length': '0'} 2024-03-04T17:53:39.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000650364f5efa67f5-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.638 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.638 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.638 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.638 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:39.638 DEBUG:boto:Data: 2024-03-04T17:53:39.638 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.638 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.638 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.638 DEBUG:boto:Params: {} 2024-03-04T17:53:39.638 DEBUG:boto:Token: None 2024-03-04T17:53:39.639 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj 2024-03-04T17:53:39.639 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ= 2024-03-04T17:53:39.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ=', 'Content-Length': '0'} 2024-03-04T17:53:39.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067f178501c5f69fc-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:39.645 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.645 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.645 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.645 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.645 DEBUG:boto:Method: GET 2024-03-04T17:53:39.646 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.646 DEBUG:boto:Data: 2024-03-04T17:53:39.646 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.646 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.646 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.646 DEBUG:boto:Params: {} 2024-03-04T17:53:39.646 DEBUG:boto:Token: None 2024-03-04T17:53:39.646 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.646 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk= 2024-03-04T17:53:39.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk=', 'Content-Length': '0'} 2024-03-04T17:53:39.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c193d2b999932d2b-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.649 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.650 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.650 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.650 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.650 DEBUG:boto:Method: GET 2024-03-04T17:53:39.650 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.650 DEBUG:boto:Data: 2024-03-04T17:53:39.650 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.650 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.650 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.650 DEBUG:boto:Params: {} 2024-03-04T17:53:39.651 DEBUG:boto:Token: None 2024-03-04T17:53:39.651 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.651 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk= 2024-03-04T17:53:39.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk=', 'Content-Length': '0'} 2024-03-04T17:53:39.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078dae6e6f370b7fb-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.696 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.696 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.696 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.696 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.696 DEBUG:boto:Method: GET 2024-03-04T17:53:39.697 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.697 DEBUG:boto:Data: 2024-03-04T17:53:39.697 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.697 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.697 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.697 DEBUG:boto:Params: {} 2024-03-04T17:53:39.697 DEBUG:boto:Token: None 2024-03-04T17:53:39.697 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.697 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000588acb979e4f035b-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.703 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.703 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.703 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.703 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.703 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.703 DEBUG:boto:Method: GET 2024-03-04T17:53:39.703 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.704 DEBUG:boto:Data: 2024-03-04T17:53:39.704 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.704 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.704 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.704 DEBUG:boto:Params: {} 2024-03-04T17:53:39.704 DEBUG:boto:Token: None 2024-03-04T17:53:39.704 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.704 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b39f39ede6ae4e3b-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.709 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:39.709 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.709 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.709 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.709 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.710 DEBUG:boto:Method: GET 2024-03-04T17:53:39.710 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.710 DEBUG:boto:Data: 2024-03-04T17:53:39.710 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.710 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.710 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.710 DEBUG:boto:Params: {} 2024-03-04T17:53:39.710 DEBUG:boto:Token: None 2024-03-04T17:53:39.710 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.710 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s25YZWQB/tXaX4gWPBAMfBteMBs= 2024-03-04T17:53:39.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s25YZWQB/tXaX4gWPBAMfBteMBs=', 'Content-Length': '0'} 2024-03-04T17:53:39.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008201c95d7e915e6c-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.756 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.756 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.756 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.756 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.756 DEBUG:boto:Method: GET 2024-03-04T17:53:39.757 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.757 DEBUG:boto:Data: 2024-03-04T17:53:39.757 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.757 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.757 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.757 DEBUG:boto:Params: {} 2024-03-04T17:53:39.757 DEBUG:boto:Token: None 2024-03-04T17:53:39.757 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.757 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s25YZWQB/tXaX4gWPBAMfBteMBs= 2024-03-04T17:53:39.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s25YZWQB/tXaX4gWPBAMfBteMBs=', 'Content-Length': '0'} 2024-03-04T17:53:39.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000359d2e616e2c06f9-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.804 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.804 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.804 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.804 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.805 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.805 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.805 DEBUG:boto:Data: 2024-03-04T17:53:39.805 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.805 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.805 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.805 DEBUG:boto:Params: {} 2024-03-04T17:53:39.805 DEBUG:boto:Token: None 2024-03-04T17:53:39.805 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.806 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9686JJbjyfeq88TADN5D31N39AQ= 2024-03-04T17:53:39.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9686JJbjyfeq88TADN5D31N39AQ=', 'Content-Length': '0'} 2024-03-04T17:53:39.808 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf23ef8943f9ff35-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.809 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.809 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.809 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.809 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.809 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.809 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.809 DEBUG:boto:Data: 2024-03-04T17:53:39.809 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.809 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.810 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.810 DEBUG:boto:Params: {} 2024-03-04T17:53:39.810 DEBUG:boto:Token: None 2024-03-04T17:53:39.810 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.810 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9686JJbjyfeq88TADN5D31N39AQ= 2024-03-04T17:53:39.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9686JJbjyfeq88TADN5D31N39AQ=', 'Content-Length': '0'} 2024-03-04T17:53:39.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bb8447c2e4f8d83-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:39.813 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.813 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.814 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.814 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.814 DEBUG:boto:Method: GET 2024-03-04T17:53:39.814 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.814 DEBUG:boto:Data: 2024-03-04T17:53:39.814 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.814 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.814 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.814 DEBUG:boto:Params: {} 2024-03-04T17:53:39.815 DEBUG:boto:Token: None 2024-03-04T17:53:39.815 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.815 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s25YZWQB/tXaX4gWPBAMfBteMBs= 2024-03-04T17:53:39.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s25YZWQB/tXaX4gWPBAMfBteMBs=', 'Content-Length': '0'} 2024-03-04T17:53:39.818 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000455c7ecde1a11da4-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.818 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.818 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.818 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.818 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.818 DEBUG:boto:Method: GET 2024-03-04T17:53:39.818 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.818 DEBUG:boto:Data: 2024-03-04T17:53:39.819 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.819 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.819 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.819 DEBUG:boto:Params: {} 2024-03-04T17:53:39.819 DEBUG:boto:Token: None 2024-03-04T17:53:39.819 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:39.819 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s25YZWQB/tXaX4gWPBAMfBteMBs= 2024-03-04T17:53:39.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s25YZWQB/tXaX4gWPBAMfBteMBs=', 'Content-Length': '0'} 2024-03-04T17:53:39.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000123f379a9fba7686-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.864 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.864 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.864 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.864 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.865 DEBUG:boto:Method: GET 2024-03-04T17:53:39.865 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.865 DEBUG:boto:Data: 2024-03-04T17:53:39.865 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.865 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.865 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.865 DEBUG:boto:Params: {} 2024-03-04T17:53:39.865 DEBUG:boto:Token: None 2024-03-04T17:53:39.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.866 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000860767ce88450236-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.871 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.872 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.872 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.872 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.872 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.872 DEBUG:boto:Method: GET 2024-03-04T17:53:39.872 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.873 DEBUG:boto:Data: 2024-03-04T17:53:39.873 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.873 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.873 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.873 DEBUG:boto:Params: {} 2024-03-04T17:53:39.873 DEBUG:boto:Token: None 2024-03-04T17:53:39.873 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.873 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8ee87a7246109dd-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.878 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.878 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.878 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.878 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.878 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:39.878 DEBUG:boto:Data: 2024-03-04T17:53:39.879 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.879 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.879 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.879 DEBUG:boto:Params: {} 2024-03-04T17:53:39.879 DEBUG:boto:Token: None 2024-03-04T17:53:39.879 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj 2024-03-04T17:53:39.879 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ= 2024-03-04T17:53:39.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ=', 'Content-Length': '0'} 2024-03-04T17:53:39.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d206db2cb826c9ee-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.885 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.885 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.886 DEBUG:boto:Method: HEAD 2024-03-04T17:53:39.886 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:39.886 DEBUG:boto:Data: 2024-03-04T17:53:39.886 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.886 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.886 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.886 DEBUG:boto:Params: {} 2024-03-04T17:53:39.886 DEBUG:boto:Token: None 2024-03-04T17:53:39.886 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj 2024-03-04T17:53:39.887 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ= 2024-03-04T17:53:39.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2ivTIAEM5EFay33ZjG/F6wM5TUQ=', 'Content-Length': '0'} 2024-03-04T17:53:39.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000708d981afdc5847f-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:39.892 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.893 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.893 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.893 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.893 DEBUG:boto:Method: GET 2024-03-04T17:53:39.893 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.893 DEBUG:boto:Data: 2024-03-04T17:53:39.893 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.893 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.893 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.893 DEBUG:boto:Params: {} 2024-03-04T17:53:39.894 DEBUG:boto:Token: None 2024-03-04T17:53:39.894 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.894 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk= 2024-03-04T17:53:39.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk=', 'Content-Length': '0'} 2024-03-04T17:53:39.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026e90bc88065acaa-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.944 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:39.944 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:39.944 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.944 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.944 DEBUG:boto:Method: GET 2024-03-04T17:53:39.945 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.945 DEBUG:boto:Data: 2024-03-04T17:53:39.945 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.945 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:39.945 DEBUG:boto:Port: 8000 2024-03-04T17:53:39.945 DEBUG:boto:Params: {} 2024-03-04T17:53:39.945 DEBUG:boto:Token: None 2024-03-04T17:53:39.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:39.945 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk= 2024-03-04T17:53:39.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jlgFP0LjZaskHBH8kJZMOjJCfkk=', 'Content-Length': '0'} 2024-03-04T17:53:39.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aad4da76cd54862b-0065e60aa3-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.992 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.992 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.992 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.992 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.992 DEBUG:boto:Method: GET 2024-03-04T17:53:39.993 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:39.993 DEBUG:boto:Data: 2024-03-04T17:53:39.993 DEBUG:boto:Headers: {} 2024-03-04T17:53:39.993 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:39.993 DEBUG:boto:Port: 8001 2024-03-04T17:53:39.993 DEBUG:boto:Params: {} 2024-03-04T17:53:39.993 DEBUG:boto:Token: None 2024-03-04T17:53:39.993 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:39 GMT /ppfahn-22/ 2024-03-04T17:53:39.993 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY= 2024-03-04T17:53:39.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XIt0mvdfBBVUQ6wQPTMYSqPDPtY=', 'Content-Length': '0'} 2024-03-04T17:53:39.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081940dbcb7d5be71-0065e60aa3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:39.999 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:39.999 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:39.999 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:39.999 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:39.999 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.000 DEBUG:boto:Method: GET 2024-03-04T17:53:40.000 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.000 DEBUG:boto:Data: 2024-03-04T17:53:40.000 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.000 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.000 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.000 DEBUG:boto:Params: {} 2024-03-04T17:53:40.000 DEBUG:boto:Token: None 2024-03-04T17:53:40.000 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.001 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c22e507f54979960-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.006 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:40.007 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.007 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.007 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.007 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.007 DEBUG:boto:Method: GET 2024-03-04T17:53:40.007 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.007 DEBUG:boto:Data: 2024-03-04T17:53:40.007 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.008 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.008 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.008 DEBUG:boto:Params: {} 2024-03-04T17:53:40.008 DEBUG:boto:Token: None 2024-03-04T17:53:40.008 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.008 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fMYb6m0xEQg9np1k96RH6Y7CCeI= 2024-03-04T17:53:40.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fMYb6m0xEQg9np1k96RH6Y7CCeI=', 'Content-Length': '0'} 2024-03-04T17:53:40.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bdcc6000725a76a-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.052 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.052 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.052 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.052 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.052 DEBUG:boto:Method: GET 2024-03-04T17:53:40.053 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.053 DEBUG:boto:Data: 2024-03-04T17:53:40.053 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.053 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.053 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.053 DEBUG:boto:Params: {} 2024-03-04T17:53:40.053 DEBUG:boto:Token: None 2024-03-04T17:53:40.053 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.053 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fMYb6m0xEQg9np1k96RH6Y7CCeI= 2024-03-04T17:53:40.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fMYb6m0xEQg9np1k96RH6Y7CCeI=', 'Content-Length': '0'} 2024-03-04T17:53:40.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bde9c3ac04b663b5-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.100 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.100 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.100 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.100 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.100 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.101 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.101 DEBUG:boto:Data: 2024-03-04T17:53:40.101 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.101 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.101 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.101 DEBUG:boto:Params: {} 2024-03-04T17:53:40.101 DEBUG:boto:Token: None 2024-03-04T17:53:40.101 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.101 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4ALpjUIcgn3WL+WqPle2CgDCsI4= 2024-03-04T17:53:40.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4ALpjUIcgn3WL+WqPle2CgDCsI4=', 'Content-Length': '0'} 2024-03-04T17:53:40.104 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000afcf49116451d515-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.105 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.105 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.105 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.105 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.105 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.105 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.105 DEBUG:boto:Data: 2024-03-04T17:53:40.105 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.105 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.106 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.106 DEBUG:boto:Params: {} 2024-03-04T17:53:40.106 DEBUG:boto:Token: None 2024-03-04T17:53:40.106 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.106 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:4ALpjUIcgn3WL+WqPle2CgDCsI4= 2024-03-04T17:53:40.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4ALpjUIcgn3WL+WqPle2CgDCsI4=', 'Content-Length': '0'} 2024-03-04T17:53:40.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005752bb9b13ebebf8-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:40.109 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.109 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.109 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.109 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.109 DEBUG:boto:Method: GET 2024-03-04T17:53:40.110 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.110 DEBUG:boto:Data: 2024-03-04T17:53:40.110 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.110 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.110 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.110 DEBUG:boto:Params: {} 2024-03-04T17:53:40.110 DEBUG:boto:Token: None 2024-03-04T17:53:40.110 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.110 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fMYb6m0xEQg9np1k96RH6Y7CCeI= 2024-03-04T17:53:40.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fMYb6m0xEQg9np1k96RH6Y7CCeI=', 'Content-Length': '0'} 2024-03-04T17:53:40.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea73d5898bc4d8de-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.156 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.156 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.156 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.156 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.157 DEBUG:boto:Method: GET 2024-03-04T17:53:40.157 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.157 DEBUG:boto:Data: 2024-03-04T17:53:40.157 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.157 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.157 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.157 DEBUG:boto:Params: {} 2024-03-04T17:53:40.157 DEBUG:boto:Token: None 2024-03-04T17:53:40.157 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:40.158 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:fMYb6m0xEQg9np1k96RH6Y7CCeI= 2024-03-04T17:53:40.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fMYb6m0xEQg9np1k96RH6Y7CCeI=', 'Content-Length': '0'} 2024-03-04T17:53:40.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6df3111b2a38009-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.204 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.204 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.204 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.204 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.204 DEBUG:boto:Method: GET 2024-03-04T17:53:40.205 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.205 DEBUG:boto:Data: 2024-03-04T17:53:40.205 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.205 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.205 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.205 DEBUG:boto:Params: {} 2024-03-04T17:53:40.205 DEBUG:boto:Token: None 2024-03-04T17:53:40.205 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.205 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a60f4fe2565074d9-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.211 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.211 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.211 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.211 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.211 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.212 DEBUG:boto:Method: GET 2024-03-04T17:53:40.212 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.212 DEBUG:boto:Data: 2024-03-04T17:53:40.212 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.212 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.212 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.212 DEBUG:boto:Params: {} 2024-03-04T17:53:40.212 DEBUG:boto:Token: None 2024-03-04T17:53:40.212 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.213 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071637afa2eb136d0-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.217 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.217 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.218 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.218 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.218 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:40.218 DEBUG:boto:Data: 2024-03-04T17:53:40.218 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.218 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.218 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.218 DEBUG:boto:Params: {} 2024-03-04T17:53:40.218 DEBUG:boto:Token: None 2024-03-04T17:53:40.219 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj 2024-03-04T17:53:40.219 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AG+Sfi8wfUiQtDNg6R+wD0cehfo= 2024-03-04T17:53:40.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AG+Sfi8wfUiQtDNg6R+wD0cehfo=', 'Content-Length': '0'} 2024-03-04T17:53:40.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b72cb412d3148f92-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.225 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.225 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.225 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.225 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:40.226 DEBUG:boto:Data: 2024-03-04T17:53:40.226 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.226 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.226 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.226 DEBUG:boto:Params: {} 2024-03-04T17:53:40.226 DEBUG:boto:Token: None 2024-03-04T17:53:40.226 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj 2024-03-04T17:53:40.226 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AG+Sfi8wfUiQtDNg6R+wD0cehfo= 2024-03-04T17:53:40.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AG+Sfi8wfUiQtDNg6R+wD0cehfo=', 'Content-Length': '0'} 2024-03-04T17:53:40.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053644a13ece701ea-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:40.233 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.233 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.233 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.233 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.233 DEBUG:boto:Method: GET 2024-03-04T17:53:40.233 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.233 DEBUG:boto:Data: 2024-03-04T17:53:40.233 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.233 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.233 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.234 DEBUG:boto:Params: {} 2024-03-04T17:53:40.234 DEBUG:boto:Token: None 2024-03-04T17:53:40.234 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.234 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1aTuncJlmU8Q6XIYvMAAUx0gqbY= 2024-03-04T17:53:40.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1aTuncJlmU8Q6XIYvMAAUx0gqbY=', 'Content-Length': '0'} 2024-03-04T17:53:40.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077fb307f386f06ab-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.237 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.237 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.237 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.237 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.238 DEBUG:boto:Method: GET 2024-03-04T17:53:40.238 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.238 DEBUG:boto:Data: 2024-03-04T17:53:40.238 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.238 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.238 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.238 DEBUG:boto:Params: {} 2024-03-04T17:53:40.238 DEBUG:boto:Token: None 2024-03-04T17:53:40.238 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.239 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1aTuncJlmU8Q6XIYvMAAUx0gqbY= 2024-03-04T17:53:40.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1aTuncJlmU8Q6XIYvMAAUx0gqbY=', 'Content-Length': '0'} 2024-03-04T17:53:40.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093d25a07dee2817b-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.284 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.284 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.284 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.284 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.284 DEBUG:boto:Method: GET 2024-03-04T17:53:40.285 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.285 DEBUG:boto:Data: 2024-03-04T17:53:40.285 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.285 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.285 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.285 DEBUG:boto:Params: {} 2024-03-04T17:53:40.285 DEBUG:boto:Token: None 2024-03-04T17:53:40.285 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.285 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a7b2de770197ab3-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.291 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.291 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.291 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.291 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.291 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.291 DEBUG:boto:Method: GET 2024-03-04T17:53:40.292 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.292 DEBUG:boto:Data: 2024-03-04T17:53:40.292 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.292 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.292 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.292 DEBUG:boto:Params: {} 2024-03-04T17:53:40.292 DEBUG:boto:Token: None 2024-03-04T17:53:40.292 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.292 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0012d7a9aa02a58-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.298 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:40.298 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.298 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.298 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.298 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.298 DEBUG:boto:Method: GET 2024-03-04T17:53:40.299 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.299 DEBUG:boto:Data: 2024-03-04T17:53:40.299 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.299 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.299 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.299 DEBUG:boto:Params: {} 2024-03-04T17:53:40.299 DEBUG:boto:Token: None 2024-03-04T17:53:40.299 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.299 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LZVXBA5JyrCm3Gwm8lngajrR7VY= 2024-03-04T17:53:40.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LZVXBA5JyrCm3Gwm8lngajrR7VY=', 'Content-Length': '0'} 2024-03-04T17:53:40.303 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc39da4983708a99-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.348 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.348 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.348 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.348 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.348 DEBUG:boto:Method: GET 2024-03-04T17:53:40.349 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.349 DEBUG:boto:Data: 2024-03-04T17:53:40.349 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.349 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.349 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.349 DEBUG:boto:Params: {} 2024-03-04T17:53:40.349 DEBUG:boto:Token: None 2024-03-04T17:53:40.350 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.350 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LZVXBA5JyrCm3Gwm8lngajrR7VY= 2024-03-04T17:53:40.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LZVXBA5JyrCm3Gwm8lngajrR7VY=', 'Content-Length': '0'} 2024-03-04T17:53:40.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dda5291215c5cab0-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.353 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.353 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.353 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.353 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.353 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.353 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.354 DEBUG:boto:Data: 2024-03-04T17:53:40.354 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.354 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.354 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.354 DEBUG:boto:Params: {} 2024-03-04T17:53:40.354 DEBUG:boto:Token: None 2024-03-04T17:53:40.354 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.354 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mJYyPX0Utc8YmWvfAbyHiBCUsV8= 2024-03-04T17:53:40.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mJYyPX0Utc8YmWvfAbyHiBCUsV8=', 'Content-Length': '0'} 2024-03-04T17:53:40.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000730d4f06914ae9a2-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.358 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.358 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.358 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.358 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.358 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.358 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.358 DEBUG:boto:Data: 2024-03-04T17:53:40.358 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.358 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.358 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.359 DEBUG:boto:Params: {} 2024-03-04T17:53:40.359 DEBUG:boto:Token: None 2024-03-04T17:53:40.359 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.359 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mJYyPX0Utc8YmWvfAbyHiBCUsV8= 2024-03-04T17:53:40.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mJYyPX0Utc8YmWvfAbyHiBCUsV8=', 'Content-Length': '0'} 2024-03-04T17:53:40.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3681ab561c6024d-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:40.362 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.362 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.362 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.362 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.362 DEBUG:boto:Method: GET 2024-03-04T17:53:40.363 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.363 DEBUG:boto:Data: 2024-03-04T17:53:40.363 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.363 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.363 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.363 DEBUG:boto:Params: {} 2024-03-04T17:53:40.363 DEBUG:boto:Token: None 2024-03-04T17:53:40.363 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.363 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LZVXBA5JyrCm3Gwm8lngajrR7VY= 2024-03-04T17:53:40.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LZVXBA5JyrCm3Gwm8lngajrR7VY=', 'Content-Length': '0'} 2024-03-04T17:53:40.367 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036ef2fde46a36d76-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.408 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.408 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.408 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.408 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.408 DEBUG:boto:Method: GET 2024-03-04T17:53:40.409 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.409 DEBUG:boto:Data: 2024-03-04T17:53:40.409 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.409 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.409 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.409 DEBUG:boto:Params: {} 2024-03-04T17:53:40.409 DEBUG:boto:Token: None 2024-03-04T17:53:40.409 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:40.410 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LZVXBA5JyrCm3Gwm8lngajrR7VY= 2024-03-04T17:53:40.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LZVXBA5JyrCm3Gwm8lngajrR7VY=', 'Content-Length': '0'} 2024-03-04T17:53:40.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfdfda458369bc3d-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.413 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.413 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.413 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.413 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.413 DEBUG:boto:Method: GET 2024-03-04T17:53:40.413 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.413 DEBUG:boto:Data: 2024-03-04T17:53:40.413 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.413 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.414 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.414 DEBUG:boto:Params: {} 2024-03-04T17:53:40.414 DEBUG:boto:Token: None 2024-03-04T17:53:40.414 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.414 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be68d46c2f03bd5d-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.460 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.461 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.461 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.461 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.461 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.461 DEBUG:boto:Method: GET 2024-03-04T17:53:40.461 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.461 DEBUG:boto:Data: 2024-03-04T17:53:40.461 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.461 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.462 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.462 DEBUG:boto:Params: {} 2024-03-04T17:53:40.462 DEBUG:boto:Token: None 2024-03-04T17:53:40.462 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.462 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef95ed0179a20ed2-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.467 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.468 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.468 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.468 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.468 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:40.468 DEBUG:boto:Data: 2024-03-04T17:53:40.468 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.468 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.468 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.468 DEBUG:boto:Params: {} 2024-03-04T17:53:40.469 DEBUG:boto:Token: None 2024-03-04T17:53:40.469 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj 2024-03-04T17:53:40.469 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AG+Sfi8wfUiQtDNg6R+wD0cehfo= 2024-03-04T17:53:40.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AG+Sfi8wfUiQtDNg6R+wD0cehfo=', 'Content-Length': '0'} 2024-03-04T17:53:40.475 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000068a7cabc1d4146e-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.475 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.475 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.475 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.475 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:40.476 DEBUG:boto:Data: 2024-03-04T17:53:40.476 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.476 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.476 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.476 DEBUG:boto:Params: {} 2024-03-04T17:53:40.476 DEBUG:boto:Token: None 2024-03-04T17:53:40.476 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj 2024-03-04T17:53:40.476 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AG+Sfi8wfUiQtDNg6R+wD0cehfo= 2024-03-04T17:53:40.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AG+Sfi8wfUiQtDNg6R+wD0cehfo=', 'Content-Length': '0'} 2024-03-04T17:53:40.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f73a69b65b06f6b-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:40.482 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.482 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.483 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.483 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.483 DEBUG:boto:Method: GET 2024-03-04T17:53:40.483 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.483 DEBUG:boto:Data: 2024-03-04T17:53:40.483 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.483 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.483 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.483 DEBUG:boto:Params: {} 2024-03-04T17:53:40.483 DEBUG:boto:Token: None 2024-03-04T17:53:40.483 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.484 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1aTuncJlmU8Q6XIYvMAAUx0gqbY= 2024-03-04T17:53:40.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1aTuncJlmU8Q6XIYvMAAUx0gqbY=', 'Content-Length': '0'} 2024-03-04T17:53:40.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e1b87600a3d4cfe-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.487 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.487 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.488 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.488 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.488 DEBUG:boto:Method: GET 2024-03-04T17:53:40.488 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.488 DEBUG:boto:Data: 2024-03-04T17:53:40.488 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.488 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.488 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.488 DEBUG:boto:Params: {} 2024-03-04T17:53:40.489 DEBUG:boto:Token: None 2024-03-04T17:53:40.489 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.489 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1aTuncJlmU8Q6XIYvMAAUx0gqbY= 2024-03-04T17:53:40.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1aTuncJlmU8Q6XIYvMAAUx0gqbY=', 'Content-Length': '0'} 2024-03-04T17:53:40.492 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000871c6606166ce5dd-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.536 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.536 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.536 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.536 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.536 DEBUG:boto:Method: GET 2024-03-04T17:53:40.537 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.537 DEBUG:boto:Data: 2024-03-04T17:53:40.537 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.537 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.537 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.537 DEBUG:boto:Params: {} 2024-03-04T17:53:40.537 DEBUG:boto:Token: None 2024-03-04T17:53:40.537 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.537 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e84534cd0e09eaa4-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.543 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.543 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.543 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.544 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.544 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.544 DEBUG:boto:Method: GET 2024-03-04T17:53:40.544 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.544 DEBUG:boto:Data: 2024-03-04T17:53:40.544 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.544 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.544 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.544 DEBUG:boto:Params: {} 2024-03-04T17:53:40.544 DEBUG:boto:Token: None 2024-03-04T17:53:40.545 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.545 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bb7722bc2759ded-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.550 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:40.550 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.550 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.550 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.550 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.550 DEBUG:boto:Method: GET 2024-03-04T17:53:40.550 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.551 DEBUG:boto:Data: 2024-03-04T17:53:40.551 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.551 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.551 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.551 DEBUG:boto:Params: {} 2024-03-04T17:53:40.551 DEBUG:boto:Token: None 2024-03-04T17:53:40.551 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.551 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9X66EnWA5Zej9RsXGg4nMM04f1k= 2024-03-04T17:53:40.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9X66EnWA5Zej9RsXGg4nMM04f1k=', 'Content-Length': '0'} 2024-03-04T17:53:40.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec3422dcf4ecef4d-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.600 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.600 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.600 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.600 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.600 DEBUG:boto:Method: GET 2024-03-04T17:53:40.601 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.601 DEBUG:boto:Data: 2024-03-04T17:53:40.601 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.601 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.601 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.601 DEBUG:boto:Params: {} 2024-03-04T17:53:40.601 DEBUG:boto:Token: None 2024-03-04T17:53:40.601 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.601 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9X66EnWA5Zej9RsXGg4nMM04f1k= 2024-03-04T17:53:40.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9X66EnWA5Zej9RsXGg4nMM04f1k=', 'Content-Length': '0'} 2024-03-04T17:53:40.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bf6fae8f7ca20af-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.652 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.652 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.652 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.652 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.652 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.652 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.653 DEBUG:boto:Data: 2024-03-04T17:53:40.653 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.653 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.653 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.653 DEBUG:boto:Params: {} 2024-03-04T17:53:40.653 DEBUG:boto:Token: None 2024-03-04T17:53:40.653 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.653 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SVYcZcyW9uSa7e8SpPVkZwbgmMQ= 2024-03-04T17:53:40.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SVYcZcyW9uSa7e8SpPVkZwbgmMQ=', 'Content-Length': '0'} 2024-03-04T17:53:40.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cc6eee53dc1f503-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.657 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.657 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.657 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.657 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.657 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.657 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.658 DEBUG:boto:Data: 2024-03-04T17:53:40.658 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.658 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.658 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.658 DEBUG:boto:Params: {} 2024-03-04T17:53:40.658 DEBUG:boto:Token: None 2024-03-04T17:53:40.658 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.658 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SVYcZcyW9uSa7e8SpPVkZwbgmMQ= 2024-03-04T17:53:40.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SVYcZcyW9uSa7e8SpPVkZwbgmMQ=', 'Content-Length': '0'} 2024-03-04T17:53:40.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032a6ba75759cbdb7-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:40.662 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.662 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.662 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.662 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.662 DEBUG:boto:Method: GET 2024-03-04T17:53:40.662 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.663 DEBUG:boto:Data: 2024-03-04T17:53:40.663 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.663 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.663 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.663 DEBUG:boto:Params: {} 2024-03-04T17:53:40.663 DEBUG:boto:Token: None 2024-03-04T17:53:40.663 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.663 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9X66EnWA5Zej9RsXGg4nMM04f1k= 2024-03-04T17:53:40.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9X66EnWA5Zej9RsXGg4nMM04f1k=', 'Content-Length': '0'} 2024-03-04T17:53:40.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b384dcbee16a102-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.666 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.667 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.667 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.667 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.667 DEBUG:boto:Method: GET 2024-03-04T17:53:40.667 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.667 DEBUG:boto:Data: 2024-03-04T17:53:40.667 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.667 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.667 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.667 DEBUG:boto:Params: {} 2024-03-04T17:53:40.667 DEBUG:boto:Token: None 2024-03-04T17:53:40.668 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:40.668 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9X66EnWA5Zej9RsXGg4nMM04f1k= 2024-03-04T17:53:40.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9X66EnWA5Zej9RsXGg4nMM04f1k=', 'Content-Length': '0'} 2024-03-04T17:53:40.671 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014cbc7f2ca397c39-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.712 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.712 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.712 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.712 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.713 DEBUG:boto:Method: GET 2024-03-04T17:53:40.713 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.713 DEBUG:boto:Data: 2024-03-04T17:53:40.713 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.713 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.713 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.713 DEBUG:boto:Params: {} 2024-03-04T17:53:40.713 DEBUG:boto:Token: None 2024-03-04T17:53:40.713 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.713 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2b7cfba98723619-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.719 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.719 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.719 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.719 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.719 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.720 DEBUG:boto:Method: GET 2024-03-04T17:53:40.720 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.720 DEBUG:boto:Data: 2024-03-04T17:53:40.720 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.720 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.720 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.720 DEBUG:boto:Params: {} 2024-03-04T17:53:40.721 DEBUG:boto:Token: None 2024-03-04T17:53:40.721 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.721 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014d7a6e460bd8294-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.726 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.726 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.726 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.726 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.726 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:40.726 DEBUG:boto:Data: 2024-03-04T17:53:40.726 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.727 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.727 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.727 DEBUG:boto:Params: {} 2024-03-04T17:53:40.727 DEBUG:boto:Token: None 2024-03-04T17:53:40.727 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj 2024-03-04T17:53:40.727 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AG+Sfi8wfUiQtDNg6R+wD0cehfo= 2024-03-04T17:53:40.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AG+Sfi8wfUiQtDNg6R+wD0cehfo=', 'Content-Length': '0'} 2024-03-04T17:53:40.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005dd260414612be44-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.734 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.734 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.734 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.734 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:40.734 DEBUG:boto:Data: 2024-03-04T17:53:40.734 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.735 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.735 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.735 DEBUG:boto:Params: {} 2024-03-04T17:53:40.735 DEBUG:boto:Token: None 2024-03-04T17:53:40.735 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj 2024-03-04T17:53:40.735 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AG+Sfi8wfUiQtDNg6R+wD0cehfo= 2024-03-04T17:53:40.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AG+Sfi8wfUiQtDNg6R+wD0cehfo=', 'Content-Length': '0'} 2024-03-04T17:53:40.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064583d1aa8c7e854-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:40.742 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.742 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.742 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.742 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.742 DEBUG:boto:Method: GET 2024-03-04T17:53:40.743 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.743 DEBUG:boto:Data: 2024-03-04T17:53:40.743 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.743 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.743 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.743 DEBUG:boto:Params: {} 2024-03-04T17:53:40.743 DEBUG:boto:Token: None 2024-03-04T17:53:40.743 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.743 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1aTuncJlmU8Q6XIYvMAAUx0gqbY= 2024-03-04T17:53:40.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1aTuncJlmU8Q6XIYvMAAUx0gqbY=', 'Content-Length': '0'} 2024-03-04T17:53:40.746 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f27464c42539377c-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.788 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.788 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.788 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.788 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.788 DEBUG:boto:Method: GET 2024-03-04T17:53:40.789 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.789 DEBUG:boto:Data: 2024-03-04T17:53:40.789 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.789 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.789 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.789 DEBUG:boto:Params: {} 2024-03-04T17:53:40.789 DEBUG:boto:Token: None 2024-03-04T17:53:40.789 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:40.789 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1aTuncJlmU8Q6XIYvMAAUx0gqbY= 2024-03-04T17:53:40.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1aTuncJlmU8Q6XIYvMAAUx0gqbY=', 'Content-Length': '0'} 2024-03-04T17:53:40.792 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000584434a7eaaa91b2-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.836 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.836 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.836 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.836 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.836 DEBUG:boto:Method: GET 2024-03-04T17:53:40.837 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.837 DEBUG:boto:Data: 2024-03-04T17:53:40.837 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.837 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.837 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.837 DEBUG:boto:Params: {} 2024-03-04T17:53:40.837 DEBUG:boto:Token: None 2024-03-04T17:53:40.837 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.837 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078f845580341d7c3-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.842 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.843 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.843 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.843 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.843 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.843 DEBUG:boto:Method: GET 2024-03-04T17:53:40.843 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.843 DEBUG:boto:Data: 2024-03-04T17:53:40.843 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.843 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.843 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.843 DEBUG:boto:Params: {} 2024-03-04T17:53:40.844 DEBUG:boto:Token: None 2024-03-04T17:53:40.844 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.844 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba2dbab0b7c35653-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.849 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:40.849 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.849 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.849 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.850 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.850 DEBUG:boto:Method: GET 2024-03-04T17:53:40.850 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.850 DEBUG:boto:Data: 2024-03-04T17:53:40.850 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.850 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.850 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.850 DEBUG:boto:Params: {} 2024-03-04T17:53:40.850 DEBUG:boto:Token: None 2024-03-04T17:53:40.850 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.851 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kR3KHjQgbNIuIu0tikE5mBtiKg4= 2024-03-04T17:53:40.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kR3KHjQgbNIuIu0tikE5mBtiKg4=', 'Content-Length': '0'} 2024-03-04T17:53:40.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000776f610004d6ccc0-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.896 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.896 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.896 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.897 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.897 DEBUG:boto:Method: GET 2024-03-04T17:53:40.897 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.897 DEBUG:boto:Data: 2024-03-04T17:53:40.897 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.897 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.897 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.897 DEBUG:boto:Params: {} 2024-03-04T17:53:40.897 DEBUG:boto:Token: None 2024-03-04T17:53:40.898 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.898 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kR3KHjQgbNIuIu0tikE5mBtiKg4= 2024-03-04T17:53:40.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kR3KHjQgbNIuIu0tikE5mBtiKg4=', 'Content-Length': '0'} 2024-03-04T17:53:40.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe5cd25386795dae-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.901 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.901 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.902 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.902 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.902 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.902 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.902 DEBUG:boto:Data: 2024-03-04T17:53:40.902 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.902 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.902 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.902 DEBUG:boto:Params: {} 2024-03-04T17:53:40.902 DEBUG:boto:Token: None 2024-03-04T17:53:40.903 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.903 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s7H7Hb+KriiMeD0eX+4nLl3CYyA= 2024-03-04T17:53:40.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s7H7Hb+KriiMeD0eX+4nLl3CYyA=', 'Content-Length': '0'} 2024-03-04T17:53:40.906 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a8efc6af88066e1-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.906 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.906 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.906 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.906 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.907 DEBUG:boto:Method: HEAD 2024-03-04T17:53:40.907 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.907 DEBUG:boto:Data: 2024-03-04T17:53:40.907 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.907 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.907 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.907 DEBUG:boto:Params: {} 2024-03-04T17:53:40.907 DEBUG:boto:Token: None 2024-03-04T17:53:40.907 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.907 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s7H7Hb+KriiMeD0eX+4nLl3CYyA= 2024-03-04T17:53:40.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s7H7Hb+KriiMeD0eX+4nLl3CYyA=', 'Content-Length': '0'} 2024-03-04T17:53:40.910 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a489ad81839821e5-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:40.910 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.911 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.911 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.911 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.911 DEBUG:boto:Method: GET 2024-03-04T17:53:40.911 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.911 DEBUG:boto:Data: 2024-03-04T17:53:40.911 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.911 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.911 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.911 DEBUG:boto:Params: {} 2024-03-04T17:53:40.912 DEBUG:boto:Token: None 2024-03-04T17:53:40.912 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.912 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kR3KHjQgbNIuIu0tikE5mBtiKg4= 2024-03-04T17:53:40.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kR3KHjQgbNIuIu0tikE5mBtiKg4=', 'Content-Length': '0'} 2024-03-04T17:53:40.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8059da2314cd47c-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.956 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:40.956 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:40.956 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.956 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.956 DEBUG:boto:Method: GET 2024-03-04T17:53:40.957 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.957 DEBUG:boto:Data: 2024-03-04T17:53:40.957 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.957 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.957 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.957 DEBUG:boto:Params: {} 2024-03-04T17:53:40.957 DEBUG:boto:Token: None 2024-03-04T17:53:40.957 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:40.957 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kR3KHjQgbNIuIu0tikE5mBtiKg4= 2024-03-04T17:53:40.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kR3KHjQgbNIuIu0tikE5mBtiKg4=', 'Content-Length': '0'} 2024-03-04T17:53:40.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6b0a9f2d60b93e3-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.961 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.961 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.961 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.961 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.961 DEBUG:boto:Method: GET 2024-03-04T17:53:40.962 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.962 DEBUG:boto:Data: 2024-03-04T17:53:40.962 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.962 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:40.962 DEBUG:boto:Port: 8001 2024-03-04T17:53:40.962 DEBUG:boto:Params: {} 2024-03-04T17:53:40.962 DEBUG:boto:Token: None 2024-03-04T17:53:40.962 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.962 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a637d0a89bfe7d8-0065e60aa4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:40.967 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:40.968 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:40.968 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:40.968 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.968 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:40.968 DEBUG:boto:Method: GET 2024-03-04T17:53:40.969 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:40.969 DEBUG:boto:Data: 2024-03-04T17:53:40.969 DEBUG:boto:Headers: {} 2024-03-04T17:53:40.969 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:40.969 DEBUG:boto:Port: 8000 2024-03-04T17:53:40.969 DEBUG:boto:Params: {} 2024-03-04T17:53:40.969 DEBUG:boto:Token: None 2024-03-04T17:53:40.969 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:40 GMT /ppfahn-22/ 2024-03-04T17:53:40.969 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc= 2024-03-04T17:53:40.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:40 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jH+dDe/8TIhBZCErDeuAyjbn+uc=', 'Content-Length': '0'} 2024-03-04T17:53:40.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064c80b107f7e53e3-0065e60aa4-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.016 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:41.016 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.016 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.017 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.017 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:41.017 DEBUG:boto:Data: 2024-03-04T17:53:41.017 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.017 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.017 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.017 DEBUG:boto:Params: {} 2024-03-04T17:53:41.017 DEBUG:boto:Token: None 2024-03-04T17:53:41.017 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj 2024-03-04T17:53:41.017 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY= 2024-03-04T17:53:41.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY=', 'Content-Length': '0'} 2024-03-04T17:53:41.024 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d1183851d5f1d9f-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.024 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.024 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.024 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.024 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:41.025 DEBUG:boto:Data: 2024-03-04T17:53:41.025 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.025 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.025 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.025 DEBUG:boto:Params: {} 2024-03-04T17:53:41.025 DEBUG:boto:Token: None 2024-03-04T17:53:41.025 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj 2024-03-04T17:53:41.025 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY= 2024-03-04T17:53:41.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY=', 'Content-Length': '0'} 2024-03-04T17:53:41.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e8aca5b59acb885-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:41.031 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.031 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.032 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.032 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.032 DEBUG:boto:Method: GET 2024-03-04T17:53:41.032 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.032 DEBUG:boto:Data: 2024-03-04T17:53:41.032 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.032 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.032 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.032 DEBUG:boto:Params: {} 2024-03-04T17:53:41.032 DEBUG:boto:Token: None 2024-03-04T17:53:41.033 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.033 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw= 2024-03-04T17:53:41.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw=', 'Content-Length': '0'} 2024-03-04T17:53:41.036 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005629aa2b3a20d20d-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.036 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.036 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.037 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.037 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.037 DEBUG:boto:Method: GET 2024-03-04T17:53:41.037 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.037 DEBUG:boto:Data: 2024-03-04T17:53:41.037 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.037 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.037 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.037 DEBUG:boto:Params: {} 2024-03-04T17:53:41.037 DEBUG:boto:Token: None 2024-03-04T17:53:41.038 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.038 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw= 2024-03-04T17:53:41.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw=', 'Content-Length': '0'} 2024-03-04T17:53:41.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6facb73ba9df469-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.084 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:41.084 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:41.084 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.084 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.084 DEBUG:boto:Method: GET 2024-03-04T17:53:41.085 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:41.085 DEBUG:boto:Data: 2024-03-04T17:53:41.085 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.085 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.085 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.085 DEBUG:boto:Params: {} 2024-03-04T17:53:41.085 DEBUG:boto:Token: None 2024-03-04T17:53:41.085 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/ 2024-03-04T17:53:41.086 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA= 2024-03-04T17:53:41.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA=', 'Content-Length': '0'} 2024-03-04T17:53:41.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c868ff5e290a493a-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.091 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:41.091 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:41.091 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:41.092 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.092 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.092 DEBUG:boto:Method: GET 2024-03-04T17:53:41.092 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:41.092 DEBUG:boto:Data: 2024-03-04T17:53:41.092 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.092 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.092 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.092 DEBUG:boto:Params: {} 2024-03-04T17:53:41.093 DEBUG:boto:Token: None 2024-03-04T17:53:41.093 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/ 2024-03-04T17:53:41.093 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA= 2024-03-04T17:53:41.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA=', 'Content-Length': '0'} 2024-03-04T17:53:41.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d1ffd8669fbcdf6-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.140 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:41.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:41.140 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.141 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.141 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.141 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.141 DEBUG:boto:Method: GET 2024-03-04T17:53:41.141 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.141 DEBUG:boto:Data: 2024-03-04T17:53:41.141 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.141 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.141 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.141 DEBUG:boto:Params: {} 2024-03-04T17:53:41.142 DEBUG:boto:Token: None 2024-03-04T17:53:41.142 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.142 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XffJnLOOLZI7ZQhlIrcdpsmB5HA= 2024-03-04T17:53:41.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XffJnLOOLZI7ZQhlIrcdpsmB5HA=', 'Content-Length': '0'} 2024-03-04T17:53:41.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc8f9eaf7fba7bd5-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.145 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.145 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.145 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.145 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.146 DEBUG:boto:Method: GET 2024-03-04T17:53:41.146 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.146 DEBUG:boto:Data: 2024-03-04T17:53:41.146 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.146 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.146 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.146 DEBUG:boto:Params: {} 2024-03-04T17:53:41.146 DEBUG:boto:Token: None 2024-03-04T17:53:41.146 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.146 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XffJnLOOLZI7ZQhlIrcdpsmB5HA= 2024-03-04T17:53:41.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XffJnLOOLZI7ZQhlIrcdpsmB5HA=', 'Content-Length': '0'} 2024-03-04T17:53:41.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c417073a77742c2-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.196 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.196 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.196 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.196 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.196 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.197 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.197 DEBUG:boto:Data: 2024-03-04T17:53:41.197 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.197 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.197 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.197 DEBUG:boto:Params: {} 2024-03-04T17:53:41.197 DEBUG:boto:Token: None 2024-03-04T17:53:41.197 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.197 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Uz/iw/6TVDBQwlWwpqvyGQCDE9Y= 2024-03-04T17:53:41.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Uz/iw/6TVDBQwlWwpqvyGQCDE9Y=', 'Content-Length': '0'} 2024-03-04T17:53:41.200 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd891b29b28bcbde-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.201 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.201 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.201 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.201 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.201 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.201 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.201 DEBUG:boto:Data: 2024-03-04T17:53:41.201 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.201 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.202 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.202 DEBUG:boto:Params: {} 2024-03-04T17:53:41.202 DEBUG:boto:Token: None 2024-03-04T17:53:41.202 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.202 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Uz/iw/6TVDBQwlWwpqvyGQCDE9Y= 2024-03-04T17:53:41.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Uz/iw/6TVDBQwlWwpqvyGQCDE9Y=', 'Content-Length': '0'} 2024-03-04T17:53:41.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd2073fb46be0862-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:41.205 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.205 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.206 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.206 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.206 DEBUG:boto:Method: GET 2024-03-04T17:53:41.206 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.206 DEBUG:boto:Data: 2024-03-04T17:53:41.206 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.206 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.206 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.206 DEBUG:boto:Params: {} 2024-03-04T17:53:41.206 DEBUG:boto:Token: None 2024-03-04T17:53:41.207 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.207 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XffJnLOOLZI7ZQhlIrcdpsmB5HA= 2024-03-04T17:53:41.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XffJnLOOLZI7ZQhlIrcdpsmB5HA=', 'Content-Length': '0'} 2024-03-04T17:53:41.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e5237a3963dc4a1-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.252 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.252 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.252 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.252 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.252 DEBUG:boto:Method: GET 2024-03-04T17:53:41.252 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.253 DEBUG:boto:Data: 2024-03-04T17:53:41.253 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.253 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.253 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.253 DEBUG:boto:Params: {} 2024-03-04T17:53:41.253 DEBUG:boto:Token: None 2024-03-04T17:53:41.253 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:41.253 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XffJnLOOLZI7ZQhlIrcdpsmB5HA= 2024-03-04T17:53:41.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XffJnLOOLZI7ZQhlIrcdpsmB5HA=', 'Content-Length': '0'} 2024-03-04T17:53:41.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab5762df3edd6e66-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.300 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:41.300 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:41.300 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.300 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.301 DEBUG:boto:Method: GET 2024-03-04T17:53:41.301 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:41.301 DEBUG:boto:Data: 2024-03-04T17:53:41.301 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.301 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.301 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.301 DEBUG:boto:Params: {} 2024-03-04T17:53:41.301 DEBUG:boto:Token: None 2024-03-04T17:53:41.301 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/ 2024-03-04T17:53:41.301 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA= 2024-03-04T17:53:41.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA=', 'Content-Length': '0'} 2024-03-04T17:53:41.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd90f4c179299aa9-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.307 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:41.308 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:41.308 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:41.308 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.308 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.308 DEBUG:boto:Method: GET 2024-03-04T17:53:41.308 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:41.308 DEBUG:boto:Data: 2024-03-04T17:53:41.308 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.308 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.308 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.309 DEBUG:boto:Params: {} 2024-03-04T17:53:41.309 DEBUG:boto:Token: None 2024-03-04T17:53:41.309 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/ 2024-03-04T17:53:41.309 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA= 2024-03-04T17:53:41.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA=', 'Content-Length': '0'} 2024-03-04T17:53:41.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b5639e5068f5612-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.313 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:41.314 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.314 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.314 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.314 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:41.314 DEBUG:boto:Data: 2024-03-04T17:53:41.314 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.314 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.314 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.314 DEBUG:boto:Params: {} 2024-03-04T17:53:41.315 DEBUG:boto:Token: None 2024-03-04T17:53:41.315 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj 2024-03-04T17:53:41.315 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY= 2024-03-04T17:53:41.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY=', 'Content-Length': '0'} 2024-03-04T17:53:41.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5288c589d8ca217-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.321 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.321 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.321 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.322 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:41.322 DEBUG:boto:Data: 2024-03-04T17:53:41.322 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.322 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.322 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.322 DEBUG:boto:Params: {} 2024-03-04T17:53:41.322 DEBUG:boto:Token: None 2024-03-04T17:53:41.322 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj 2024-03-04T17:53:41.322 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY= 2024-03-04T17:53:41.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY=', 'Content-Length': '0'} 2024-03-04T17:53:41.328 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000507a7c3070f610aa-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:41.328 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.329 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.329 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.329 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.329 DEBUG:boto:Method: GET 2024-03-04T17:53:41.329 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.329 DEBUG:boto:Data: 2024-03-04T17:53:41.329 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.329 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.329 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.329 DEBUG:boto:Params: {} 2024-03-04T17:53:41.330 DEBUG:boto:Token: None 2024-03-04T17:53:41.330 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.330 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw= 2024-03-04T17:53:41.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw=', 'Content-Length': '0'} 2024-03-04T17:53:41.333 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086b693e9de7993b7-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.333 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.333 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.333 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.333 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.333 DEBUG:boto:Method: GET 2024-03-04T17:53:41.333 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.334 DEBUG:boto:Data: 2024-03-04T17:53:41.334 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.334 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.334 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.334 DEBUG:boto:Params: {} 2024-03-04T17:53:41.334 DEBUG:boto:Token: None 2024-03-04T17:53:41.334 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.334 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw= 2024-03-04T17:53:41.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw=', 'Content-Length': '0'} 2024-03-04T17:53:41.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be661713c754b845-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.380 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:41.380 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:41.380 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.380 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.380 DEBUG:boto:Method: GET 2024-03-04T17:53:41.381 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:41.381 DEBUG:boto:Data: 2024-03-04T17:53:41.381 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.381 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.381 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.381 DEBUG:boto:Params: {} 2024-03-04T17:53:41.381 DEBUG:boto:Token: None 2024-03-04T17:53:41.381 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/ 2024-03-04T17:53:41.381 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA= 2024-03-04T17:53:41.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA=', 'Content-Length': '0'} 2024-03-04T17:53:41.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d90c69de97c6e4f0-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.388 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:41.389 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:41.389 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:41.389 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.389 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.389 DEBUG:boto:Method: GET 2024-03-04T17:53:41.389 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:41.389 DEBUG:boto:Data: 2024-03-04T17:53:41.389 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.389 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.389 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.390 DEBUG:boto:Params: {} 2024-03-04T17:53:41.390 DEBUG:boto:Token: None 2024-03-04T17:53:41.390 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/ 2024-03-04T17:53:41.390 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA= 2024-03-04T17:53:41.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA=', 'Content-Length': '0'} 2024-03-04T17:53:41.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6d77ee1a03be103-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.394 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:41.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:41.395 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.395 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.395 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.395 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.395 DEBUG:boto:Method: GET 2024-03-04T17:53:41.395 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.395 DEBUG:boto:Data: 2024-03-04T17:53:41.395 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.396 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.396 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.396 DEBUG:boto:Params: {} 2024-03-04T17:53:41.396 DEBUG:boto:Token: None 2024-03-04T17:53:41.396 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.396 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RmXg5WGTjoYpSbsjJ2T3Hwt2icc= 2024-03-04T17:53:41.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RmXg5WGTjoYpSbsjJ2T3Hwt2icc=', 'Content-Length': '0'} 2024-03-04T17:53:41.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000000ba717f9df94fc-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.440 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.440 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.440 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.440 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.440 DEBUG:boto:Method: GET 2024-03-04T17:53:41.440 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.441 DEBUG:boto:Data: 2024-03-04T17:53:41.441 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.441 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.441 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.441 DEBUG:boto:Params: {} 2024-03-04T17:53:41.441 DEBUG:boto:Token: None 2024-03-04T17:53:41.441 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.441 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RmXg5WGTjoYpSbsjJ2T3Hwt2icc= 2024-03-04T17:53:41.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RmXg5WGTjoYpSbsjJ2T3Hwt2icc=', 'Content-Length': '0'} 2024-03-04T17:53:41.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcf0f7976c5dede1-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.488 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.488 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.488 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.488 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.488 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.489 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.489 DEBUG:boto:Data: 2024-03-04T17:53:41.489 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.489 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.489 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.489 DEBUG:boto:Params: {} 2024-03-04T17:53:41.489 DEBUG:boto:Token: None 2024-03-04T17:53:41.489 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.489 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dg43PhFqhkvNxYQmgafoD+aKKJ8= 2024-03-04T17:53:41.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dg43PhFqhkvNxYQmgafoD+aKKJ8=', 'Content-Length': '0'} 2024-03-04T17:53:41.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a1f71a86440d31f-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.493 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.493 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.493 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.493 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.493 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.493 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.494 DEBUG:boto:Data: 2024-03-04T17:53:41.494 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.494 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.494 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.494 DEBUG:boto:Params: {} 2024-03-04T17:53:41.494 DEBUG:boto:Token: None 2024-03-04T17:53:41.494 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.494 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dg43PhFqhkvNxYQmgafoD+aKKJ8= 2024-03-04T17:53:41.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dg43PhFqhkvNxYQmgafoD+aKKJ8=', 'Content-Length': '0'} 2024-03-04T17:53:41.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000164da240ac788251-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:41.498 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.498 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.498 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.498 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.498 DEBUG:boto:Method: GET 2024-03-04T17:53:41.498 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.499 DEBUG:boto:Data: 2024-03-04T17:53:41.499 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.499 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.499 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.499 DEBUG:boto:Params: {} 2024-03-04T17:53:41.499 DEBUG:boto:Token: None 2024-03-04T17:53:41.499 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.499 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RmXg5WGTjoYpSbsjJ2T3Hwt2icc= 2024-03-04T17:53:41.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RmXg5WGTjoYpSbsjJ2T3Hwt2icc=', 'Content-Length': '0'} 2024-03-04T17:53:41.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cd5f7ba339f1f2c-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.548 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.548 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.548 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.548 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.548 DEBUG:boto:Method: GET 2024-03-04T17:53:41.549 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.549 DEBUG:boto:Data: 2024-03-04T17:53:41.549 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.549 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.549 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.549 DEBUG:boto:Params: {} 2024-03-04T17:53:41.549 DEBUG:boto:Token: None 2024-03-04T17:53:41.549 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:41.549 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:RmXg5WGTjoYpSbsjJ2T3Hwt2icc= 2024-03-04T17:53:41.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RmXg5WGTjoYpSbsjJ2T3Hwt2icc=', 'Content-Length': '0'} 2024-03-04T17:53:41.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ec5da1b83ce75d8-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.596 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:41.596 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:41.596 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.596 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.596 DEBUG:boto:Method: GET 2024-03-04T17:53:41.597 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:41.597 DEBUG:boto:Data: 2024-03-04T17:53:41.597 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.597 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.597 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.597 DEBUG:boto:Params: {} 2024-03-04T17:53:41.597 DEBUG:boto:Token: None 2024-03-04T17:53:41.597 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/ 2024-03-04T17:53:41.598 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA= 2024-03-04T17:53:41.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA=', 'Content-Length': '0'} 2024-03-04T17:53:41.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086e93106d380d54d-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.603 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:41.603 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:41.603 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:41.604 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.604 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.604 DEBUG:boto:Method: GET 2024-03-04T17:53:41.604 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:41.604 DEBUG:boto:Data: 2024-03-04T17:53:41.604 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.604 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.604 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.604 DEBUG:boto:Params: {} 2024-03-04T17:53:41.604 DEBUG:boto:Token: None 2024-03-04T17:53:41.605 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/ 2024-03-04T17:53:41.605 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA= 2024-03-04T17:53:41.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA=', 'Content-Length': '0'} 2024-03-04T17:53:41.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bcefcd2f5356c2a-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.610 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:41.610 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.610 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.611 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.611 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:41.611 DEBUG:boto:Data: 2024-03-04T17:53:41.611 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.611 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.611 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.611 DEBUG:boto:Params: {} 2024-03-04T17:53:41.611 DEBUG:boto:Token: None 2024-03-04T17:53:41.611 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj 2024-03-04T17:53:41.612 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY= 2024-03-04T17:53:41.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY=', 'Content-Length': '0'} 2024-03-04T17:53:41.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000125dec6d5f22b4a4-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.618 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.619 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.619 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.619 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:41.619 DEBUG:boto:Data: 2024-03-04T17:53:41.619 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.619 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.619 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.619 DEBUG:boto:Params: {} 2024-03-04T17:53:41.619 DEBUG:boto:Token: None 2024-03-04T17:53:41.620 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj 2024-03-04T17:53:41.620 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY= 2024-03-04T17:53:41.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY=', 'Content-Length': '0'} 2024-03-04T17:53:41.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b53c66b9cc09a6db-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:41.626 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.626 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.627 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.627 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.627 DEBUG:boto:Method: GET 2024-03-04T17:53:41.627 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.627 DEBUG:boto:Data: 2024-03-04T17:53:41.627 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.627 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.627 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.627 DEBUG:boto:Params: {} 2024-03-04T17:53:41.627 DEBUG:boto:Token: None 2024-03-04T17:53:41.628 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.628 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw= 2024-03-04T17:53:41.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw=', 'Content-Length': '0'} 2024-03-04T17:53:41.631 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008c02f5bdfca0f38-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.672 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.672 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.672 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.672 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.672 DEBUG:boto:Method: GET 2024-03-04T17:53:41.673 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.673 DEBUG:boto:Data: 2024-03-04T17:53:41.673 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.673 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.673 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.673 DEBUG:boto:Params: {} 2024-03-04T17:53:41.673 DEBUG:boto:Token: None 2024-03-04T17:53:41.673 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.673 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw= 2024-03-04T17:53:41.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw=', 'Content-Length': '0'} 2024-03-04T17:53:41.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3c8daf2601c7943-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.720 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:41.720 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:41.720 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.720 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.721 DEBUG:boto:Method: GET 2024-03-04T17:53:41.721 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:41.721 DEBUG:boto:Data: 2024-03-04T17:53:41.721 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.721 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.721 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.721 DEBUG:boto:Params: {} 2024-03-04T17:53:41.721 DEBUG:boto:Token: None 2024-03-04T17:53:41.721 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/ 2024-03-04T17:53:41.721 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA= 2024-03-04T17:53:41.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA=', 'Content-Length': '0'} 2024-03-04T17:53:41.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f5891bfc2317e6e-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.727 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:41.727 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:41.727 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:41.727 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.728 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.728 DEBUG:boto:Method: GET 2024-03-04T17:53:41.728 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:41.728 DEBUG:boto:Data: 2024-03-04T17:53:41.728 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.728 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.728 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.728 DEBUG:boto:Params: {} 2024-03-04T17:53:41.728 DEBUG:boto:Token: None 2024-03-04T17:53:41.728 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/ 2024-03-04T17:53:41.729 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA= 2024-03-04T17:53:41.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA=', 'Content-Length': '0'} 2024-03-04T17:53:41.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d86a3ac0eae0c4e-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.734 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:41.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:41.734 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.734 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.734 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.735 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.735 DEBUG:boto:Method: GET 2024-03-04T17:53:41.735 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.735 DEBUG:boto:Data: 2024-03-04T17:53:41.735 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.735 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.735 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.735 DEBUG:boto:Params: {} 2024-03-04T17:53:41.735 DEBUG:boto:Token: None 2024-03-04T17:53:41.735 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.736 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:i6AWvi89LEQLbNPyNvFuDiVflwQ= 2024-03-04T17:53:41.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:i6AWvi89LEQLbNPyNvFuDiVflwQ=', 'Content-Length': '0'} 2024-03-04T17:53:41.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016b4e0af9f431a2b-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.780 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.780 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.780 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.780 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.780 DEBUG:boto:Method: GET 2024-03-04T17:53:41.781 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.781 DEBUG:boto:Data: 2024-03-04T17:53:41.781 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.781 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.781 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.781 DEBUG:boto:Params: {} 2024-03-04T17:53:41.781 DEBUG:boto:Token: None 2024-03-04T17:53:41.781 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.781 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:i6AWvi89LEQLbNPyNvFuDiVflwQ= 2024-03-04T17:53:41.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:i6AWvi89LEQLbNPyNvFuDiVflwQ=', 'Content-Length': '0'} 2024-03-04T17:53:41.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2b72f1b66127b8c-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.828 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.828 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.828 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.828 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.828 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.829 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.829 DEBUG:boto:Data: 2024-03-04T17:53:41.829 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.829 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.829 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.829 DEBUG:boto:Params: {} 2024-03-04T17:53:41.829 DEBUG:boto:Token: None 2024-03-04T17:53:41.829 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.829 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aT4TwbkCPYDF/1jFhb4m5D5mQgk= 2024-03-04T17:53:41.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aT4TwbkCPYDF/1jFhb4m5D5mQgk=', 'Content-Length': '0'} 2024-03-04T17:53:41.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d70ff211539ad19-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.833 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.833 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.833 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.833 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.833 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.833 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.834 DEBUG:boto:Data: 2024-03-04T17:53:41.834 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.834 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.834 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.834 DEBUG:boto:Params: {} 2024-03-04T17:53:41.834 DEBUG:boto:Token: None 2024-03-04T17:53:41.834 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.834 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aT4TwbkCPYDF/1jFhb4m5D5mQgk= 2024-03-04T17:53:41.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aT4TwbkCPYDF/1jFhb4m5D5mQgk=', 'Content-Length': '0'} 2024-03-04T17:53:41.837 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e26118b71fba843d-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:41.838 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.838 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.838 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.838 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.838 DEBUG:boto:Method: GET 2024-03-04T17:53:41.838 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.838 DEBUG:boto:Data: 2024-03-04T17:53:41.838 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.838 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.838 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.839 DEBUG:boto:Params: {} 2024-03-04T17:53:41.839 DEBUG:boto:Token: None 2024-03-04T17:53:41.839 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.839 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:i6AWvi89LEQLbNPyNvFuDiVflwQ= 2024-03-04T17:53:41.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:i6AWvi89LEQLbNPyNvFuDiVflwQ=', 'Content-Length': '0'} 2024-03-04T17:53:41.842 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073a0cf43b68d7481-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.884 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.884 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.884 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.884 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.885 DEBUG:boto:Method: GET 2024-03-04T17:53:41.885 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.885 DEBUG:boto:Data: 2024-03-04T17:53:41.885 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.885 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.885 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.885 DEBUG:boto:Params: {} 2024-03-04T17:53:41.885 DEBUG:boto:Token: None 2024-03-04T17:53:41.885 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:41.886 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:i6AWvi89LEQLbNPyNvFuDiVflwQ= 2024-03-04T17:53:41.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:i6AWvi89LEQLbNPyNvFuDiVflwQ=', 'Content-Length': '0'} 2024-03-04T17:53:41.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d3faff2ab73abe6-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.932 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:41.932 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:41.932 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.932 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.933 DEBUG:boto:Method: GET 2024-03-04T17:53:41.933 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:41.933 DEBUG:boto:Data: 2024-03-04T17:53:41.933 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.933 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.933 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.933 DEBUG:boto:Params: {} 2024-03-04T17:53:41.933 DEBUG:boto:Token: None 2024-03-04T17:53:41.933 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/ 2024-03-04T17:53:41.933 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA= 2024-03-04T17:53:41.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA=', 'Content-Length': '0'} 2024-03-04T17:53:41.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001876e55c69bcfcfb-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.938 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:41.939 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:41.939 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:41.939 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.939 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:41.939 DEBUG:boto:Method: GET 2024-03-04T17:53:41.939 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:41.939 DEBUG:boto:Data: 2024-03-04T17:53:41.939 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.939 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.939 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.939 DEBUG:boto:Params: {} 2024-03-04T17:53:41.940 DEBUG:boto:Token: None 2024-03-04T17:53:41.940 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/ 2024-03-04T17:53:41.940 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA= 2024-03-04T17:53:41.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K6zs/nTILviIQceRniEDgDR2anA=', 'Content-Length': '0'} 2024-03-04T17:53:41.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000422a0f50a5ffe93b-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.945 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:41.945 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.945 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.946 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.946 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:41.946 DEBUG:boto:Data: 2024-03-04T17:53:41.946 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.946 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.946 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.946 DEBUG:boto:Params: {} 2024-03-04T17:53:41.946 DEBUG:boto:Token: None 2024-03-04T17:53:41.946 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj 2024-03-04T17:53:41.947 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY= 2024-03-04T17:53:41.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY=', 'Content-Length': '0'} 2024-03-04T17:53:41.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000401a05b6587310f4-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.953 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.953 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.953 DEBUG:boto:Method: HEAD 2024-03-04T17:53:41.954 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:41.954 DEBUG:boto:Data: 2024-03-04T17:53:41.954 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.954 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.954 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.954 DEBUG:boto:Params: {} 2024-03-04T17:53:41.954 DEBUG:boto:Token: None 2024-03-04T17:53:41.954 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj 2024-03-04T17:53:41.954 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY= 2024-03-04T17:53:41.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DFnPmLS59eIRCUdTHgorKGmBqkY=', 'Content-Length': '0'} 2024-03-04T17:53:41.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000915307f57f154e12-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:41.961 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.961 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.961 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.962 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.962 DEBUG:boto:Method: GET 2024-03-04T17:53:41.962 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.962 DEBUG:boto:Data: 2024-03-04T17:53:41.962 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.962 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:41.962 DEBUG:boto:Port: 8001 2024-03-04T17:53:41.962 DEBUG:boto:Params: {} 2024-03-04T17:53:41.962 DEBUG:boto:Token: None 2024-03-04T17:53:41.963 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.963 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw= 2024-03-04T17:53:41.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw=', 'Content-Length': '0'} 2024-03-04T17:53:41.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e10734fdbc40815-0065e60aa5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:41.966 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:41.966 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:41.966 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.966 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.966 DEBUG:boto:Method: GET 2024-03-04T17:53:41.966 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.967 DEBUG:boto:Data: 2024-03-04T17:53:41.967 DEBUG:boto:Headers: {} 2024-03-04T17:53:41.967 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:41.967 DEBUG:boto:Port: 8000 2024-03-04T17:53:41.967 DEBUG:boto:Params: {} 2024-03-04T17:53:41.967 DEBUG:boto:Token: None 2024-03-04T17:53:41.967 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:41 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:41.967 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw= 2024-03-04T17:53:41.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:41 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lmMFPkcFK+Jd6dM6ZRWhq7rNlqw=', 'Content-Length': '0'} 2024-03-04T17:53:41.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094cff250ed1de299-0065e60aa5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.012 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.012 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.012 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.012 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.012 DEBUG:boto:Method: GET 2024-03-04T17:53:42.013 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:42.013 DEBUG:boto:Data: 2024-03-04T17:53:42.013 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.013 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.013 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.013 DEBUG:boto:Params: {} 2024-03-04T17:53:42.013 DEBUG:boto:Token: None 2024-03-04T17:53:42.013 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.013 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2b67f080bc59b68-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.019 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.019 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.019 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.019 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.019 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.020 DEBUG:boto:Method: GET 2024-03-04T17:53:42.020 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:42.020 DEBUG:boto:Data: 2024-03-04T17:53:42.020 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.020 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.020 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.020 DEBUG:boto:Params: {} 2024-03-04T17:53:42.020 DEBUG:boto:Token: None 2024-03-04T17:53:42.020 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.021 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000836771f36b389b91-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.028 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:42.028 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.028 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.028 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.028 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.029 DEBUG:boto:Method: GET 2024-03-04T17:53:42.029 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.029 DEBUG:boto:Data: 2024-03-04T17:53:42.029 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.029 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.029 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.029 DEBUG:boto:Params: {} 2024-03-04T17:53:42.029 DEBUG:boto:Token: None 2024-03-04T17:53:42.029 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.030 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s7Lm8szSv38/aZVea+H37n+rbms= 2024-03-04T17:53:42.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s7Lm8szSv38/aZVea+H37n+rbms=', 'Content-Length': '0'} 2024-03-04T17:53:42.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045c79fa2d2659aa4-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.076 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.076 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.076 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.076 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.076 DEBUG:boto:Method: GET 2024-03-04T17:53:42.076 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.077 DEBUG:boto:Data: 2024-03-04T17:53:42.077 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.077 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.077 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.077 DEBUG:boto:Params: {} 2024-03-04T17:53:42.077 DEBUG:boto:Token: None 2024-03-04T17:53:42.077 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.077 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s7Lm8szSv38/aZVea+H37n+rbms= 2024-03-04T17:53:42.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s7Lm8szSv38/aZVea+H37n+rbms=', 'Content-Length': '0'} 2024-03-04T17:53:42.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094929eecf9b79b72-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.082 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.082 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.082 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.082 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.082 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.082 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.082 DEBUG:boto:Data: 2024-03-04T17:53:42.082 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.082 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.083 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.083 DEBUG:boto:Params: {} 2024-03-04T17:53:42.083 DEBUG:boto:Token: None 2024-03-04T17:53:42.083 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.083 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IPjZbYPPt1W/zpAfUIYcy/svckE= 2024-03-04T17:53:42.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IPjZbYPPt1W/zpAfUIYcy/svckE=', 'Content-Length': '0'} 2024-03-04T17:53:42.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d585ccaab06c15f0-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.093 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.093 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.093 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.093 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.093 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.093 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.093 DEBUG:boto:Data: 2024-03-04T17:53:42.093 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.094 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.094 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.094 DEBUG:boto:Params: {} 2024-03-04T17:53:42.094 DEBUG:boto:Token: None 2024-03-04T17:53:42.094 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.094 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IPjZbYPPt1W/zpAfUIYcy/svckE= 2024-03-04T17:53:42.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IPjZbYPPt1W/zpAfUIYcy/svckE=', 'Content-Length': '0'} 2024-03-04T17:53:42.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008509ec8f08ede257-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:42.098 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.098 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.098 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.098 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.098 DEBUG:boto:Method: GET 2024-03-04T17:53:42.098 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.098 DEBUG:boto:Data: 2024-03-04T17:53:42.098 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.099 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.099 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.099 DEBUG:boto:Params: {} 2024-03-04T17:53:42.099 DEBUG:boto:Token: None 2024-03-04T17:53:42.099 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.099 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s7Lm8szSv38/aZVea+H37n+rbms= 2024-03-04T17:53:42.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s7Lm8szSv38/aZVea+H37n+rbms=', 'Content-Length': '0'} 2024-03-04T17:53:42.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ef72afa6e70fd52-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.144 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.144 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.144 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.144 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.144 DEBUG:boto:Method: GET 2024-03-04T17:53:42.144 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.145 DEBUG:boto:Data: 2024-03-04T17:53:42.145 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.145 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.145 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.145 DEBUG:boto:Params: {} 2024-03-04T17:53:42.145 DEBUG:boto:Token: None 2024-03-04T17:53:42.145 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:42.145 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:s7Lm8szSv38/aZVea+H37n+rbms= 2024-03-04T17:53:42.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s7Lm8szSv38/aZVea+H37n+rbms=', 'Content-Length': '0'} 2024-03-04T17:53:42.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038581d23e062957f-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.149 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.149 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.149 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.149 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.149 DEBUG:boto:Method: GET 2024-03-04T17:53:42.149 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:42.150 DEBUG:boto:Data: 2024-03-04T17:53:42.150 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.150 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.150 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.150 DEBUG:boto:Params: {} 2024-03-04T17:53:42.150 DEBUG:boto:Token: None 2024-03-04T17:53:42.150 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.150 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81c20a6bcda0160-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.196 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.196 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.196 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.197 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.197 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.197 DEBUG:boto:Method: GET 2024-03-04T17:53:42.197 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:42.197 DEBUG:boto:Data: 2024-03-04T17:53:42.197 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.197 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.197 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.197 DEBUG:boto:Params: {} 2024-03-04T17:53:42.197 DEBUG:boto:Token: None 2024-03-04T17:53:42.198 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.198 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a34da759f11e4a4-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.202 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.202 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.202 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.202 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.202 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:42.203 DEBUG:boto:Data: 2024-03-04T17:53:42.203 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.203 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.203 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.203 DEBUG:boto:Params: {} 2024-03-04T17:53:42.203 DEBUG:boto:Token: None 2024-03-04T17:53:42.203 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj 2024-03-04T17:53:42.203 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l9wBIajfI4nC7IK17NG6jyLtk1A= 2024-03-04T17:53:42.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l9wBIajfI4nC7IK17NG6jyLtk1A=', 'Content-Length': '0'} 2024-03-04T17:53:42.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd980e6c5d5924a6-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.209 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.209 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.210 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.210 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:42.210 DEBUG:boto:Data: 2024-03-04T17:53:42.210 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.210 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.210 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.210 DEBUG:boto:Params: {} 2024-03-04T17:53:42.210 DEBUG:boto:Token: None 2024-03-04T17:53:42.210 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj 2024-03-04T17:53:42.211 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l9wBIajfI4nC7IK17NG6jyLtk1A= 2024-03-04T17:53:42.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l9wBIajfI4nC7IK17NG6jyLtk1A=', 'Content-Length': '0'} 2024-03-04T17:53:42.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004247b053d7f4fd37-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:42.217 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.217 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.217 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.217 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.217 DEBUG:boto:Method: GET 2024-03-04T17:53:42.217 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.217 DEBUG:boto:Data: 2024-03-04T17:53:42.217 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.217 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.218 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.218 DEBUG:boto:Params: {} 2024-03-04T17:53:42.218 DEBUG:boto:Token: None 2024-03-04T17:53:42.218 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.218 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs= 2024-03-04T17:53:42.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs=', 'Content-Length': '0'} 2024-03-04T17:53:42.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000dccdb6ed1818ae4-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.221 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.221 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.221 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.221 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.221 DEBUG:boto:Method: GET 2024-03-04T17:53:42.222 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.222 DEBUG:boto:Data: 2024-03-04T17:53:42.222 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.222 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.222 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.222 DEBUG:boto:Params: {} 2024-03-04T17:53:42.222 DEBUG:boto:Token: None 2024-03-04T17:53:42.222 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.222 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs= 2024-03-04T17:53:42.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs=', 'Content-Length': '0'} 2024-03-04T17:53:42.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007287348d4afb571f-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.272 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.272 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.272 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.272 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.272 DEBUG:boto:Method: GET 2024-03-04T17:53:42.273 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:42.273 DEBUG:boto:Data: 2024-03-04T17:53:42.273 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.273 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.273 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.273 DEBUG:boto:Params: {} 2024-03-04T17:53:42.273 DEBUG:boto:Token: None 2024-03-04T17:53:42.273 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.273 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b8db4083a172457-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.278 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.278 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.279 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.279 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.279 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.279 DEBUG:boto:Method: GET 2024-03-04T17:53:42.279 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:42.279 DEBUG:boto:Data: 2024-03-04T17:53:42.279 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.279 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.279 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.279 DEBUG:boto:Params: {} 2024-03-04T17:53:42.280 DEBUG:boto:Token: None 2024-03-04T17:53:42.280 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.280 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000481e5548de6a1106-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.285 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:42.285 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.285 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.286 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.286 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.286 DEBUG:boto:Method: GET 2024-03-04T17:53:42.286 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.286 DEBUG:boto:Data: 2024-03-04T17:53:42.286 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.286 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.286 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.286 DEBUG:boto:Params: {} 2024-03-04T17:53:42.286 DEBUG:boto:Token: None 2024-03-04T17:53:42.287 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.287 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ufU5r4yLIENBC93EU5RTalPLuq4= 2024-03-04T17:53:42.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ufU5r4yLIENBC93EU5RTalPLuq4=', 'Content-Length': '0'} 2024-03-04T17:53:42.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bf198bb74fed97b-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.332 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.332 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.332 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.332 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.332 DEBUG:boto:Method: GET 2024-03-04T17:53:42.333 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.333 DEBUG:boto:Data: 2024-03-04T17:53:42.333 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.333 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.333 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.333 DEBUG:boto:Params: {} 2024-03-04T17:53:42.333 DEBUG:boto:Token: None 2024-03-04T17:53:42.333 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.333 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ufU5r4yLIENBC93EU5RTalPLuq4= 2024-03-04T17:53:42.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ufU5r4yLIENBC93EU5RTalPLuq4=', 'Content-Length': '0'} 2024-03-04T17:53:42.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ceb085ba138e480-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.380 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.380 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.380 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.380 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.380 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.380 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.381 DEBUG:boto:Data: 2024-03-04T17:53:42.381 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.381 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.381 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.381 DEBUG:boto:Params: {} 2024-03-04T17:53:42.381 DEBUG:boto:Token: None 2024-03-04T17:53:42.381 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.381 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SAfT8p5ya56PrLJ5uPKw0zD8cP4= 2024-03-04T17:53:42.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SAfT8p5ya56PrLJ5uPKw0zD8cP4=', 'Content-Length': '0'} 2024-03-04T17:53:42.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f8402fba9b032d2-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.385 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.385 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.385 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.385 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.385 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.385 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.385 DEBUG:boto:Data: 2024-03-04T17:53:42.385 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.385 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.385 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.386 DEBUG:boto:Params: {} 2024-03-04T17:53:42.386 DEBUG:boto:Token: None 2024-03-04T17:53:42.386 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.386 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:SAfT8p5ya56PrLJ5uPKw0zD8cP4= 2024-03-04T17:53:42.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SAfT8p5ya56PrLJ5uPKw0zD8cP4=', 'Content-Length': '0'} 2024-03-04T17:53:42.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037d3b7647a0c2d1c-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:42.390 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.390 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.390 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.391 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.391 DEBUG:boto:Method: GET 2024-03-04T17:53:42.391 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.391 DEBUG:boto:Data: 2024-03-04T17:53:42.391 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.391 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.391 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.391 DEBUG:boto:Params: {} 2024-03-04T17:53:42.391 DEBUG:boto:Token: None 2024-03-04T17:53:42.391 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.392 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ufU5r4yLIENBC93EU5RTalPLuq4= 2024-03-04T17:53:42.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ufU5r4yLIENBC93EU5RTalPLuq4=', 'Content-Length': '0'} 2024-03-04T17:53:42.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b263e0c0ace43d50-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.397 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.397 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.397 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.397 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.397 DEBUG:boto:Method: GET 2024-03-04T17:53:42.397 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.397 DEBUG:boto:Data: 2024-03-04T17:53:42.397 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.398 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.398 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.398 DEBUG:boto:Params: {} 2024-03-04T17:53:42.398 DEBUG:boto:Token: None 2024-03-04T17:53:42.398 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:42.398 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ufU5r4yLIENBC93EU5RTalPLuq4= 2024-03-04T17:53:42.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ufU5r4yLIENBC93EU5RTalPLuq4=', 'Content-Length': '0'} 2024-03-04T17:53:42.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f47dbbd5037df03-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.444 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.444 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.444 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.444 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.445 DEBUG:boto:Method: GET 2024-03-04T17:53:42.445 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:42.445 DEBUG:boto:Data: 2024-03-04T17:53:42.445 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.445 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.445 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.445 DEBUG:boto:Params: {} 2024-03-04T17:53:42.445 DEBUG:boto:Token: None 2024-03-04T17:53:42.445 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.445 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d8d9c96a211ff9e-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.450 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.450 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.451 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.451 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.451 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.451 DEBUG:boto:Method: GET 2024-03-04T17:53:42.451 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:42.451 DEBUG:boto:Data: 2024-03-04T17:53:42.451 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.451 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.451 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.451 DEBUG:boto:Params: {} 2024-03-04T17:53:42.452 DEBUG:boto:Token: None 2024-03-04T17:53:42.452 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.452 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041dd3039399e65a1-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.456 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.456 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.457 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.459 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.459 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:42.459 DEBUG:boto:Data: 2024-03-04T17:53:42.459 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.459 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.459 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.459 DEBUG:boto:Params: {} 2024-03-04T17:53:42.459 DEBUG:boto:Token: None 2024-03-04T17:53:42.459 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj 2024-03-04T17:53:42.460 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l9wBIajfI4nC7IK17NG6jyLtk1A= 2024-03-04T17:53:42.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l9wBIajfI4nC7IK17NG6jyLtk1A=', 'Content-Length': '0'} 2024-03-04T17:53:42.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1be465b3f03ec75-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.471 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.471 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.471 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.471 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:42.471 DEBUG:boto:Data: 2024-03-04T17:53:42.471 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.471 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.472 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.472 DEBUG:boto:Params: {} 2024-03-04T17:53:42.472 DEBUG:boto:Token: None 2024-03-04T17:53:42.472 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj 2024-03-04T17:53:42.472 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l9wBIajfI4nC7IK17NG6jyLtk1A= 2024-03-04T17:53:42.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l9wBIajfI4nC7IK17NG6jyLtk1A=', 'Content-Length': '0'} 2024-03-04T17:53:42.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009331ef2f99d6d4c1-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:42.479 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.479 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.479 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.479 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.479 DEBUG:boto:Method: GET 2024-03-04T17:53:42.479 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.480 DEBUG:boto:Data: 2024-03-04T17:53:42.480 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.480 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.480 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.480 DEBUG:boto:Params: {} 2024-03-04T17:53:42.480 DEBUG:boto:Token: None 2024-03-04T17:53:42.480 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.480 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs= 2024-03-04T17:53:42.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs=', 'Content-Length': '0'} 2024-03-04T17:53:42.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f540f2efa11a956e-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.528 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.528 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.528 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.528 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.528 DEBUG:boto:Method: GET 2024-03-04T17:53:42.528 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.529 DEBUG:boto:Data: 2024-03-04T17:53:42.529 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.529 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.529 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.529 DEBUG:boto:Params: {} 2024-03-04T17:53:42.529 DEBUG:boto:Token: None 2024-03-04T17:53:42.529 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.529 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs= 2024-03-04T17:53:42.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs=', 'Content-Length': '0'} 2024-03-04T17:53:42.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffe102381b31666f-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.580 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.580 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.580 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.580 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.580 DEBUG:boto:Method: GET 2024-03-04T17:53:42.580 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:42.581 DEBUG:boto:Data: 2024-03-04T17:53:42.581 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.581 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.581 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.581 DEBUG:boto:Params: {} 2024-03-04T17:53:42.581 DEBUG:boto:Token: None 2024-03-04T17:53:42.581 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.581 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e704f07e4e81bd85-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.584 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.584 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.584 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.584 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.585 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.585 DEBUG:boto:Method: GET 2024-03-04T17:53:42.585 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:42.585 DEBUG:boto:Data: 2024-03-04T17:53:42.585 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.585 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:42.585 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.585 DEBUG:boto:Params: {} 2024-03-04T17:53:42.585 DEBUG:boto:Token: None 2024-03-04T17:53:42.585 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.586 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcf04c7e949de8b4-0065e60aa6-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.588 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.589 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-22 zones={test-zone3, test-zone1} 2024-03-04T17:53:42.589 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-22 zones={test-zone3, test-zone2} 2024-03-04T17:53:42.589 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.589 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.589 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.589 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:53:42.589 DEBUG:boto:Data: 2024-03-04T17:53:42.589 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.589 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.590 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.590 DEBUG:boto:Params: {} 2024-03-04T17:53:42.590 DEBUG:boto:Token: None 2024-03-04T17:53:42.590 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.590 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:u0gIDzOEcNeSh+XijsFbi6OAlcQ= 2024-03-04T17:53:42.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:u0gIDzOEcNeSh+XijsFbi6OAlcQ=', 'Content-Length': '0'} 2024-03-04T17:53:42.592 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045b9ec63cbb68d60-0065e60aa6-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.592 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.592 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.592 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.593 DEBUG:boto:Path: /ppfahn-22/ 2024-03-04T17:53:42.593 DEBUG:boto:Data: 2024-03-04T17:53:42.593 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.593 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.593 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.593 DEBUG:boto:Params: {} 2024-03-04T17:53:42.593 DEBUG:boto:Token: None 2024-03-04T17:53:42.593 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.593 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:u0gIDzOEcNeSh+XijsFbi6OAlcQ= 2024-03-04T17:53:42.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:u0gIDzOEcNeSh+XijsFbi6OAlcQ=', 'Content-Length': '0'} 2024-03-04T17:53:42.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018075e38b758c1c6-0065e60aa6-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.598 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:53:42.598 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.598 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.598 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:42.598 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:42.598 DEBUG:boto:Method: GET 2024-03-04T17:53:42.598 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:42.599 DEBUG:boto:Data: 2024-03-04T17:53:42.599 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.599 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.599 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.599 DEBUG:boto:Params: {} 2024-03-04T17:53:42.599 DEBUG:boto:Token: None 2024-03-04T17:53:42.599 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/?versions 2024-03-04T17:53:42.599 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F2Yp2PVQouBqyVv9Fup9YpkCXmI= 2024-03-04T17:53:42.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F2Yp2PVQouBqyVv9Fup9YpkCXmI=', 'Content-Length': '0'} 2024-03-04T17:53:42.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002815055057cc2588-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.603 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG6tTJPArSLLFYQGx5Z9J94Qp3btVq8wtrue2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQDkS502cdQ0sD3MSSr4AJ.nNDwrboFEfalse2024-03-04T17:51:31.005Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfexW91D920ZLyPSn0KVa8L-0Jf11s3qfalse2024-03-04T17:51:31.057Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4aDzoElev9M4kJpBvZIFgGfNSrnTu1hfalse2024-03-04T17:51:30.946Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrBR7sHkOppK-w8aON9rPlfIadTwy8LKfalse2024-03-04T17:51:30.974Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtp1YBJlmM1cmYYj2xRDAuLHN-EIcD23false2024-03-04T17:51:30.916Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxsfalse2024-03-04T17:51:30.859Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIgnYdWgPuuz09tkj9qWq7P8qzgBSCIyfalse2024-03-04T17:51:30.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZw7l4YYB37p1xa7jtLR7kDu1d5Xjl6Gfalse2024-03-04T17:51:30.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10Vjj0fS8rYkr3OIS1pBSR-3fCH6svyfalse2024-03-04T17:51:30.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXak5SqkkJ7eVzuGI.8.u99lZqCC3tNwfalse2024-03-04T17:51:30.749Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzyjvmlB6z-cVoQlri1b6gVvpnPVUCjFfalse2024-03-04T17:51:30.721Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6BBhxVS-A79T2MI6AHE.THWDO21iftffalse2024-03-04T17:51:30.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7kfalse2024-03-04T17:51:30.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji.eNnYbM.4yY91CfDbCxKIqa-Uec-fxfalse2024-03-04T17:51:30.610Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkUfalse2024-03-04T17:51:30.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje9K3WlwCsFUmkYneucDfMv9rNQO6cdpfalse2024-03-04T17:51:30.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRTfalse2024-03-04T17:51:30.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgRxtOPcYyW7cm-IBYtAu0zyYNScRyoTfalse2024-03-04T17:51:30.416Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjme6.y9cJ8bkVcZ04PNMrwIXjDQUGzXRfalse2024-03-04T17:51:30.386Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoHs8pGuS1bc-GYUbjIaqPQPwQGDukkcfalse2024-03-04T17:51:30.469Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj160IY3UeXb5qH4Rh4JB09PNigxsTa5gfalse2024-03-04T17:51:30.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9WE9mNbplYdU449Vkwh8lwrICYxU6v7false2024-03-04T17:51:30.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZJZPq8HwR1LmdjGaT37eTcVO7gvndKgfalse2024-03-04T17:51:30.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lwBSkQlMAA54TFAsh7J7MCnzo0gTfafalse2024-03-04T17:51:30.157Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq4h-uWPuCpU7FK9SfN-LHLv-vtuKwuEfalse2024-03-04T17:51:30.239Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzKZVhMV59rWXQPf05XlhVIimSaZTr40false2024-03-04T17:51:30.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAcrzzw6Ded5-7UcCqpx-w2RGP5BSbOUfalse2024-03-04T17:51:30.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdVzfZSb65W93TGSPX8waGA5tm4N9lW6false2024-03-04T17:51:30.037Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlwBZf1YBiU2Wqy250Elw-2.f6FuQF0efalse2024-03-04T17:51:30.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.604 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:53:42.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.607 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:53:42.607 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.607 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.607 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:42.607 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:42.607 DEBUG:boto:Method: GET 2024-03-04T17:53:42.607 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:42.607 DEBUG:boto:Data: 2024-03-04T17:53:42.607 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.608 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.608 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.608 DEBUG:boto:Params: {} 2024-03-04T17:53:42.608 DEBUG:boto:Token: None 2024-03-04T17:53:42.608 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/?versions 2024-03-04T17:53:42.608 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F2Yp2PVQouBqyVv9Fup9YpkCXmI= 2024-03-04T17:53:42.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F2Yp2PVQouBqyVv9Fup9YpkCXmI=', 'Content-Length': '0'} 2024-03-04T17:53:42.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099d1624a1bd4f25b-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.614 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG6tTJPArSLLFYQGx5Z9J94Qp3btVq8wtrue2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQDkS502cdQ0sD3MSSr4AJ.nNDwrboFEfalse2024-03-04T17:51:31.005Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfexW91D920ZLyPSn0KVa8L-0Jf11s3qfalse2024-03-04T17:51:31.057Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4aDzoElev9M4kJpBvZIFgGfNSrnTu1hfalse2024-03-04T17:51:30.946Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrBR7sHkOppK-w8aON9rPlfIadTwy8LKfalse2024-03-04T17:51:30.974Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtp1YBJlmM1cmYYj2xRDAuLHN-EIcD23false2024-03-04T17:51:30.916Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxsfalse2024-03-04T17:51:30.859Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIgnYdWgPuuz09tkj9qWq7P8qzgBSCIyfalse2024-03-04T17:51:30.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZw7l4YYB37p1xa7jtLR7kDu1d5Xjl6Gfalse2024-03-04T17:51:30.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10Vjj0fS8rYkr3OIS1pBSR-3fCH6svyfalse2024-03-04T17:51:30.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXak5SqkkJ7eVzuGI.8.u99lZqCC3tNwfalse2024-03-04T17:51:30.749Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzyjvmlB6z-cVoQlri1b6gVvpnPVUCjFfalse2024-03-04T17:51:30.721Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6BBhxVS-A79T2MI6AHE.THWDO21iftffalse2024-03-04T17:51:30.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7kfalse2024-03-04T17:51:30.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji.eNnYbM.4yY91CfDbCxKIqa-Uec-fxfalse2024-03-04T17:51:30.610Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkUfalse2024-03-04T17:51:30.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje9K3WlwCsFUmkYneucDfMv9rNQO6cdpfalse2024-03-04T17:51:30.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRTfalse2024-03-04T17:51:30.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgRxtOPcYyW7cm-IBYtAu0zyYNScRyoTfalse2024-03-04T17:51:30.416Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjme6.y9cJ8bkVcZ04PNMrwIXjDQUGzXRfalse2024-03-04T17:51:30.386Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoHs8pGuS1bc-GYUbjIaqPQPwQGDukkcfalse2024-03-04T17:51:30.469Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj160IY3UeXb5qH4Rh4JB09PNigxsTa5gfalse2024-03-04T17:51:30.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9WE9mNbplYdU449Vkwh8lwrICYxU6v7false2024-03-04T17:51:30.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZJZPq8HwR1LmdjGaT37eTcVO7gvndKgfalse2024-03-04T17:51:30.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lwBSkQlMAA54TFAsh7J7MCnzo0gTfafalse2024-03-04T17:51:30.157Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq4h-uWPuCpU7FK9SfN-LHLv-vtuKwuEfalse2024-03-04T17:51:30.239Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzKZVhMV59rWXQPf05XlhVIimSaZTr40false2024-03-04T17:51:30.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAcrzzw6Ded5-7UcCqpx-w2RGP5BSbOUfalse2024-03-04T17:51:30.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdVzfZSb65W93TGSPX8waGA5tm4N9lW6false2024-03-04T17:51:30.037Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlwBZf1YBiU2Wqy250Elw-2.f6FuQF0efalse2024-03-04T17:51:30.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.616 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T17:53:42.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:53:42.618 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.618 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.619 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:42.619 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:42.619 DEBUG:boto:Method: GET 2024-03-04T17:53:42.619 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:42.619 DEBUG:boto:Data: 2024-03-04T17:53:42.619 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.619 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.619 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.619 DEBUG:boto:Params: {} 2024-03-04T17:53:42.619 DEBUG:boto:Token: None 2024-03-04T17:53:42.619 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/?versions 2024-03-04T17:53:42.619 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F2Yp2PVQouBqyVv9Fup9YpkCXmI= 2024-03-04T17:53:42.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F2Yp2PVQouBqyVv9Fup9YpkCXmI=', 'Content-Length': '0'} 2024-03-04T17:53:42.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4087e03a2f3a17d-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.623 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG6tTJPArSLLFYQGx5Z9J94Qp3btVq8wtrue2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQDkS502cdQ0sD3MSSr4AJ.nNDwrboFEfalse2024-03-04T17:51:31.005Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfexW91D920ZLyPSn0KVa8L-0Jf11s3qfalse2024-03-04T17:51:31.057Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4aDzoElev9M4kJpBvZIFgGfNSrnTu1hfalse2024-03-04T17:51:30.946Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrBR7sHkOppK-w8aON9rPlfIadTwy8LKfalse2024-03-04T17:51:30.974Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtp1YBJlmM1cmYYj2xRDAuLHN-EIcD23false2024-03-04T17:51:30.916Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxsfalse2024-03-04T17:51:30.859Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIgnYdWgPuuz09tkj9qWq7P8qzgBSCIyfalse2024-03-04T17:51:30.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZw7l4YYB37p1xa7jtLR7kDu1d5Xjl6Gfalse2024-03-04T17:51:30.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10Vjj0fS8rYkr3OIS1pBSR-3fCH6svyfalse2024-03-04T17:51:30.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXak5SqkkJ7eVzuGI.8.u99lZqCC3tNwfalse2024-03-04T17:51:30.749Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzyjvmlB6z-cVoQlri1b6gVvpnPVUCjFfalse2024-03-04T17:51:30.721Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6BBhxVS-A79T2MI6AHE.THWDO21iftffalse2024-03-04T17:51:30.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7kfalse2024-03-04T17:51:30.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji.eNnYbM.4yY91CfDbCxKIqa-Uec-fxfalse2024-03-04T17:51:30.610Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkUfalse2024-03-04T17:51:30.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje9K3WlwCsFUmkYneucDfMv9rNQO6cdpfalse2024-03-04T17:51:30.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRTfalse2024-03-04T17:51:30.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgRxtOPcYyW7cm-IBYtAu0zyYNScRyoTfalse2024-03-04T17:51:30.416Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjme6.y9cJ8bkVcZ04PNMrwIXjDQUGzXRfalse2024-03-04T17:51:30.386Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoHs8pGuS1bc-GYUbjIaqPQPwQGDukkcfalse2024-03-04T17:51:30.469Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj160IY3UeXb5qH4Rh4JB09PNigxsTa5gfalse2024-03-04T17:51:30.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9WE9mNbplYdU449Vkwh8lwrICYxU6v7false2024-03-04T17:51:30.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZJZPq8HwR1LmdjGaT37eTcVO7gvndKgfalse2024-03-04T17:51:30.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lwBSkQlMAA54TFAsh7J7MCnzo0gTfafalse2024-03-04T17:51:30.157Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq4h-uWPuCpU7FK9SfN-LHLv-vtuKwuEfalse2024-03-04T17:51:30.239Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzKZVhMV59rWXQPf05XlhVIimSaZTr40false2024-03-04T17:51:30.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAcrzzw6Ded5-7UcCqpx-w2RGP5BSbOUfalse2024-03-04T17:51:30.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdVzfZSb65W93TGSPX8waGA5tm4N9lW6false2024-03-04T17:51:30.037Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlwBZf1YBiU2Wqy250Elw-2.f6FuQF0efalse2024-03-04T17:51:30.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.624 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.624 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.624 DEBUG:boto:path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:42.624 DEBUG:boto:auth_path=/ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:42.624 DEBUG:boto:Method: GET 2024-03-04T17:53:42.625 DEBUG:boto:Path: /ppfahn-22/?versions&max-keys=999 2024-03-04T17:53:42.625 DEBUG:boto:Data: 2024-03-04T17:53:42.625 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.625 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.625 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.625 DEBUG:boto:Params: {} 2024-03-04T17:53:42.625 DEBUG:boto:Token: None 2024-03-04T17:53:42.625 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/?versions 2024-03-04T17:53:42.625 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F2Yp2PVQouBqyVv9Fup9YpkCXmI= 2024-03-04T17:53:42.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F2Yp2PVQouBqyVv9Fup9YpkCXmI=', 'Content-Length': '0'} 2024-03-04T17:53:42.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbf8dbb2ee5f8cdd-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.632 DEBUG:boto:b'ppfahn-22999falsedummy5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQtrue2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG6tTJPArSLLFYQGx5Z9J94Qp3btVq8wtrue2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQDkS502cdQ0sD3MSSr4AJ.nNDwrboFEfalse2024-03-04T17:51:31.005Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfexW91D920ZLyPSn0KVa8L-0Jf11s3qfalse2024-03-04T17:51:31.057Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4aDzoElev9M4kJpBvZIFgGfNSrnTu1hfalse2024-03-04T17:51:30.946Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrBR7sHkOppK-w8aON9rPlfIadTwy8LKfalse2024-03-04T17:51:30.974Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtp1YBJlmM1cmYYj2xRDAuLHN-EIcD23false2024-03-04T17:51:30.916Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxsfalse2024-03-04T17:51:30.859Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIgnYdWgPuuz09tkj9qWq7P8qzgBSCIyfalse2024-03-04T17:51:30.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZw7l4YYB37p1xa7jtLR7kDu1d5Xjl6Gfalse2024-03-04T17:51:30.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10Vjj0fS8rYkr3OIS1pBSR-3fCH6svyfalse2024-03-04T17:51:30.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXak5SqkkJ7eVzuGI.8.u99lZqCC3tNwfalse2024-03-04T17:51:30.749Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzyjvmlB6z-cVoQlri1b6gVvpnPVUCjFfalse2024-03-04T17:51:30.721Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6BBhxVS-A79T2MI6AHE.THWDO21iftffalse2024-03-04T17:51:30.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7kfalse2024-03-04T17:51:30.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji.eNnYbM.4yY91CfDbCxKIqa-Uec-fxfalse2024-03-04T17:51:30.610Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkUfalse2024-03-04T17:51:30.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje9K3WlwCsFUmkYneucDfMv9rNQO6cdpfalse2024-03-04T17:51:30.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRTfalse2024-03-04T17:51:30.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgRxtOPcYyW7cm-IBYtAu0zyYNScRyoTfalse2024-03-04T17:51:30.416Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjme6.y9cJ8bkVcZ04PNMrwIXjDQUGzXRfalse2024-03-04T17:51:30.386Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoHs8pGuS1bc-GYUbjIaqPQPwQGDukkcfalse2024-03-04T17:51:30.469Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj160IY3UeXb5qH4Rh4JB09PNigxsTa5gfalse2024-03-04T17:51:30.302Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9WE9mNbplYdU449Vkwh8lwrICYxU6v7false2024-03-04T17:51:30.268Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZJZPq8HwR1LmdjGaT37eTcVO7gvndKgfalse2024-03-04T17:51:30.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lwBSkQlMAA54TFAsh7J7MCnzo0gTfafalse2024-03-04T17:51:30.157Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq4h-uWPuCpU7FK9SfN-LHLv-vtuKwuEfalse2024-03-04T17:51:30.239Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzKZVhMV59rWXQPf05XlhVIimSaZTr40false2024-03-04T17:51:30.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAcrzzw6Ded5-7UcCqpx-w2RGP5BSbOUfalse2024-03-04T17:51:30.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdVzfZSb65W93TGSPX8waGA5tm4N9lW6false2024-03-04T17:51:30.037Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlwBZf1YBiU2Wqy250Elw-2.f6FuQF0efalse2024-03-04T17:51:30.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:53:42.633 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:42.633 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:42.633 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.633 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.634 DEBUG:boto:Method: GET 2024-03-04T17:53:42.634 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.634 DEBUG:boto:Data: 2024-03-04T17:53:42.634 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.634 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.634 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.634 DEBUG:boto:Params: {} 2024-03-04T17:53:42.634 DEBUG:boto:Token: None 2024-03-04T17:53:42.634 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.634 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KhNlMK0uQ+nZySGjQsmg7beYuUM= 2024-03-04T17:53:42.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KhNlMK0uQ+nZySGjQsmg7beYuUM=', 'Content-Length': '0'} 2024-03-04T17:53:42.637 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000089131534e3609877-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.637 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:42.637 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:42.637 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.638 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.638 DEBUG:boto:Method: GET 2024-03-04T17:53:42.638 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.638 DEBUG:boto:Data: 2024-03-04T17:53:42.638 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.638 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.638 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.638 DEBUG:boto:Params: {} 2024-03-04T17:53:42.638 DEBUG:boto:Token: None 2024-03-04T17:53:42.638 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.638 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KhNlMK0uQ+nZySGjQsmg7beYuUM= 2024-03-04T17:53:42.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KhNlMK0uQ+nZySGjQsmg7beYuUM=', 'Content-Length': '0'} 2024-03-04T17:53:42.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000075a788323d20f706-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.641 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:42.641 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:42.642 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.642 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.642 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.642 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.642 DEBUG:boto:Data: 2024-03-04T17:53:42.642 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.642 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.642 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.642 DEBUG:boto:Params: {} 2024-03-04T17:53:42.642 DEBUG:boto:Token: None 2024-03-04T17:53:42.643 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.643 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:niOSZpGyWyHf58SuUMLDUpErTkk= 2024-03-04T17:53:42.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:niOSZpGyWyHf58SuUMLDUpErTkk=', 'Content-Length': '0'} 2024-03-04T17:53:42.645 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000385f2cca7f6f4f27-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.645 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:42.646 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:42.646 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.646 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.646 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.646 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.646 DEBUG:boto:Data: 2024-03-04T17:53:42.646 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.646 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.646 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.646 DEBUG:boto:Params: {} 2024-03-04T17:53:42.646 DEBUG:boto:Token: None 2024-03-04T17:53:42.647 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.647 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:niOSZpGyWyHf58SuUMLDUpErTkk= 2024-03-04T17:53:42.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:niOSZpGyWyHf58SuUMLDUpErTkk=', 'Content-Length': '0'} 2024-03-04T17:53:42.649 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d212208e7fe9bca8-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:53:42.650 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:42.650 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:42.650 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.650 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.650 DEBUG:boto:Method: GET 2024-03-04T17:53:42.650 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.650 DEBUG:boto:Data: 2024-03-04T17:53:42.650 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.650 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.651 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.651 DEBUG:boto:Params: {} 2024-03-04T17:53:42.651 DEBUG:boto:Token: None 2024-03-04T17:53:42.651 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.651 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KhNlMK0uQ+nZySGjQsmg7beYuUM= 2024-03-04T17:53:42.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KhNlMK0uQ+nZySGjQsmg7beYuUM=', 'Content-Length': '0'} 2024-03-04T17:53:42.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000024c46deea5097b73-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.654 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:42.654 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:42.654 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.654 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.654 DEBUG:boto:Method: GET 2024-03-04T17:53:42.654 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.655 DEBUG:boto:Data: 2024-03-04T17:53:42.655 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.655 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.655 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.655 DEBUG:boto:Params: {} 2024-03-04T17:53:42.655 DEBUG:boto:Token: None 2024-03-04T17:53:42.655 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.655 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KhNlMK0uQ+nZySGjQsmg7beYuUM= 2024-03-04T17:53:42.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KhNlMK0uQ+nZySGjQsmg7beYuUM=', 'Content-Length': '0'} 2024-03-04T17:53:42.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba448163e6065f28-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.658 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.658 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.658 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:42.659 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:42.659 DEBUG:boto:Method: GET 2024-03-04T17:53:42.659 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:53:42.659 DEBUG:boto:Data: 2024-03-04T17:53:42.659 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.659 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.659 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.659 DEBUG:boto:Params: {} 2024-03-04T17:53:42.659 DEBUG:boto:Token: None 2024-03-04T17:53:42.659 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.659 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe7ee96a8224cf77-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.704 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.704 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.704 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.704 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:42.705 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:42.705 DEBUG:boto:Method: GET 2024-03-04T17:53:42.705 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:53:42.705 DEBUG:boto:Data: 2024-03-04T17:53:42.705 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.705 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.705 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.705 DEBUG:boto:Params: {} 2024-03-04T17:53:42.705 DEBUG:boto:Token: None 2024-03-04T17:53:42.705 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.705 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed02e5d17e74684e-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.711 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.711 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:42.711 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:42.711 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.711 DEBUG:boto:Path: /ppfahn-22/dummy 2024-03-04T17:53:42.711 DEBUG:boto:Data: 2024-03-04T17:53:42.711 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.711 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.711 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.712 DEBUG:boto:Params: {} 2024-03-04T17:53:42.712 DEBUG:boto:Token: None 2024-03-04T17:53:42.712 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/dummy 2024-03-04T17:53:42.712 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vugiD8DY+h231Vi6ANBnIr/Onjw= 2024-03-04T17:53:42.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vugiD8DY+h231Vi6ANBnIr/Onjw=', 'Content-Length': '0'} 2024-03-04T17:53:42.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b2951621e98627d-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.716 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:42.716 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:42.716 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.716 DEBUG:boto:Path: /ppfahn-22/dummy 2024-03-04T17:53:42.716 DEBUG:boto:Data: 2024-03-04T17:53:42.716 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.716 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.716 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.717 DEBUG:boto:Params: {} 2024-03-04T17:53:42.717 DEBUG:boto:Token: None 2024-03-04T17:53:42.717 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/dummy 2024-03-04T17:53:42.717 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vugiD8DY+h231Vi6ANBnIr/Onjw= 2024-03-04T17:53:42.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vugiD8DY+h231Vi6ANBnIr/Onjw=', 'Content-Length': '0'} 2024-03-04T17:53:42.721 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b684353f6979ab42-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T17:53:42.721 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:42.721 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:42.721 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.722 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.722 DEBUG:boto:Method: GET 2024-03-04T17:53:42.722 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.722 DEBUG:boto:Data: 2024-03-04T17:53:42.722 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.722 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.722 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.722 DEBUG:boto:Params: {} 2024-03-04T17:53:42.722 DEBUG:boto:Token: None 2024-03-04T17:53:42.722 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.722 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KhNlMK0uQ+nZySGjQsmg7beYuUM= 2024-03-04T17:53:42.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KhNlMK0uQ+nZySGjQsmg7beYuUM=', 'Content-Length': '0'} 2024-03-04T17:53:42.725 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb6fea795359b741-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.726 DEBUG:boto:path=/ppfahn-22/dummy 2024-03-04T17:53:42.726 DEBUG:boto:auth_path=/ppfahn-22/dummy 2024-03-04T17:53:42.726 DEBUG:boto:path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.726 DEBUG:boto:auth_path=/ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.726 DEBUG:boto:Method: GET 2024-03-04T17:53:42.726 DEBUG:boto:Path: /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.726 DEBUG:boto:Data: 2024-03-04T17:53:42.726 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.726 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.726 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.726 DEBUG:boto:Params: {} 2024-03-04T17:53:42.726 DEBUG:boto:Token: None 2024-03-04T17:53:42.727 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/dummy?versionId=5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ 2024-03-04T17:53:42.727 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KhNlMK0uQ+nZySGjQsmg7beYuUM= 2024-03-04T17:53:42.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KhNlMK0uQ+nZySGjQsmg7beYuUM=', 'Content-Length': '0'} 2024-03-04T17:53:42.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:50:43 GMT'), ('x-amz-version-id', '5Pp.Sj8Fy6UtuYnJH.kufZqt2RyGgiQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004b4f103dba2b7e1e-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.730 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.730 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.730 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:42.730 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:42.730 DEBUG:boto:Method: GET 2024-03-04T17:53:42.730 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:53:42.731 DEBUG:boto:Data: 2024-03-04T17:53:42.731 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.731 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.731 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.731 DEBUG:boto:Params: {} 2024-03-04T17:53:42.731 DEBUG:boto:Token: None 2024-03-04T17:53:42.731 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.731 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad832e70c8b1bd2f-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.736 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.736 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.736 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.737 DEBUG:boto:path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:42.737 DEBUG:boto:auth_path=/ppfahn-22/?prefix=dummy 2024-03-04T17:53:42.737 DEBUG:boto:Method: GET 2024-03-04T17:53:42.737 DEBUG:boto:Path: /ppfahn-22/?prefix=dummy 2024-03-04T17:53:42.737 DEBUG:boto:Data: 2024-03-04T17:53:42.737 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.737 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.737 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.737 DEBUG:boto:Params: {} 2024-03-04T17:53:42.737 DEBUG:boto:Token: None 2024-03-04T17:53:42.737 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.738 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013f3a9cb74528d70-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.784 DEBUG:boto:b'ppfahn-22dummy1000falsedummy2024-03-04T17:50:43.358Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:42.784 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.785 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.785 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.785 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.785 DEBUG:boto:Method: GET 2024-03-04T17:53:42.785 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.785 DEBUG:boto:Data: 2024-03-04T17:53:42.785 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.785 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.785 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.785 DEBUG:boto:Params: {} 2024-03-04T17:53:42.785 DEBUG:boto:Token: None 2024-03-04T17:53:42.786 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.786 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs= 2024-03-04T17:53:42.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs=', 'Content-Length': '0'} 2024-03-04T17:53:42.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6c8f4fa4b761aa7-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.789 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.789 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.789 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.789 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.789 DEBUG:boto:Method: GET 2024-03-04T17:53:42.789 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.789 DEBUG:boto:Data: 2024-03-04T17:53:42.789 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.790 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.790 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.790 DEBUG:boto:Params: {} 2024-03-04T17:53:42.790 DEBUG:boto:Token: None 2024-03-04T17:53:42.790 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.790 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs= 2024-03-04T17:53:42.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs=', 'Content-Length': '0'} 2024-03-04T17:53:42.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fea15ecb2e2b60a5-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.836 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.836 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.836 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.836 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.836 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.836 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.837 DEBUG:boto:Data: 2024-03-04T17:53:42.837 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.837 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.837 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.837 DEBUG:boto:Params: {} 2024-03-04T17:53:42.837 DEBUG:boto:Token: None 2024-03-04T17:53:42.837 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.837 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TwORQLGlwQE1hQBDOs7ObgPvcOM= 2024-03-04T17:53:42.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TwORQLGlwQE1hQBDOs7ObgPvcOM=', 'Content-Length': '0'} 2024-03-04T17:53:42.840 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bcd445b1353004e-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.840 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.840 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.841 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.841 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.841 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.841 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.841 DEBUG:boto:Data: 2024-03-04T17:53:42.841 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.841 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.841 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.841 DEBUG:boto:Params: {} 2024-03-04T17:53:42.841 DEBUG:boto:Token: None 2024-03-04T17:53:42.841 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.842 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TwORQLGlwQE1hQBDOs7ObgPvcOM= 2024-03-04T17:53:42.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TwORQLGlwQE1hQBDOs7ObgPvcOM=', 'Content-Length': '0'} 2024-03-04T17:53:42.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc24c6a7d44cc673-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:42.845 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.845 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.845 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.845 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.845 DEBUG:boto:Method: GET 2024-03-04T17:53:42.845 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.846 DEBUG:boto:Data: 2024-03-04T17:53:42.846 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.846 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.846 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.846 DEBUG:boto:Params: {} 2024-03-04T17:53:42.846 DEBUG:boto:Token: None 2024-03-04T17:53:42.846 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.846 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs= 2024-03-04T17:53:42.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs=', 'Content-Length': '0'} 2024-03-04T17:53:42.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002df7bbe08ec0e2d7-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.892 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.892 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.892 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.892 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.892 DEBUG:boto:Method: GET 2024-03-04T17:53:42.892 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.893 DEBUG:boto:Data: 2024-03-04T17:53:42.893 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.893 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.893 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.893 DEBUG:boto:Params: {} 2024-03-04T17:53:42.893 DEBUG:boto:Token: None 2024-03-04T17:53:42.893 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.893 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs= 2024-03-04T17:53:42.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs=', 'Content-Length': '0'} 2024-03-04T17:53:42.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004079ea9d064d711d-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.944 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.944 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.944 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.944 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.944 DEBUG:boto:Method: GET 2024-03-04T17:53:42.945 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:42.945 DEBUG:boto:Data: 2024-03-04T17:53:42.945 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.945 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.945 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.945 DEBUG:boto:Params: {} 2024-03-04T17:53:42.945 DEBUG:boto:Token: None 2024-03-04T17:53:42.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.945 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6661500d6440b83-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.950 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.950 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:42.950 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:42.950 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.950 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:42.951 DEBUG:boto:Method: GET 2024-03-04T17:53:42.951 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:42.951 DEBUG:boto:Data: 2024-03-04T17:53:42.951 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.951 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.951 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.951 DEBUG:boto:Params: {} 2024-03-04T17:53:42.951 DEBUG:boto:Token: None 2024-03-04T17:53:42.951 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/ 2024-03-04T17:53:42.951 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s= 2024-03-04T17:53:42.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aVSC3p1qxbOoQzJKtM0LTbBUU+s=', 'Content-Length': '0'} 2024-03-04T17:53:42.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae591bf37f126ee9-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.956 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:42.956 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.956 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.956 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.957 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:42.957 DEBUG:boto:Data: 2024-03-04T17:53:42.957 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.957 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.957 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.957 DEBUG:boto:Params: {} 2024-03-04T17:53:42.957 DEBUG:boto:Token: None 2024-03-04T17:53:42.957 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj 2024-03-04T17:53:42.957 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l9wBIajfI4nC7IK17NG6jyLtk1A= 2024-03-04T17:53:42.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l9wBIajfI4nC7IK17NG6jyLtk1A=', 'Content-Length': '0'} 2024-03-04T17:53:42.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002cd2373b87081af8-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.963 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.964 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.964 DEBUG:boto:Method: HEAD 2024-03-04T17:53:42.964 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:42.964 DEBUG:boto:Data: 2024-03-04T17:53:42.964 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.964 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.964 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.964 DEBUG:boto:Params: {} 2024-03-04T17:53:42.964 DEBUG:boto:Token: None 2024-03-04T17:53:42.964 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj 2024-03-04T17:53:42.965 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l9wBIajfI4nC7IK17NG6jyLtk1A= 2024-03-04T17:53:42.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l9wBIajfI4nC7IK17NG6jyLtk1A=', 'Content-Length': '0'} 2024-03-04T17:53:42.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000744b072d025dd989-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:42.971 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.971 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.971 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.971 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.971 DEBUG:boto:Method: GET 2024-03-04T17:53:42.971 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.971 DEBUG:boto:Data: 2024-03-04T17:53:42.971 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.972 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:42.972 DEBUG:boto:Port: 8001 2024-03-04T17:53:42.972 DEBUG:boto:Params: {} 2024-03-04T17:53:42.972 DEBUG:boto:Token: None 2024-03-04T17:53:42.972 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.972 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs= 2024-03-04T17:53:42.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs=', 'Content-Length': '0'} 2024-03-04T17:53:42.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007eaed96a17ce2279-0065e60aa6-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:42.975 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:42.975 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:42.975 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.975 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.975 DEBUG:boto:Method: GET 2024-03-04T17:53:42.976 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.976 DEBUG:boto:Data: 2024-03-04T17:53:42.976 DEBUG:boto:Headers: {} 2024-03-04T17:53:42.976 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:42.976 DEBUG:boto:Port: 8000 2024-03-04T17:53:42.976 DEBUG:boto:Params: {} 2024-03-04T17:53:42.976 DEBUG:boto:Token: None 2024-03-04T17:53:42.976 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:42 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:42.976 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs= 2024-03-04T17:53:42.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:umHlHxCoYh3UY+zoIqCJ8A0HqNs=', 'Content-Length': '0'} 2024-03-04T17:53:42.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000004d123306979db3-0065e60aa6-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.020 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.020 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.020 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.020 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.020 DEBUG:boto:Method: GET 2024-03-04T17:53:43.021 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.021 DEBUG:boto:Data: 2024-03-04T17:53:43.021 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.021 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.021 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.021 DEBUG:boto:Params: {} 2024-03-04T17:53:43.021 DEBUG:boto:Token: None 2024-03-04T17:53:43.021 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.021 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e33762e305b6c93-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.026 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.027 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.027 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.027 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.027 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.027 DEBUG:boto:Method: GET 2024-03-04T17:53:43.027 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.027 DEBUG:boto:Data: 2024-03-04T17:53:43.027 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.027 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.027 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.027 DEBUG:boto:Params: {} 2024-03-04T17:53:43.028 DEBUG:boto:Token: None 2024-03-04T17:53:43.028 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.028 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096cf835dc8377968-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.076 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:43.076 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.077 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.077 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.077 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.077 DEBUG:boto:Method: GET 2024-03-04T17:53:43.077 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.077 DEBUG:boto:Data: 2024-03-04T17:53:43.077 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.077 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.077 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.077 DEBUG:boto:Params: {} 2024-03-04T17:53:43.077 DEBUG:boto:Token: None 2024-03-04T17:53:43.078 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.078 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6UAgIWzWZ7v1YKc4pbDFuenTc1I= 2024-03-04T17:53:43.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6UAgIWzWZ7v1YKc4pbDFuenTc1I=', 'Content-Length': '0'} 2024-03-04T17:53:43.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037f046b49fa38a59-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.128 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.128 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.128 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.128 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.128 DEBUG:boto:Method: GET 2024-03-04T17:53:43.129 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.129 DEBUG:boto:Data: 2024-03-04T17:53:43.129 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.129 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.129 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.129 DEBUG:boto:Params: {} 2024-03-04T17:53:43.129 DEBUG:boto:Token: None 2024-03-04T17:53:43.129 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.129 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6UAgIWzWZ7v1YKc4pbDFuenTc1I= 2024-03-04T17:53:43.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6UAgIWzWZ7v1YKc4pbDFuenTc1I=', 'Content-Length': '0'} 2024-03-04T17:53:43.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de281b583d99a572-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.133 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.133 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.133 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.133 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.133 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.133 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.133 DEBUG:boto:Data: 2024-03-04T17:53:43.133 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.133 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.134 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.134 DEBUG:boto:Params: {} 2024-03-04T17:53:43.134 DEBUG:boto:Token: None 2024-03-04T17:53:43.134 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.134 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pelxI3xmN3GrefwxRd7Fx6C1C+0= 2024-03-04T17:53:43.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pelxI3xmN3GrefwxRd7Fx6C1C+0=', 'Content-Length': '0'} 2024-03-04T17:53:43.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072c446e61d05ee72-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.138 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.138 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.138 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.138 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.138 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.138 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.138 DEBUG:boto:Data: 2024-03-04T17:53:43.138 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.138 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.138 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.138 DEBUG:boto:Params: {} 2024-03-04T17:53:43.139 DEBUG:boto:Token: None 2024-03-04T17:53:43.139 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.139 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pelxI3xmN3GrefwxRd7Fx6C1C+0= 2024-03-04T17:53:43.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pelxI3xmN3GrefwxRd7Fx6C1C+0=', 'Content-Length': '0'} 2024-03-04T17:53:43.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e07707f5cba2095-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:43.142 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.143 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.143 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.143 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.143 DEBUG:boto:Method: GET 2024-03-04T17:53:43.143 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.143 DEBUG:boto:Data: 2024-03-04T17:53:43.143 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.143 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.143 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.143 DEBUG:boto:Params: {} 2024-03-04T17:53:43.143 DEBUG:boto:Token: None 2024-03-04T17:53:43.144 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.144 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6UAgIWzWZ7v1YKc4pbDFuenTc1I= 2024-03-04T17:53:43.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6UAgIWzWZ7v1YKc4pbDFuenTc1I=', 'Content-Length': '0'} 2024-03-04T17:53:43.147 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093f7e706a0e37eb2-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.188 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.188 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.188 DEBUG:boto:path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.188 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.189 DEBUG:boto:Method: GET 2024-03-04T17:53:43.189 DEBUG:boto:Path: /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.189 DEBUG:boto:Data: 2024-03-04T17:53:43.189 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.189 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.189 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.189 DEBUG:boto:Params: {} 2024-03-04T17:53:43.189 DEBUG:boto:Token: None 2024-03-04T17:53:43.189 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE 2024-03-04T17:53:43.189 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6UAgIWzWZ7v1YKc4pbDFuenTc1I= 2024-03-04T17:53:43.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6UAgIWzWZ7v1YKc4pbDFuenTc1I=', 'Content-Length': '0'} 2024-03-04T17:53:43.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'QDkS502cdQ0sD3MSSr4AJ.nNDwrboFE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072599cabe9150b41-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.236 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.236 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.236 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.236 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.236 DEBUG:boto:Method: GET 2024-03-04T17:53:43.236 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.237 DEBUG:boto:Data: 2024-03-04T17:53:43.237 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.237 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.237 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.237 DEBUG:boto:Params: {} 2024-03-04T17:53:43.237 DEBUG:boto:Token: None 2024-03-04T17:53:43.237 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.237 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093f8da0c1d858eca-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.243 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.243 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.244 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.244 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.244 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.244 DEBUG:boto:Method: GET 2024-03-04T17:53:43.244 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.244 DEBUG:boto:Data: 2024-03-04T17:53:43.244 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.244 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.244 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.244 DEBUG:boto:Params: {} 2024-03-04T17:53:43.244 DEBUG:boto:Token: None 2024-03-04T17:53:43.245 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.245 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034c7218fa7bcdb69-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.249 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.250 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.250 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.250 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.250 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:43.250 DEBUG:boto:Data: 2024-03-04T17:53:43.250 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.250 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.250 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.250 DEBUG:boto:Params: {} 2024-03-04T17:53:43.250 DEBUG:boto:Token: None 2024-03-04T17:53:43.250 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj 2024-03-04T17:53:43.251 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g= 2024-03-04T17:53:43.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g=', 'Content-Length': '0'} 2024-03-04T17:53:43.256 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f62a0715a9ca1cff-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.256 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.257 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.257 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.257 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:43.257 DEBUG:boto:Data: 2024-03-04T17:53:43.257 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.257 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.257 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.257 DEBUG:boto:Params: {} 2024-03-04T17:53:43.257 DEBUG:boto:Token: None 2024-03-04T17:53:43.257 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj 2024-03-04T17:53:43.258 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g= 2024-03-04T17:53:43.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g=', 'Content-Length': '0'} 2024-03-04T17:53:43.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003a0195b79416c51-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:43.264 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.264 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.264 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.264 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.264 DEBUG:boto:Method: GET 2024-03-04T17:53:43.264 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.264 DEBUG:boto:Data: 2024-03-04T17:53:43.264 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.264 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.264 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.265 DEBUG:boto:Params: {} 2024-03-04T17:53:43.265 DEBUG:boto:Token: None 2024-03-04T17:53:43.265 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.265 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sux+cqse2+3aj8t3icq14fwnR5I= 2024-03-04T17:53:43.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sux+cqse2+3aj8t3icq14fwnR5I=', 'Content-Length': '0'} 2024-03-04T17:53:43.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbdf1db91ab7a415-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.268 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.268 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.268 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.268 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.268 DEBUG:boto:Method: GET 2024-03-04T17:53:43.269 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.269 DEBUG:boto:Data: 2024-03-04T17:53:43.269 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.269 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.269 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.269 DEBUG:boto:Params: {} 2024-03-04T17:53:43.269 DEBUG:boto:Token: None 2024-03-04T17:53:43.269 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.269 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sux+cqse2+3aj8t3icq14fwnR5I= 2024-03-04T17:53:43.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sux+cqse2+3aj8t3icq14fwnR5I=', 'Content-Length': '0'} 2024-03-04T17:53:43.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002cf29fd025ea5386-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.316 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.316 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.316 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.316 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.316 DEBUG:boto:Method: GET 2024-03-04T17:53:43.317 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.317 DEBUG:boto:Data: 2024-03-04T17:53:43.317 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.317 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.317 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.317 DEBUG:boto:Params: {} 2024-03-04T17:53:43.317 DEBUG:boto:Token: None 2024-03-04T17:53:43.317 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.317 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009055915e02217ec3-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.323 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.323 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.323 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.323 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.323 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.323 DEBUG:boto:Method: GET 2024-03-04T17:53:43.323 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.323 DEBUG:boto:Data: 2024-03-04T17:53:43.323 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.324 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.324 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.324 DEBUG:boto:Params: {} 2024-03-04T17:53:43.324 DEBUG:boto:Token: None 2024-03-04T17:53:43.324 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.324 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e02d2008e4735409-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.329 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:43.330 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.330 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.330 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.330 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.330 DEBUG:boto:Method: GET 2024-03-04T17:53:43.330 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.330 DEBUG:boto:Data: 2024-03-04T17:53:43.330 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.330 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.330 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.330 DEBUG:boto:Params: {} 2024-03-04T17:53:43.331 DEBUG:boto:Token: None 2024-03-04T17:53:43.331 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.331 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MBLLuQDcbb5KjwWFzIbhpE1bEzs= 2024-03-04T17:53:43.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MBLLuQDcbb5KjwWFzIbhpE1bEzs=', 'Content-Length': '0'} 2024-03-04T17:53:43.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023610c7f6373414b-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.376 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.376 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.376 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.376 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.376 DEBUG:boto:Method: GET 2024-03-04T17:53:43.376 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.377 DEBUG:boto:Data: 2024-03-04T17:53:43.377 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.377 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.377 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.377 DEBUG:boto:Params: {} 2024-03-04T17:53:43.377 DEBUG:boto:Token: None 2024-03-04T17:53:43.377 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.377 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MBLLuQDcbb5KjwWFzIbhpE1bEzs= 2024-03-04T17:53:43.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MBLLuQDcbb5KjwWFzIbhpE1bEzs=', 'Content-Length': '0'} 2024-03-04T17:53:43.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca7bee0972a50b6e-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.381 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.381 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.381 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.381 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.381 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.381 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.381 DEBUG:boto:Data: 2024-03-04T17:53:43.381 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.381 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.381 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.382 DEBUG:boto:Params: {} 2024-03-04T17:53:43.382 DEBUG:boto:Token: None 2024-03-04T17:53:43.382 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.382 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Rr77dEWS8pCZBvVqNlYiPEz/di8= 2024-03-04T17:53:43.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Rr77dEWS8pCZBvVqNlYiPEz/di8=', 'Content-Length': '0'} 2024-03-04T17:53:43.385 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0d75af531ce5bfd-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.386 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.386 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.386 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.386 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.386 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.386 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.386 DEBUG:boto:Data: 2024-03-04T17:53:43.386 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.386 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.386 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.387 DEBUG:boto:Params: {} 2024-03-04T17:53:43.387 DEBUG:boto:Token: None 2024-03-04T17:53:43.387 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.387 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Rr77dEWS8pCZBvVqNlYiPEz/di8= 2024-03-04T17:53:43.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Rr77dEWS8pCZBvVqNlYiPEz/di8=', 'Content-Length': '0'} 2024-03-04T17:53:43.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef0da521157565e1-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:43.390 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.390 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.390 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.390 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.390 DEBUG:boto:Method: GET 2024-03-04T17:53:43.390 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.390 DEBUG:boto:Data: 2024-03-04T17:53:43.390 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.390 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.391 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.391 DEBUG:boto:Params: {} 2024-03-04T17:53:43.391 DEBUG:boto:Token: None 2024-03-04T17:53:43.391 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.391 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MBLLuQDcbb5KjwWFzIbhpE1bEzs= 2024-03-04T17:53:43.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MBLLuQDcbb5KjwWFzIbhpE1bEzs=', 'Content-Length': '0'} 2024-03-04T17:53:43.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb7ced831fa9c004-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.436 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.436 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.436 DEBUG:boto:path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.436 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.436 DEBUG:boto:Method: GET 2024-03-04T17:53:43.436 DEBUG:boto:Path: /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.437 DEBUG:boto:Data: 2024-03-04T17:53:43.437 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.437 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.437 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.437 DEBUG:boto:Params: {} 2024-03-04T17:53:43.437 DEBUG:boto:Token: None 2024-03-04T17:53:43.437 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=fexW91D920ZLyPSn0KVa8L-0Jf11s3q 2024-03-04T17:53:43.437 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MBLLuQDcbb5KjwWFzIbhpE1bEzs= 2024-03-04T17:53:43.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MBLLuQDcbb5KjwWFzIbhpE1bEzs=', 'Content-Length': '0'} 2024-03-04T17:53:43.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'fexW91D920ZLyPSn0KVa8L-0Jf11s3q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000904f558a4fb7ffb-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.440 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.441 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.441 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.441 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.441 DEBUG:boto:Method: GET 2024-03-04T17:53:43.441 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.441 DEBUG:boto:Data: 2024-03-04T17:53:43.441 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.441 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.441 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.441 DEBUG:boto:Params: {} 2024-03-04T17:53:43.441 DEBUG:boto:Token: None 2024-03-04T17:53:43.442 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.442 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cec5b850f27c1eff-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.488 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.489 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.489 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.489 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.489 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.489 DEBUG:boto:Method: GET 2024-03-04T17:53:43.489 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.489 DEBUG:boto:Data: 2024-03-04T17:53:43.489 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.489 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.489 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.489 DEBUG:boto:Params: {} 2024-03-04T17:53:43.490 DEBUG:boto:Token: None 2024-03-04T17:53:43.490 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.490 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055f368eae70df3b0-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.495 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.495 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.495 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.495 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.495 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:43.495 DEBUG:boto:Data: 2024-03-04T17:53:43.496 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.496 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.496 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.496 DEBUG:boto:Params: {} 2024-03-04T17:53:43.496 DEBUG:boto:Token: None 2024-03-04T17:53:43.496 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj 2024-03-04T17:53:43.496 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g= 2024-03-04T17:53:43.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g=', 'Content-Length': '0'} 2024-03-04T17:53:43.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028b085f817954eb8-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.502 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.502 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.502 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.502 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:43.502 DEBUG:boto:Data: 2024-03-04T17:53:43.503 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.503 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.503 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.503 DEBUG:boto:Params: {} 2024-03-04T17:53:43.503 DEBUG:boto:Token: None 2024-03-04T17:53:43.503 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj 2024-03-04T17:53:43.503 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g= 2024-03-04T17:53:43.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g=', 'Content-Length': '0'} 2024-03-04T17:53:43.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000afd580e394ae6dcf-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:43.509 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.509 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.509 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.510 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.510 DEBUG:boto:Method: GET 2024-03-04T17:53:43.510 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.510 DEBUG:boto:Data: 2024-03-04T17:53:43.510 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.510 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.510 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.510 DEBUG:boto:Params: {} 2024-03-04T17:53:43.510 DEBUG:boto:Token: None 2024-03-04T17:53:43.510 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.510 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sux+cqse2+3aj8t3icq14fwnR5I= 2024-03-04T17:53:43.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sux+cqse2+3aj8t3icq14fwnR5I=', 'Content-Length': '0'} 2024-03-04T17:53:43.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5ad365846e418b0-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.514 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.514 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.514 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.514 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.514 DEBUG:boto:Method: GET 2024-03-04T17:53:43.514 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.514 DEBUG:boto:Data: 2024-03-04T17:53:43.514 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.514 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.515 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.515 DEBUG:boto:Params: {} 2024-03-04T17:53:43.515 DEBUG:boto:Token: None 2024-03-04T17:53:43.515 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.515 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sux+cqse2+3aj8t3icq14fwnR5I= 2024-03-04T17:53:43.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sux+cqse2+3aj8t3icq14fwnR5I=', 'Content-Length': '0'} 2024-03-04T17:53:43.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e2d475f249d432e-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.560 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.560 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.560 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.560 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.560 DEBUG:boto:Method: GET 2024-03-04T17:53:43.560 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.561 DEBUG:boto:Data: 2024-03-04T17:53:43.561 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.561 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.561 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.561 DEBUG:boto:Params: {} 2024-03-04T17:53:43.561 DEBUG:boto:Token: None 2024-03-04T17:53:43.561 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.561 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059ffa926d7773d2f-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.566 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.567 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.567 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.567 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.567 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.567 DEBUG:boto:Method: GET 2024-03-04T17:53:43.567 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.567 DEBUG:boto:Data: 2024-03-04T17:53:43.567 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.567 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.567 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.567 DEBUG:boto:Params: {} 2024-03-04T17:53:43.567 DEBUG:boto:Token: None 2024-03-04T17:53:43.568 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.568 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbcfda5ea6d790e4-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.572 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:43.573 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.573 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.573 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.573 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.573 DEBUG:boto:Method: GET 2024-03-04T17:53:43.573 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.573 DEBUG:boto:Data: 2024-03-04T17:53:43.573 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.573 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.573 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.574 DEBUG:boto:Params: {} 2024-03-04T17:53:43.574 DEBUG:boto:Token: None 2024-03-04T17:53:43.574 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.574 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bJxSZmq33KFtjgXiFGI+aV1bYyI= 2024-03-04T17:53:43.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bJxSZmq33KFtjgXiFGI+aV1bYyI=', 'Content-Length': '0'} 2024-03-04T17:53:43.577 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad4f51c8afc9fc6a-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.620 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.620 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.620 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.620 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.620 DEBUG:boto:Method: GET 2024-03-04T17:53:43.621 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.621 DEBUG:boto:Data: 2024-03-04T17:53:43.621 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.621 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.621 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.621 DEBUG:boto:Params: {} 2024-03-04T17:53:43.621 DEBUG:boto:Token: None 2024-03-04T17:53:43.621 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.621 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bJxSZmq33KFtjgXiFGI+aV1bYyI= 2024-03-04T17:53:43.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bJxSZmq33KFtjgXiFGI+aV1bYyI=', 'Content-Length': '0'} 2024-03-04T17:53:43.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003646b7c16079dd67-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.624 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.625 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.625 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.625 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.625 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.625 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.625 DEBUG:boto:Data: 2024-03-04T17:53:43.625 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.625 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.625 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.625 DEBUG:boto:Params: {} 2024-03-04T17:53:43.625 DEBUG:boto:Token: None 2024-03-04T17:53:43.626 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.626 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XPRUBuBFoSOsEZxzWJknpCz5xU0= 2024-03-04T17:53:43.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XPRUBuBFoSOsEZxzWJknpCz5xU0=', 'Content-Length': '0'} 2024-03-04T17:53:43.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ec47476cc62e032-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.628 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.629 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.629 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.629 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.629 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.629 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.629 DEBUG:boto:Data: 2024-03-04T17:53:43.629 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.629 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.629 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.629 DEBUG:boto:Params: {} 2024-03-04T17:53:43.629 DEBUG:boto:Token: None 2024-03-04T17:53:43.629 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.630 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XPRUBuBFoSOsEZxzWJknpCz5xU0= 2024-03-04T17:53:43.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XPRUBuBFoSOsEZxzWJknpCz5xU0=', 'Content-Length': '0'} 2024-03-04T17:53:43.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0f0d185e0d7ec03-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:43.633 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.633 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.633 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.633 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.633 DEBUG:boto:Method: GET 2024-03-04T17:53:43.633 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.633 DEBUG:boto:Data: 2024-03-04T17:53:43.633 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.633 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.633 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.633 DEBUG:boto:Params: {} 2024-03-04T17:53:43.634 DEBUG:boto:Token: None 2024-03-04T17:53:43.634 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.634 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bJxSZmq33KFtjgXiFGI+aV1bYyI= 2024-03-04T17:53:43.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bJxSZmq33KFtjgXiFGI+aV1bYyI=', 'Content-Length': '0'} 2024-03-04T17:53:43.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4a0aa4dcd9998f4-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.680 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.680 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.680 DEBUG:boto:path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.680 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.683 DEBUG:boto:Method: GET 2024-03-04T17:53:43.683 DEBUG:boto:Path: /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.683 DEBUG:boto:Data: 2024-03-04T17:53:43.683 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.683 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.683 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.683 DEBUG:boto:Params: {} 2024-03-04T17:53:43.683 DEBUG:boto:Token: None 2024-03-04T17:53:43.683 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=4aDzoElev9M4kJpBvZIFgGfNSrnTu1h 2024-03-04T17:53:43.684 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bJxSZmq33KFtjgXiFGI+aV1bYyI= 2024-03-04T17:53:43.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bJxSZmq33KFtjgXiFGI+aV1bYyI=', 'Content-Length': '0'} 2024-03-04T17:53:43.686 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '4aDzoElev9M4kJpBvZIFgGfNSrnTu1h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab0bfffa9577ea8c-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.687 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.687 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.687 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.687 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.687 DEBUG:boto:Method: GET 2024-03-04T17:53:43.687 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.687 DEBUG:boto:Data: 2024-03-04T17:53:43.687 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.687 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.687 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.687 DEBUG:boto:Params: {} 2024-03-04T17:53:43.688 DEBUG:boto:Token: None 2024-03-04T17:53:43.688 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.688 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfbbcf328753cf9a-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.736 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.736 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.736 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.736 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.737 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.737 DEBUG:boto:Method: GET 2024-03-04T17:53:43.737 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.737 DEBUG:boto:Data: 2024-03-04T17:53:43.737 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.737 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.737 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.737 DEBUG:boto:Params: {} 2024-03-04T17:53:43.737 DEBUG:boto:Token: None 2024-03-04T17:53:43.737 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.737 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017a6578c74e28e05-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.742 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.743 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.743 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.743 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.743 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:43.743 DEBUG:boto:Data: 2024-03-04T17:53:43.743 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.743 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.743 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.743 DEBUG:boto:Params: {} 2024-03-04T17:53:43.743 DEBUG:boto:Token: None 2024-03-04T17:53:43.744 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj 2024-03-04T17:53:43.744 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g= 2024-03-04T17:53:43.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g=', 'Content-Length': '0'} 2024-03-04T17:53:43.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093e5875cb4a01b21-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.750 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.750 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.750 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.750 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:43.751 DEBUG:boto:Data: 2024-03-04T17:53:43.751 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.751 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.751 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.751 DEBUG:boto:Params: {} 2024-03-04T17:53:43.751 DEBUG:boto:Token: None 2024-03-04T17:53:43.751 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj 2024-03-04T17:53:43.751 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g= 2024-03-04T17:53:43.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g=', 'Content-Length': '0'} 2024-03-04T17:53:43.757 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000314c8554e859bf6f-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:43.757 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.757 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.757 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.757 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.757 DEBUG:boto:Method: GET 2024-03-04T17:53:43.758 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.758 DEBUG:boto:Data: 2024-03-04T17:53:43.758 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.758 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.758 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.758 DEBUG:boto:Params: {} 2024-03-04T17:53:43.758 DEBUG:boto:Token: None 2024-03-04T17:53:43.758 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.758 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sux+cqse2+3aj8t3icq14fwnR5I= 2024-03-04T17:53:43.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sux+cqse2+3aj8t3icq14fwnR5I=', 'Content-Length': '0'} 2024-03-04T17:53:43.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de709da1ed3aeb7b-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.761 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.761 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.762 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.762 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.762 DEBUG:boto:Method: GET 2024-03-04T17:53:43.762 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.762 DEBUG:boto:Data: 2024-03-04T17:53:43.762 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.762 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.762 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.762 DEBUG:boto:Params: {} 2024-03-04T17:53:43.762 DEBUG:boto:Token: None 2024-03-04T17:53:43.762 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:43.762 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sux+cqse2+3aj8t3icq14fwnR5I= 2024-03-04T17:53:43.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sux+cqse2+3aj8t3icq14fwnR5I=', 'Content-Length': '0'} 2024-03-04T17:53:43.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000351eb3f710f8242e-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.808 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.808 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.808 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.808 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.808 DEBUG:boto:Method: GET 2024-03-04T17:53:43.809 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.809 DEBUG:boto:Data: 2024-03-04T17:53:43.809 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.809 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.809 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.809 DEBUG:boto:Params: {} 2024-03-04T17:53:43.809 DEBUG:boto:Token: None 2024-03-04T17:53:43.809 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.809 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea3b0044e6352557-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.815 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.815 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.816 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.816 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.816 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.816 DEBUG:boto:Method: GET 2024-03-04T17:53:43.816 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.816 DEBUG:boto:Data: 2024-03-04T17:53:43.816 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.816 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.816 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.816 DEBUG:boto:Params: {} 2024-03-04T17:53:43.816 DEBUG:boto:Token: None 2024-03-04T17:53:43.817 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.817 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b28cb37e9c886266-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.821 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:43.822 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.822 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.822 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.822 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.822 DEBUG:boto:Method: GET 2024-03-04T17:53:43.822 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.822 DEBUG:boto:Data: 2024-03-04T17:53:43.823 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.823 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.823 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.823 DEBUG:boto:Params: {} 2024-03-04T17:53:43.823 DEBUG:boto:Token: None 2024-03-04T17:53:43.823 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.823 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kIXTe3ImgrhSkloqqTBPUCZn75o= 2024-03-04T17:53:43.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kIXTe3ImgrhSkloqqTBPUCZn75o=', 'Content-Length': '0'} 2024-03-04T17:53:43.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7307678c0a5f9aa-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.868 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.868 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.868 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.868 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.869 DEBUG:boto:Method: GET 2024-03-04T17:53:43.869 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.869 DEBUG:boto:Data: 2024-03-04T17:53:43.869 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.869 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.869 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.869 DEBUG:boto:Params: {} 2024-03-04T17:53:43.869 DEBUG:boto:Token: None 2024-03-04T17:53:43.869 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.870 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kIXTe3ImgrhSkloqqTBPUCZn75o= 2024-03-04T17:53:43.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kIXTe3ImgrhSkloqqTBPUCZn75o=', 'Content-Length': '0'} 2024-03-04T17:53:43.873 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b088f09d1abe4ae-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.916 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.916 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.916 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.916 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.917 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.917 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.917 DEBUG:boto:Data: 2024-03-04T17:53:43.917 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.917 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.917 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.917 DEBUG:boto:Params: {} 2024-03-04T17:53:43.917 DEBUG:boto:Token: None 2024-03-04T17:53:43.918 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.918 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CSfP5a0H9zOsTUIqE9ezo7xwlgc= 2024-03-04T17:53:43.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CSfP5a0H9zOsTUIqE9ezo7xwlgc=', 'Content-Length': '0'} 2024-03-04T17:53:43.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4db8e02a1251aec-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.921 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.921 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.921 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.921 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.921 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.921 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.922 DEBUG:boto:Data: 2024-03-04T17:53:43.922 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.922 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.922 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.922 DEBUG:boto:Params: {} 2024-03-04T17:53:43.922 DEBUG:boto:Token: None 2024-03-04T17:53:43.922 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.922 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CSfP5a0H9zOsTUIqE9ezo7xwlgc= 2024-03-04T17:53:43.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CSfP5a0H9zOsTUIqE9ezo7xwlgc=', 'Content-Length': '0'} 2024-03-04T17:53:43.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1684548a39ae39e-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:43.926 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.926 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.926 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.926 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.926 DEBUG:boto:Method: GET 2024-03-04T17:53:43.926 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.926 DEBUG:boto:Data: 2024-03-04T17:53:43.926 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.926 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.927 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.927 DEBUG:boto:Params: {} 2024-03-04T17:53:43.927 DEBUG:boto:Token: None 2024-03-04T17:53:43.927 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.927 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kIXTe3ImgrhSkloqqTBPUCZn75o= 2024-03-04T17:53:43.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kIXTe3ImgrhSkloqqTBPUCZn75o=', 'Content-Length': '0'} 2024-03-04T17:53:43.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a38443e372fbbf2d-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.931 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.931 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.931 DEBUG:boto:path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.931 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.931 DEBUG:boto:Method: GET 2024-03-04T17:53:43.931 DEBUG:boto:Path: /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.931 DEBUG:boto:Data: 2024-03-04T17:53:43.931 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.931 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.932 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.932 DEBUG:boto:Params: {} 2024-03-04T17:53:43.932 DEBUG:boto:Token: None 2024-03-04T17:53:43.932 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj?versionId=rBR7sHkOppK-w8aON9rPlfIadTwy8LK 2024-03-04T17:53:43.932 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kIXTe3ImgrhSkloqqTBPUCZn75o= 2024-03-04T17:53:43.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kIXTe3ImgrhSkloqqTBPUCZn75o=', 'Content-Length': '0'} 2024-03-04T17:53:43.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'rBR7sHkOppK-w8aON9rPlfIadTwy8LK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bf8d630eb8b68b1-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.980 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.980 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.980 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.980 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.981 DEBUG:boto:Method: GET 2024-03-04T17:53:43.981 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.981 DEBUG:boto:Data: 2024-03-04T17:53:43.981 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.981 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.981 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.981 DEBUG:boto:Params: {} 2024-03-04T17:53:43.981 DEBUG:boto:Token: None 2024-03-04T17:53:43.981 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.981 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e66443ea67c05fbe-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.986 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.986 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:43.987 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:43.987 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.987 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:43.987 DEBUG:boto:Method: GET 2024-03-04T17:53:43.987 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:43.987 DEBUG:boto:Data: 2024-03-04T17:53:43.987 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.987 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:43.987 DEBUG:boto:Port: 8000 2024-03-04T17:53:43.987 DEBUG:boto:Params: {} 2024-03-04T17:53:43.987 DEBUG:boto:Token: None 2024-03-04T17:53:43.988 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/ 2024-03-04T17:53:43.988 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU= 2024-03-04T17:53:43.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3TZL7QMNuIZMpprxEtk88oW7PxU=', 'Content-Length': '0'} 2024-03-04T17:53:43.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d52c10b12b57206-0065e60aa7-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:43.993 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:43.993 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:43.993 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:43.994 DEBUG:boto:Method: HEAD 2024-03-04T17:53:43.994 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:43.994 DEBUG:boto:Data: 2024-03-04T17:53:43.994 DEBUG:boto:Headers: {} 2024-03-04T17:53:43.994 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:43.994 DEBUG:boto:Port: 8001 2024-03-04T17:53:43.994 DEBUG:boto:Params: {} 2024-03-04T17:53:43.994 DEBUG:boto:Token: None 2024-03-04T17:53:43.994 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:43 GMT /ppfahn-22/obj 2024-03-04T17:53:43.995 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g= 2024-03-04T17:53:43.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D1HAQJjSanJ73HG8BdBkBOS2w3g=', 'Content-Length': '0'} 2024-03-04T17:53:44.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c46d1a4b9c01964b-0065e60aa7-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.000 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.001 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.001 DEBUG:boto:Method: HEAD 2024-03-04T17:53:44.001 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:44.001 DEBUG:boto:Data: 2024-03-04T17:53:44.001 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.001 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.001 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.001 DEBUG:boto:Params: {} 2024-03-04T17:53:44.001 DEBUG:boto:Token: None 2024-03-04T17:53:44.001 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj 2024-03-04T17:53:44.002 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss= 2024-03-04T17:53:44.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss=', 'Content-Length': '0'} 2024-03-04T17:53:44.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080fb21accad55134-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:44.007 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.007 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.008 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.008 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.008 DEBUG:boto:Method: GET 2024-03-04T17:53:44.008 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.008 DEBUG:boto:Data: 2024-03-04T17:53:44.008 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.008 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.008 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.008 DEBUG:boto:Params: {} 2024-03-04T17:53:44.008 DEBUG:boto:Token: None 2024-03-04T17:53:44.009 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.009 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o= 2024-03-04T17:53:44.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o=', 'Content-Length': '0'} 2024-03-04T17:53:44.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004da74e2e8f5fbb30-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.052 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.052 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.052 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.052 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.052 DEBUG:boto:Method: GET 2024-03-04T17:53:44.053 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.053 DEBUG:boto:Data: 2024-03-04T17:53:44.053 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.053 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.053 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.053 DEBUG:boto:Params: {} 2024-03-04T17:53:44.053 DEBUG:boto:Token: None 2024-03-04T17:53:44.053 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.053 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o= 2024-03-04T17:53:44.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o=', 'Content-Length': '0'} 2024-03-04T17:53:44.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cda4fd1edf3b6b63-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.100 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.100 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.100 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.100 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.100 DEBUG:boto:Method: GET 2024-03-04T17:53:44.101 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.101 DEBUG:boto:Data: 2024-03-04T17:53:44.101 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.101 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.101 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.101 DEBUG:boto:Params: {} 2024-03-04T17:53:44.101 DEBUG:boto:Token: None 2024-03-04T17:53:44.101 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.101 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032549069525871eb-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.107 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.107 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.107 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.107 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.107 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.107 DEBUG:boto:Method: GET 2024-03-04T17:53:44.108 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.108 DEBUG:boto:Data: 2024-03-04T17:53:44.108 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.108 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.108 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.108 DEBUG:boto:Params: {} 2024-03-04T17:53:44.108 DEBUG:boto:Token: None 2024-03-04T17:53:44.108 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.108 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7b6893a4300082a-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.113 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:44.114 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.114 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.114 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.114 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.114 DEBUG:boto:Method: GET 2024-03-04T17:53:44.114 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.114 DEBUG:boto:Data: 2024-03-04T17:53:44.115 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.115 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.115 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.115 DEBUG:boto:Params: {} 2024-03-04T17:53:44.115 DEBUG:boto:Token: None 2024-03-04T17:53:44.115 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.115 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XNvz2e9SkMg9XsQ8E4tDGqyTScY= 2024-03-04T17:53:44.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XNvz2e9SkMg9XsQ8E4tDGqyTScY=', 'Content-Length': '0'} 2024-03-04T17:53:44.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064d5de0c967444c9-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.118 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.119 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.119 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.119 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.119 DEBUG:boto:Method: GET 2024-03-04T17:53:44.119 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.119 DEBUG:boto:Data: 2024-03-04T17:53:44.119 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.119 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.119 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.119 DEBUG:boto:Params: {} 2024-03-04T17:53:44.120 DEBUG:boto:Token: None 2024-03-04T17:53:44.120 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.120 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XNvz2e9SkMg9XsQ8E4tDGqyTScY= 2024-03-04T17:53:44.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XNvz2e9SkMg9XsQ8E4tDGqyTScY=', 'Content-Length': '0'} 2024-03-04T17:53:44.123 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cc5081b841ab932-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.123 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.124 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.124 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.124 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.124 DEBUG:boto:Method: HEAD 2024-03-04T17:53:44.124 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.124 DEBUG:boto:Data: 2024-03-04T17:53:44.124 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.124 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.124 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.124 DEBUG:boto:Params: {} 2024-03-04T17:53:44.124 DEBUG:boto:Token: None 2024-03-04T17:53:44.125 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.125 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vz4hSlhHfg0k9/RKy9FCh8KEHIw= 2024-03-04T17:53:44.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vz4hSlhHfg0k9/RKy9FCh8KEHIw=', 'Content-Length': '0'} 2024-03-04T17:53:44.128 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b3bc9e2b79844ac-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.128 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.128 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.128 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.128 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.128 DEBUG:boto:Method: HEAD 2024-03-04T17:53:44.129 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.129 DEBUG:boto:Data: 2024-03-04T17:53:44.129 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.129 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.129 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.129 DEBUG:boto:Params: {} 2024-03-04T17:53:44.129 DEBUG:boto:Token: None 2024-03-04T17:53:44.129 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.129 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vz4hSlhHfg0k9/RKy9FCh8KEHIw= 2024-03-04T17:53:44.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vz4hSlhHfg0k9/RKy9FCh8KEHIw=', 'Content-Length': '0'} 2024-03-04T17:53:44.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b426df4bf78335f2-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:44.132 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.132 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.133 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.133 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.133 DEBUG:boto:Method: GET 2024-03-04T17:53:44.133 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.133 DEBUG:boto:Data: 2024-03-04T17:53:44.133 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.133 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.133 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.133 DEBUG:boto:Params: {} 2024-03-04T17:53:44.133 DEBUG:boto:Token: None 2024-03-04T17:53:44.134 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.134 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XNvz2e9SkMg9XsQ8E4tDGqyTScY= 2024-03-04T17:53:44.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XNvz2e9SkMg9XsQ8E4tDGqyTScY=', 'Content-Length': '0'} 2024-03-04T17:53:44.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007dadce06c4acec2e-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.180 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.180 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.180 DEBUG:boto:path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.180 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.180 DEBUG:boto:Method: GET 2024-03-04T17:53:44.181 DEBUG:boto:Path: /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.181 DEBUG:boto:Data: 2024-03-04T17:53:44.181 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.181 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.181 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.181 DEBUG:boto:Params: {} 2024-03-04T17:53:44.181 DEBUG:boto:Token: None 2024-03-04T17:53:44.181 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23 2024-03-04T17:53:44.181 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XNvz2e9SkMg9XsQ8E4tDGqyTScY= 2024-03-04T17:53:44.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XNvz2e9SkMg9XsQ8E4tDGqyTScY=', 'Content-Length': '0'} 2024-03-04T17:53:44.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'tp1YBJlmM1cmYYj2xRDAuLHN-EIcD23'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002360f27c561c336d-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.228 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.228 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.228 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.228 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.228 DEBUG:boto:Method: GET 2024-03-04T17:53:44.228 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.229 DEBUG:boto:Data: 2024-03-04T17:53:44.229 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.229 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.229 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.229 DEBUG:boto:Params: {} 2024-03-04T17:53:44.229 DEBUG:boto:Token: None 2024-03-04T17:53:44.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.229 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009af34a7ec974f031-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.276 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.276 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.276 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.277 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.277 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.277 DEBUG:boto:Method: GET 2024-03-04T17:53:44.277 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.277 DEBUG:boto:Data: 2024-03-04T17:53:44.277 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.277 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.277 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.277 DEBUG:boto:Params: {} 2024-03-04T17:53:44.277 DEBUG:boto:Token: None 2024-03-04T17:53:44.277 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.278 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096bafa864041a488-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.328 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.328 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.328 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.329 DEBUG:boto:Method: HEAD 2024-03-04T17:53:44.329 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:44.329 DEBUG:boto:Data: 2024-03-04T17:53:44.329 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.329 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.329 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.329 DEBUG:boto:Params: {} 2024-03-04T17:53:44.329 DEBUG:boto:Token: None 2024-03-04T17:53:44.329 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj 2024-03-04T17:53:44.329 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss= 2024-03-04T17:53:44.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss=', 'Content-Length': '0'} 2024-03-04T17:53:44.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a8d1ee5639cc8e9-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.336 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.336 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.336 DEBUG:boto:Method: HEAD 2024-03-04T17:53:44.336 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:44.337 DEBUG:boto:Data: 2024-03-04T17:53:44.337 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.337 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.337 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.337 DEBUG:boto:Params: {} 2024-03-04T17:53:44.337 DEBUG:boto:Token: None 2024-03-04T17:53:44.337 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj 2024-03-04T17:53:44.337 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss= 2024-03-04T17:53:44.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss=', 'Content-Length': '0'} 2024-03-04T17:53:44.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003f39fbfcf2b718e-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:44.343 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.343 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.344 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.344 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.344 DEBUG:boto:Method: GET 2024-03-04T17:53:44.344 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.344 DEBUG:boto:Data: 2024-03-04T17:53:44.344 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.344 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.344 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.344 DEBUG:boto:Params: {} 2024-03-04T17:53:44.344 DEBUG:boto:Token: None 2024-03-04T17:53:44.345 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.345 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o= 2024-03-04T17:53:44.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o=', 'Content-Length': '0'} 2024-03-04T17:53:44.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000feb61e28d2178a41-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.348 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.348 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.348 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.348 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.348 DEBUG:boto:Method: GET 2024-03-04T17:53:44.348 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.349 DEBUG:boto:Data: 2024-03-04T17:53:44.349 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.349 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.349 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.349 DEBUG:boto:Params: {} 2024-03-04T17:53:44.349 DEBUG:boto:Token: None 2024-03-04T17:53:44.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.349 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o= 2024-03-04T17:53:44.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o=', 'Content-Length': '0'} 2024-03-04T17:53:44.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f417047e561ab4c-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.396 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.396 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.396 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.396 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.396 DEBUG:boto:Method: GET 2024-03-04T17:53:44.396 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.397 DEBUG:boto:Data: 2024-03-04T17:53:44.397 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.397 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.397 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.397 DEBUG:boto:Params: {} 2024-03-04T17:53:44.397 DEBUG:boto:Token: None 2024-03-04T17:53:44.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.397 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c7f6dacb1f5fc5c-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.402 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.403 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.403 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.403 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.403 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.403 DEBUG:boto:Method: GET 2024-03-04T17:53:44.403 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.403 DEBUG:boto:Data: 2024-03-04T17:53:44.403 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.403 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.403 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.404 DEBUG:boto:Params: {} 2024-03-04T17:53:44.404 DEBUG:boto:Token: None 2024-03-04T17:53:44.404 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.404 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da632e4ee1614055-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.456 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:44.456 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.457 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.457 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.457 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.457 DEBUG:boto:Method: GET 2024-03-04T17:53:44.457 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.457 DEBUG:boto:Data: 2024-03-04T17:53:44.457 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.457 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.457 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.457 DEBUG:boto:Params: {} 2024-03-04T17:53:44.457 DEBUG:boto:Token: None 2024-03-04T17:53:44.458 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.458 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zhyT50M+KAWBgR85XTFqZw1QKhU= 2024-03-04T17:53:44.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zhyT50M+KAWBgR85XTFqZw1QKhU=', 'Content-Length': '0'} 2024-03-04T17:53:44.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc3312e8097403e3-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.504 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.504 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.504 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.504 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.504 DEBUG:boto:Method: GET 2024-03-04T17:53:44.504 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.505 DEBUG:boto:Data: 2024-03-04T17:53:44.505 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.505 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.505 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.505 DEBUG:boto:Params: {} 2024-03-04T17:53:44.505 DEBUG:boto:Token: None 2024-03-04T17:53:44.505 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.505 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zhyT50M+KAWBgR85XTFqZw1QKhU= 2024-03-04T17:53:44.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zhyT50M+KAWBgR85XTFqZw1QKhU=', 'Content-Length': '0'} 2024-03-04T17:53:44.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075f142458ca879cf-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.509 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.509 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.510 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.510 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.510 DEBUG:boto:Method: HEAD 2024-03-04T17:53:44.510 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.510 DEBUG:boto:Data: 2024-03-04T17:53:44.510 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.510 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.510 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.510 DEBUG:boto:Params: {} 2024-03-04T17:53:44.510 DEBUG:boto:Token: None 2024-03-04T17:53:44.510 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.511 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XA1d0sj4iVJzzRoQsWUVl0tpctw= 2024-03-04T17:53:44.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XA1d0sj4iVJzzRoQsWUVl0tpctw=', 'Content-Length': '0'} 2024-03-04T17:53:44.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b09947a07bf8983a-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.514 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.514 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.514 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.514 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.514 DEBUG:boto:Method: HEAD 2024-03-04T17:53:44.514 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.514 DEBUG:boto:Data: 2024-03-04T17:53:44.514 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.514 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.514 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.515 DEBUG:boto:Params: {} 2024-03-04T17:53:44.515 DEBUG:boto:Token: None 2024-03-04T17:53:44.515 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.515 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XA1d0sj4iVJzzRoQsWUVl0tpctw= 2024-03-04T17:53:44.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XA1d0sj4iVJzzRoQsWUVl0tpctw=', 'Content-Length': '0'} 2024-03-04T17:53:44.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef2944874002ab79-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:44.518 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.518 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.518 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.518 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.518 DEBUG:boto:Method: GET 2024-03-04T17:53:44.518 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.519 DEBUG:boto:Data: 2024-03-04T17:53:44.519 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.519 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.519 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.519 DEBUG:boto:Params: {} 2024-03-04T17:53:44.519 DEBUG:boto:Token: None 2024-03-04T17:53:44.519 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.519 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zhyT50M+KAWBgR85XTFqZw1QKhU= 2024-03-04T17:53:44.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zhyT50M+KAWBgR85XTFqZw1QKhU=', 'Content-Length': '0'} 2024-03-04T17:53:44.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001fde3a16edf6f1fc-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.564 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.564 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.564 DEBUG:boto:path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.564 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.564 DEBUG:boto:Method: GET 2024-03-04T17:53:44.564 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.564 DEBUG:boto:Data: 2024-03-04T17:53:44.565 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.565 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.565 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.565 DEBUG:boto:Params: {} 2024-03-04T17:53:44.565 DEBUG:boto:Token: None 2024-03-04T17:53:44.565 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs 2024-03-04T17:53:44.565 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zhyT50M+KAWBgR85XTFqZw1QKhU= 2024-03-04T17:53:44.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zhyT50M+KAWBgR85XTFqZw1QKhU=', 'Content-Length': '0'} 2024-03-04T17:53:44.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E-xsE1Xd4eGNhM-gMPvppQ4jrbF6Sxs'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff9b0b2339883b15-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.569 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.569 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.569 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.569 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.569 DEBUG:boto:Method: GET 2024-03-04T17:53:44.569 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.569 DEBUG:boto:Data: 2024-03-04T17:53:44.569 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.569 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.569 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.570 DEBUG:boto:Params: {} 2024-03-04T17:53:44.570 DEBUG:boto:Token: None 2024-03-04T17:53:44.570 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.570 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037926dd5fe88fe28-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.576 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.577 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.577 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.577 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.577 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.577 DEBUG:boto:Method: GET 2024-03-04T17:53:44.577 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.577 DEBUG:boto:Data: 2024-03-04T17:53:44.577 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.578 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.578 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.578 DEBUG:boto:Params: {} 2024-03-04T17:53:44.578 DEBUG:boto:Token: None 2024-03-04T17:53:44.578 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.578 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3e62c24e922c97a-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.584 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.584 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.584 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.584 DEBUG:boto:Method: HEAD 2024-03-04T17:53:44.584 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:44.585 DEBUG:boto:Data: 2024-03-04T17:53:44.585 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.585 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.585 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.585 DEBUG:boto:Params: {} 2024-03-04T17:53:44.585 DEBUG:boto:Token: None 2024-03-04T17:53:44.585 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj 2024-03-04T17:53:44.585 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss= 2024-03-04T17:53:44.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss=', 'Content-Length': '0'} 2024-03-04T17:53:44.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bdb9eaf8fb1641b-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.592 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.593 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.593 DEBUG:boto:Method: HEAD 2024-03-04T17:53:44.593 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:44.593 DEBUG:boto:Data: 2024-03-04T17:53:44.593 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.593 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.593 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.593 DEBUG:boto:Params: {} 2024-03-04T17:53:44.593 DEBUG:boto:Token: None 2024-03-04T17:53:44.593 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj 2024-03-04T17:53:44.594 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss= 2024-03-04T17:53:44.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss=', 'Content-Length': '0'} 2024-03-04T17:53:44.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e36e0b690da9c2c5-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:44.600 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.600 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.600 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.600 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.601 DEBUG:boto:Method: GET 2024-03-04T17:53:44.601 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.601 DEBUG:boto:Data: 2024-03-04T17:53:44.601 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.601 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.601 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.601 DEBUG:boto:Params: {} 2024-03-04T17:53:44.601 DEBUG:boto:Token: None 2024-03-04T17:53:44.601 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.601 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o= 2024-03-04T17:53:44.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o=', 'Content-Length': '0'} 2024-03-04T17:53:44.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f63fbbc74b872ce4-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.648 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.648 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.648 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.648 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.648 DEBUG:boto:Method: GET 2024-03-04T17:53:44.648 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.648 DEBUG:boto:Data: 2024-03-04T17:53:44.649 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.649 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.649 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.649 DEBUG:boto:Params: {} 2024-03-04T17:53:44.649 DEBUG:boto:Token: None 2024-03-04T17:53:44.649 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.649 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o= 2024-03-04T17:53:44.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o=', 'Content-Length': '0'} 2024-03-04T17:53:44.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccbaabde33ebaaa8-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.696 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.696 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.696 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.696 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.697 DEBUG:boto:Method: GET 2024-03-04T17:53:44.697 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.697 DEBUG:boto:Data: 2024-03-04T17:53:44.697 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.697 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.697 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.697 DEBUG:boto:Params: {} 2024-03-04T17:53:44.697 DEBUG:boto:Token: None 2024-03-04T17:53:44.697 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.697 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d398260d857751ee-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.703 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.703 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.703 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.703 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.704 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.704 DEBUG:boto:Method: GET 2024-03-04T17:53:44.704 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.704 DEBUG:boto:Data: 2024-03-04T17:53:44.704 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.704 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.704 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.704 DEBUG:boto:Params: {} 2024-03-04T17:53:44.704 DEBUG:boto:Token: None 2024-03-04T17:53:44.705 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.705 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d85f6fb861c52343-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.709 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:44.709 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.709 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.709 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.710 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.710 DEBUG:boto:Method: GET 2024-03-04T17:53:44.710 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.710 DEBUG:boto:Data: 2024-03-04T17:53:44.710 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.710 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.710 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.710 DEBUG:boto:Params: {} 2024-03-04T17:53:44.710 DEBUG:boto:Token: None 2024-03-04T17:53:44.710 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.711 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IQJOCr3iND8IcuSfeElQWs0yZqk= 2024-03-04T17:53:44.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IQJOCr3iND8IcuSfeElQWs0yZqk=', 'Content-Length': '0'} 2024-03-04T17:53:44.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cef8aea3155efae-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.713 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.713 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.713 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.713 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.714 DEBUG:boto:Method: GET 2024-03-04T17:53:44.714 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.714 DEBUG:boto:Data: 2024-03-04T17:53:44.714 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.714 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.714 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.714 DEBUG:boto:Params: {} 2024-03-04T17:53:44.714 DEBUG:boto:Token: None 2024-03-04T17:53:44.714 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.714 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IQJOCr3iND8IcuSfeElQWs0yZqk= 2024-03-04T17:53:44.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IQJOCr3iND8IcuSfeElQWs0yZqk=', 'Content-Length': '0'} 2024-03-04T17:53:44.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000735b1ba3777c6654-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.718 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.718 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.718 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.718 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.718 DEBUG:boto:Method: HEAD 2024-03-04T17:53:44.718 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.718 DEBUG:boto:Data: 2024-03-04T17:53:44.718 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.718 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.718 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.719 DEBUG:boto:Params: {} 2024-03-04T17:53:44.719 DEBUG:boto:Token: None 2024-03-04T17:53:44.719 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.719 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/qqURw9TEb64/EX6BxH8sNnaIxk= 2024-03-04T17:53:44.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/qqURw9TEb64/EX6BxH8sNnaIxk=', 'Content-Length': '0'} 2024-03-04T17:53:44.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e75e97a448f9e37-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.721 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.722 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.722 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.722 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.722 DEBUG:boto:Method: HEAD 2024-03-04T17:53:44.722 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.722 DEBUG:boto:Data: 2024-03-04T17:53:44.722 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.722 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.722 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.722 DEBUG:boto:Params: {} 2024-03-04T17:53:44.722 DEBUG:boto:Token: None 2024-03-04T17:53:44.723 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.723 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:/qqURw9TEb64/EX6BxH8sNnaIxk= 2024-03-04T17:53:44.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/qqURw9TEb64/EX6BxH8sNnaIxk=', 'Content-Length': '0'} 2024-03-04T17:53:44.726 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000851bee0344e0a9f1-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:44.726 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.726 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.726 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.726 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.726 DEBUG:boto:Method: GET 2024-03-04T17:53:44.726 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.727 DEBUG:boto:Data: 2024-03-04T17:53:44.727 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.727 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.727 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.727 DEBUG:boto:Params: {} 2024-03-04T17:53:44.727 DEBUG:boto:Token: None 2024-03-04T17:53:44.727 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.727 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IQJOCr3iND8IcuSfeElQWs0yZqk= 2024-03-04T17:53:44.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IQJOCr3iND8IcuSfeElQWs0yZqk=', 'Content-Length': '0'} 2024-03-04T17:53:44.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9e77e5992950dd6-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.772 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.772 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.772 DEBUG:boto:path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.772 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.772 DEBUG:boto:Method: GET 2024-03-04T17:53:44.772 DEBUG:boto:Path: /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.772 DEBUG:boto:Data: 2024-03-04T17:53:44.773 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.773 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.773 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.773 DEBUG:boto:Params: {} 2024-03-04T17:53:44.773 DEBUG:boto:Token: None 2024-03-04T17:53:44.773 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy 2024-03-04T17:53:44.773 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IQJOCr3iND8IcuSfeElQWs0yZqk= 2024-03-04T17:53:44.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IQJOCr3iND8IcuSfeElQWs0yZqk=', 'Content-Length': '0'} 2024-03-04T17:53:44.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'IgnYdWgPuuz09tkj9qWq7P8qzgBSCIy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa5fd9f14c3e0e9c-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.824 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.824 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.824 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.824 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.824 DEBUG:boto:Method: GET 2024-03-04T17:53:44.825 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.825 DEBUG:boto:Data: 2024-03-04T17:53:44.825 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.825 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.825 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.825 DEBUG:boto:Params: {} 2024-03-04T17:53:44.825 DEBUG:boto:Token: None 2024-03-04T17:53:44.825 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.825 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d76fc81ef9bb236e-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.830 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.831 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.831 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.831 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.831 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.831 DEBUG:boto:Method: GET 2024-03-04T17:53:44.831 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.831 DEBUG:boto:Data: 2024-03-04T17:53:44.831 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.831 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.831 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.832 DEBUG:boto:Params: {} 2024-03-04T17:53:44.832 DEBUG:boto:Token: None 2024-03-04T17:53:44.832 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.832 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c028e72b7a212905-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.838 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.838 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.838 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.838 DEBUG:boto:Method: HEAD 2024-03-04T17:53:44.838 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:44.838 DEBUG:boto:Data: 2024-03-04T17:53:44.838 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.839 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.839 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.839 DEBUG:boto:Params: {} 2024-03-04T17:53:44.839 DEBUG:boto:Token: None 2024-03-04T17:53:44.839 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj 2024-03-04T17:53:44.839 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss= 2024-03-04T17:53:44.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss=', 'Content-Length': '0'} 2024-03-04T17:53:44.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000876c5d6f19252589-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.845 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.845 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.845 DEBUG:boto:Method: HEAD 2024-03-04T17:53:44.846 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:44.846 DEBUG:boto:Data: 2024-03-04T17:53:44.846 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.846 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.846 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.846 DEBUG:boto:Params: {} 2024-03-04T17:53:44.846 DEBUG:boto:Token: None 2024-03-04T17:53:44.846 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj 2024-03-04T17:53:44.846 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss= 2024-03-04T17:53:44.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6J1yr/YXf9aIYb/l7XBqkHRH2Ss=', 'Content-Length': '0'} 2024-03-04T17:53:44.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d822fbf661294d6-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:44.852 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.853 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.853 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.853 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.853 DEBUG:boto:Method: GET 2024-03-04T17:53:44.853 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.853 DEBUG:boto:Data: 2024-03-04T17:53:44.853 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.853 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.853 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.853 DEBUG:boto:Params: {} 2024-03-04T17:53:44.853 DEBUG:boto:Token: None 2024-03-04T17:53:44.854 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.854 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o= 2024-03-04T17:53:44.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o=', 'Content-Length': '0'} 2024-03-04T17:53:44.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021d5b57823a382b0-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.857 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.857 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.857 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.858 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.858 DEBUG:boto:Method: GET 2024-03-04T17:53:44.858 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.858 DEBUG:boto:Data: 2024-03-04T17:53:44.858 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.858 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.858 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.858 DEBUG:boto:Params: {} 2024-03-04T17:53:44.858 DEBUG:boto:Token: None 2024-03-04T17:53:44.858 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:44.858 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o= 2024-03-04T17:53:44.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKjVJ2y2LJooKjeoWVSBxNDyo7o=', 'Content-Length': '0'} 2024-03-04T17:53:44.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b227c4dfd9404cae-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.908 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.908 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.908 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.908 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.909 DEBUG:boto:Method: GET 2024-03-04T17:53:44.909 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.909 DEBUG:boto:Data: 2024-03-04T17:53:44.909 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.909 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.909 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.909 DEBUG:boto:Params: {} 2024-03-04T17:53:44.909 DEBUG:boto:Token: None 2024-03-04T17:53:44.909 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.910 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dbe31ebcc888f27-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.914 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.915 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:44.915 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:44.915 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.915 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:44.915 DEBUG:boto:Method: GET 2024-03-04T17:53:44.915 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:44.915 DEBUG:boto:Data: 2024-03-04T17:53:44.915 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.916 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.916 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.916 DEBUG:boto:Params: {} 2024-03-04T17:53:44.916 DEBUG:boto:Token: None 2024-03-04T17:53:44.916 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/ 2024-03-04T17:53:44.916 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo= 2024-03-04T17:53:44.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hCbX9NOlOor2cQi/TVU4JsXTgqo=', 'Content-Length': '0'} 2024-03-04T17:53:44.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d4d89e4e5fbd3f7-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.921 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:44.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:44.922 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.922 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.922 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:44.922 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:44.922 DEBUG:boto:Method: GET 2024-03-04T17:53:44.922 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:44.922 DEBUG:boto:Data: 2024-03-04T17:53:44.922 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.922 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:44.922 DEBUG:boto:Port: 8001 2024-03-04T17:53:44.923 DEBUG:boto:Params: {} 2024-03-04T17:53:44.923 DEBUG:boto:Token: None 2024-03-04T17:53:44.923 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:44.923 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:b5eMFsmO6MGtfwqZ1mZIY0zC20w= 2024-03-04T17:53:44.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:b5eMFsmO6MGtfwqZ1mZIY0zC20w=', 'Content-Length': '0'} 2024-03-04T17:53:44.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdd04b0cf4a028a5-0065e60aa8-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:44.968 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:44.968 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:44.968 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:44.968 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:44.968 DEBUG:boto:Method: GET 2024-03-04T17:53:44.969 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:44.969 DEBUG:boto:Data: 2024-03-04T17:53:44.969 DEBUG:boto:Headers: {} 2024-03-04T17:53:44.969 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:44.969 DEBUG:boto:Port: 8000 2024-03-04T17:53:44.969 DEBUG:boto:Params: {} 2024-03-04T17:53:44.969 DEBUG:boto:Token: None 2024-03-04T17:53:44.969 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:44 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:44.969 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:b5eMFsmO6MGtfwqZ1mZIY0zC20w= 2024-03-04T17:53:44.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:b5eMFsmO6MGtfwqZ1mZIY0zC20w=', 'Content-Length': '0'} 2024-03-04T17:53:44.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de540d7f473799e5-0065e60aa8-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.016 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.016 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.016 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.016 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.016 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.017 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.017 DEBUG:boto:Data: 2024-03-04T17:53:45.017 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.017 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.017 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.017 DEBUG:boto:Params: {} 2024-03-04T17:53:45.017 DEBUG:boto:Token: None 2024-03-04T17:53:45.017 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.017 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xyC9whZmxIyqdSn4kUN2wbNGba0= 2024-03-04T17:53:45.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xyC9whZmxIyqdSn4kUN2wbNGba0=', 'Content-Length': '0'} 2024-03-04T17:53:45.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077cbc5c5671cfeb6-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.020 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.021 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.021 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.021 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.021 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.021 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.021 DEBUG:boto:Data: 2024-03-04T17:53:45.021 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.021 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.021 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.021 DEBUG:boto:Params: {} 2024-03-04T17:53:45.022 DEBUG:boto:Token: None 2024-03-04T17:53:45.022 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.022 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xyC9whZmxIyqdSn4kUN2wbNGba0= 2024-03-04T17:53:45.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xyC9whZmxIyqdSn4kUN2wbNGba0=', 'Content-Length': '0'} 2024-03-04T17:53:45.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc3058d4c371ccdb-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:45.025 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.025 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.025 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.025 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.025 DEBUG:boto:Method: GET 2024-03-04T17:53:45.025 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.026 DEBUG:boto:Data: 2024-03-04T17:53:45.026 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.026 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.026 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.026 DEBUG:boto:Params: {} 2024-03-04T17:53:45.026 DEBUG:boto:Token: None 2024-03-04T17:53:45.026 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.026 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:geRTBWIBttPjgi2eDBxQB/AOIWY= 2024-03-04T17:53:45.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:geRTBWIBttPjgi2eDBxQB/AOIWY=', 'Content-Length': '0'} 2024-03-04T17:53:45.029 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe44d3fb3862d026-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.029 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.030 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.030 DEBUG:boto:path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.030 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.030 DEBUG:boto:Method: GET 2024-03-04T17:53:45.030 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.030 DEBUG:boto:Data: 2024-03-04T17:53:45.030 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.030 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.030 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.030 DEBUG:boto:Params: {} 2024-03-04T17:53:45.030 DEBUG:boto:Token: None 2024-03-04T17:53:45.031 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G 2024-03-04T17:53:45.031 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:geRTBWIBttPjgi2eDBxQB/AOIWY= 2024-03-04T17:53:45.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:geRTBWIBttPjgi2eDBxQB/AOIWY=', 'Content-Length': '0'} 2024-03-04T17:53:45.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Zw7l4YYB37p1xa7jtLR7kDu1d5Xjl6G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004789374064f14a1e-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.076 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.076 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.076 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.076 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.076 DEBUG:boto:Method: GET 2024-03-04T17:53:45.076 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.077 DEBUG:boto:Data: 2024-03-04T17:53:45.077 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.077 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.077 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.077 DEBUG:boto:Params: {} 2024-03-04T17:53:45.077 DEBUG:boto:Token: None 2024-03-04T17:53:45.077 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.077 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a60d512dd066772a-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.082 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.083 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.083 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.083 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.083 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.083 DEBUG:boto:Method: GET 2024-03-04T17:53:45.083 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.083 DEBUG:boto:Data: 2024-03-04T17:53:45.083 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.083 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.083 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.083 DEBUG:boto:Params: {} 2024-03-04T17:53:45.084 DEBUG:boto:Token: None 2024-03-04T17:53:45.084 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.084 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000037b6db82ad32468-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.089 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.089 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.089 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.089 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.089 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:45.089 DEBUG:boto:Data: 2024-03-04T17:53:45.089 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.090 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.090 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.090 DEBUG:boto:Params: {} 2024-03-04T17:53:45.090 DEBUG:boto:Token: None 2024-03-04T17:53:45.090 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj 2024-03-04T17:53:45.090 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o= 2024-03-04T17:53:45.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o=', 'Content-Length': '0'} 2024-03-04T17:53:45.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008939d870c47c7a51-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.096 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.096 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.096 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.097 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:45.097 DEBUG:boto:Data: 2024-03-04T17:53:45.097 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.097 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.097 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.097 DEBUG:boto:Params: {} 2024-03-04T17:53:45.097 DEBUG:boto:Token: None 2024-03-04T17:53:45.097 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj 2024-03-04T17:53:45.097 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o= 2024-03-04T17:53:45.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o=', 'Content-Length': '0'} 2024-03-04T17:53:45.104 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090f459bcf09007dc-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:45.104 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.104 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.104 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.105 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.105 DEBUG:boto:Method: GET 2024-03-04T17:53:45.105 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.105 DEBUG:boto:Data: 2024-03-04T17:53:45.105 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.105 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.105 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.105 DEBUG:boto:Params: {} 2024-03-04T17:53:45.105 DEBUG:boto:Token: None 2024-03-04T17:53:45.105 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.105 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo= 2024-03-04T17:53:45.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo=', 'Content-Length': '0'} 2024-03-04T17:53:45.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd5f401f9b5c0ac0-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.152 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.152 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.152 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.152 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.153 DEBUG:boto:Method: GET 2024-03-04T17:53:45.153 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.153 DEBUG:boto:Data: 2024-03-04T17:53:45.153 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.153 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.153 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.153 DEBUG:boto:Params: {} 2024-03-04T17:53:45.153 DEBUG:boto:Token: None 2024-03-04T17:53:45.153 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.153 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo= 2024-03-04T17:53:45.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo=', 'Content-Length': '0'} 2024-03-04T17:53:45.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b49ce7e400077a5b-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.200 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.200 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.200 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.200 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.200 DEBUG:boto:Method: GET 2024-03-04T17:53:45.201 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.201 DEBUG:boto:Data: 2024-03-04T17:53:45.201 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.201 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.201 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.201 DEBUG:boto:Params: {} 2024-03-04T17:53:45.201 DEBUG:boto:Token: None 2024-03-04T17:53:45.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.201 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067088699a55926cc-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.206 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.206 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.207 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.207 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.207 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.207 DEBUG:boto:Method: GET 2024-03-04T17:53:45.207 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.207 DEBUG:boto:Data: 2024-03-04T17:53:45.207 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.207 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.207 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.207 DEBUG:boto:Params: {} 2024-03-04T17:53:45.207 DEBUG:boto:Token: None 2024-03-04T17:53:45.208 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.208 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006640472f7303935b-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.212 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:45.213 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.213 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.213 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.213 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.213 DEBUG:boto:Method: GET 2024-03-04T17:53:45.213 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.213 DEBUG:boto:Data: 2024-03-04T17:53:45.213 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.213 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.213 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.214 DEBUG:boto:Params: {} 2024-03-04T17:53:45.214 DEBUG:boto:Token: None 2024-03-04T17:53:45.214 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.214 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CeSZBcutEhD9btye9XxGUBGZHpk= 2024-03-04T17:53:45.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CeSZBcutEhD9btye9XxGUBGZHpk=', 'Content-Length': '0'} 2024-03-04T17:53:45.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb259da0d88bc0b1-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.217 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.217 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.218 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.218 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.218 DEBUG:boto:Method: GET 2024-03-04T17:53:45.218 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.218 DEBUG:boto:Data: 2024-03-04T17:53:45.218 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.218 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.218 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.218 DEBUG:boto:Params: {} 2024-03-04T17:53:45.218 DEBUG:boto:Token: None 2024-03-04T17:53:45.219 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.219 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CeSZBcutEhD9btye9XxGUBGZHpk= 2024-03-04T17:53:45.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CeSZBcutEhD9btye9XxGUBGZHpk=', 'Content-Length': '0'} 2024-03-04T17:53:45.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000feb9ac2ba4ca2023-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.264 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.264 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.264 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.264 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.264 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.264 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.265 DEBUG:boto:Data: 2024-03-04T17:53:45.265 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.265 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.265 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.265 DEBUG:boto:Params: {} 2024-03-04T17:53:45.265 DEBUG:boto:Token: None 2024-03-04T17:53:45.265 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.265 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9igBWQ4QR7mpuESasFmArmavRgM= 2024-03-04T17:53:45.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9igBWQ4QR7mpuESasFmArmavRgM=', 'Content-Length': '0'} 2024-03-04T17:53:45.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094a2cbea456a554a-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.269 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.269 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.269 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.269 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.269 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.269 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.269 DEBUG:boto:Data: 2024-03-04T17:53:45.269 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.269 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.270 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.270 DEBUG:boto:Params: {} 2024-03-04T17:53:45.270 DEBUG:boto:Token: None 2024-03-04T17:53:45.270 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.270 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9igBWQ4QR7mpuESasFmArmavRgM= 2024-03-04T17:53:45.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9igBWQ4QR7mpuESasFmArmavRgM=', 'Content-Length': '0'} 2024-03-04T17:53:45.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023bbd80173d5dce2-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:45.273 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.273 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.273 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.273 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.274 DEBUG:boto:Method: GET 2024-03-04T17:53:45.274 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.274 DEBUG:boto:Data: 2024-03-04T17:53:45.274 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.274 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.274 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.274 DEBUG:boto:Params: {} 2024-03-04T17:53:45.274 DEBUG:boto:Token: None 2024-03-04T17:53:45.274 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.275 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CeSZBcutEhD9btye9XxGUBGZHpk= 2024-03-04T17:53:45.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CeSZBcutEhD9btye9XxGUBGZHpk=', 'Content-Length': '0'} 2024-03-04T17:53:45.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010c25b9e4d10ea28-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.278 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.278 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.278 DEBUG:boto:path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.278 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.278 DEBUG:boto:Method: GET 2024-03-04T17:53:45.278 DEBUG:boto:Path: /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.278 DEBUG:boto:Data: 2024-03-04T17:53:45.279 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.279 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.279 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.279 DEBUG:boto:Params: {} 2024-03-04T17:53:45.279 DEBUG:boto:Token: None 2024-03-04T17:53:45.279 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy 2024-03-04T17:53:45.279 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CeSZBcutEhD9btye9XxGUBGZHpk= 2024-03-04T17:53:45.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CeSZBcutEhD9btye9XxGUBGZHpk=', 'Content-Length': '0'} 2024-03-04T17:53:45.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '10Vjj0fS8rYkr3OIS1pBSR-3fCH6svy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096cdb6865bdffc9b-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.324 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.324 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.324 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.324 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.324 DEBUG:boto:Method: GET 2024-03-04T17:53:45.325 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.325 DEBUG:boto:Data: 2024-03-04T17:53:45.325 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.325 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.325 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.325 DEBUG:boto:Params: {} 2024-03-04T17:53:45.325 DEBUG:boto:Token: None 2024-03-04T17:53:45.325 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.325 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da9cf54282e35d73-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.331 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.331 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.332 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.332 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.332 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.332 DEBUG:boto:Method: GET 2024-03-04T17:53:45.332 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.332 DEBUG:boto:Data: 2024-03-04T17:53:45.332 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.332 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.332 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.332 DEBUG:boto:Params: {} 2024-03-04T17:53:45.333 DEBUG:boto:Token: None 2024-03-04T17:53:45.333 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.333 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c07de0800e47baef-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.338 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.339 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.339 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.339 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.339 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:45.339 DEBUG:boto:Data: 2024-03-04T17:53:45.339 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.339 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.339 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.339 DEBUG:boto:Params: {} 2024-03-04T17:53:45.340 DEBUG:boto:Token: None 2024-03-04T17:53:45.340 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj 2024-03-04T17:53:45.340 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o= 2024-03-04T17:53:45.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o=', 'Content-Length': '0'} 2024-03-04T17:53:45.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f7b704384230aaa-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.346 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.346 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.346 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.346 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:45.346 DEBUG:boto:Data: 2024-03-04T17:53:45.346 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.346 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.346 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.346 DEBUG:boto:Params: {} 2024-03-04T17:53:45.347 DEBUG:boto:Token: None 2024-03-04T17:53:45.347 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj 2024-03-04T17:53:45.347 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o= 2024-03-04T17:53:45.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o=', 'Content-Length': '0'} 2024-03-04T17:53:45.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe0e4cf34186af9e-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:45.353 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.353 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.353 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.353 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.353 DEBUG:boto:Method: GET 2024-03-04T17:53:45.353 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.353 DEBUG:boto:Data: 2024-03-04T17:53:45.354 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.354 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.354 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.354 DEBUG:boto:Params: {} 2024-03-04T17:53:45.354 DEBUG:boto:Token: None 2024-03-04T17:53:45.354 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.354 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo= 2024-03-04T17:53:45.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo=', 'Content-Length': '0'} 2024-03-04T17:53:45.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a738917548cba388-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.400 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.400 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.400 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.400 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.400 DEBUG:boto:Method: GET 2024-03-04T17:53:45.401 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.401 DEBUG:boto:Data: 2024-03-04T17:53:45.401 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.401 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.401 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.401 DEBUG:boto:Params: {} 2024-03-04T17:53:45.401 DEBUG:boto:Token: None 2024-03-04T17:53:45.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.401 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo= 2024-03-04T17:53:45.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo=', 'Content-Length': '0'} 2024-03-04T17:53:45.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000690cb6c95f37a6be-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.452 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.452 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.452 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.452 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.452 DEBUG:boto:Method: GET 2024-03-04T17:53:45.452 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.453 DEBUG:boto:Data: 2024-03-04T17:53:45.453 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.453 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.453 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.453 DEBUG:boto:Params: {} 2024-03-04T17:53:45.453 DEBUG:boto:Token: None 2024-03-04T17:53:45.453 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.453 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aaffe2125ecb256-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.458 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.458 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.459 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.459 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.459 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.459 DEBUG:boto:Method: GET 2024-03-04T17:53:45.459 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.459 DEBUG:boto:Data: 2024-03-04T17:53:45.459 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.459 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.459 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.459 DEBUG:boto:Params: {} 2024-03-04T17:53:45.459 DEBUG:boto:Token: None 2024-03-04T17:53:45.460 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.460 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d0851e6a65247cc-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.465 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:45.465 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.465 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.465 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.465 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.465 DEBUG:boto:Method: GET 2024-03-04T17:53:45.465 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.466 DEBUG:boto:Data: 2024-03-04T17:53:45.466 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.466 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.466 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.466 DEBUG:boto:Params: {} 2024-03-04T17:53:45.466 DEBUG:boto:Token: None 2024-03-04T17:53:45.466 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.466 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8r3ixT45Va0GW2VRdUy+FrWbn/U= 2024-03-04T17:53:45.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8r3ixT45Va0GW2VRdUy+FrWbn/U=', 'Content-Length': '0'} 2024-03-04T17:53:45.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004619c627f3db0e10-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.512 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.512 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.512 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.512 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.512 DEBUG:boto:Method: GET 2024-03-04T17:53:45.513 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.513 DEBUG:boto:Data: 2024-03-04T17:53:45.513 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.513 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.513 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.513 DEBUG:boto:Params: {} 2024-03-04T17:53:45.513 DEBUG:boto:Token: None 2024-03-04T17:53:45.513 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.513 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8r3ixT45Va0GW2VRdUy+FrWbn/U= 2024-03-04T17:53:45.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8r3ixT45Va0GW2VRdUy+FrWbn/U=', 'Content-Length': '0'} 2024-03-04T17:53:45.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033258fd3017543c0-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.516 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.516 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.516 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.517 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.517 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.517 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.517 DEBUG:boto:Data: 2024-03-04T17:53:45.517 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.517 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.517 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.517 DEBUG:boto:Params: {} 2024-03-04T17:53:45.517 DEBUG:boto:Token: None 2024-03-04T17:53:45.517 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.518 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xtNacAe5NoNrjbmIdyqOXJOTMo4= 2024-03-04T17:53:45.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xtNacAe5NoNrjbmIdyqOXJOTMo4=', 'Content-Length': '0'} 2024-03-04T17:53:45.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a6bd726f246dbad-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.521 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.521 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.521 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.521 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.521 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.521 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.521 DEBUG:boto:Data: 2024-03-04T17:53:45.521 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.521 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.521 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.522 DEBUG:boto:Params: {} 2024-03-04T17:53:45.522 DEBUG:boto:Token: None 2024-03-04T17:53:45.522 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.522 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xtNacAe5NoNrjbmIdyqOXJOTMo4= 2024-03-04T17:53:45.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xtNacAe5NoNrjbmIdyqOXJOTMo4=', 'Content-Length': '0'} 2024-03-04T17:53:45.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069ad3d02b24d9edc-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:45.525 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.525 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.525 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.526 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.526 DEBUG:boto:Method: GET 2024-03-04T17:53:45.526 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.526 DEBUG:boto:Data: 2024-03-04T17:53:45.526 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.526 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.526 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.526 DEBUG:boto:Params: {} 2024-03-04T17:53:45.526 DEBUG:boto:Token: None 2024-03-04T17:53:45.526 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.527 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8r3ixT45Va0GW2VRdUy+FrWbn/U= 2024-03-04T17:53:45.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8r3ixT45Va0GW2VRdUy+FrWbn/U=', 'Content-Length': '0'} 2024-03-04T17:53:45.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006da572c05b8d65db-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.572 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.572 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.572 DEBUG:boto:path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.572 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.572 DEBUG:boto:Method: GET 2024-03-04T17:53:45.573 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.573 DEBUG:boto:Data: 2024-03-04T17:53:45.573 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.573 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.573 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.573 DEBUG:boto:Params: {} 2024-03-04T17:53:45.573 DEBUG:boto:Token: None 2024-03-04T17:53:45.573 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw 2024-03-04T17:53:45.573 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8r3ixT45Va0GW2VRdUy+FrWbn/U= 2024-03-04T17:53:45.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8r3ixT45Va0GW2VRdUy+FrWbn/U=', 'Content-Length': '0'} 2024-03-04T17:53:45.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Xak5SqkkJ7eVzuGI.8.u99lZqCC3tNw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000176ff63091585535-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.576 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.576 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.577 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.577 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.577 DEBUG:boto:Method: GET 2024-03-04T17:53:45.577 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.577 DEBUG:boto:Data: 2024-03-04T17:53:45.577 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.577 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.577 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.577 DEBUG:boto:Params: {} 2024-03-04T17:53:45.577 DEBUG:boto:Token: None 2024-03-04T17:53:45.578 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.578 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c027dea9f41060-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.582 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.583 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.583 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.583 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.583 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.583 DEBUG:boto:Method: GET 2024-03-04T17:53:45.583 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.583 DEBUG:boto:Data: 2024-03-04T17:53:45.583 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.583 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.583 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.583 DEBUG:boto:Params: {} 2024-03-04T17:53:45.584 DEBUG:boto:Token: None 2024-03-04T17:53:45.584 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.584 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000161f0f3849c83f4-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.632 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.632 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.632 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.633 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.633 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:45.633 DEBUG:boto:Data: 2024-03-04T17:53:45.633 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.633 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.633 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.633 DEBUG:boto:Params: {} 2024-03-04T17:53:45.633 DEBUG:boto:Token: None 2024-03-04T17:53:45.633 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj 2024-03-04T17:53:45.633 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o= 2024-03-04T17:53:45.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o=', 'Content-Length': '0'} 2024-03-04T17:53:45.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006676a811d64f3ae6-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.640 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.640 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.640 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.640 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:45.640 DEBUG:boto:Data: 2024-03-04T17:53:45.640 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.640 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.640 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.640 DEBUG:boto:Params: {} 2024-03-04T17:53:45.640 DEBUG:boto:Token: None 2024-03-04T17:53:45.641 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj 2024-03-04T17:53:45.641 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o= 2024-03-04T17:53:45.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o=', 'Content-Length': '0'} 2024-03-04T17:53:45.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000769986d1d8603099-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:45.647 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.647 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.647 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.647 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.647 DEBUG:boto:Method: GET 2024-03-04T17:53:45.647 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.647 DEBUG:boto:Data: 2024-03-04T17:53:45.647 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.647 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.647 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.648 DEBUG:boto:Params: {} 2024-03-04T17:53:45.648 DEBUG:boto:Token: None 2024-03-04T17:53:45.648 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.648 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo= 2024-03-04T17:53:45.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo=', 'Content-Length': '0'} 2024-03-04T17:53:45.651 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e49c4e4d1143c5e3-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.652 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.652 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.652 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.652 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.652 DEBUG:boto:Method: GET 2024-03-04T17:53:45.652 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.652 DEBUG:boto:Data: 2024-03-04T17:53:45.652 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.652 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.652 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.652 DEBUG:boto:Params: {} 2024-03-04T17:53:45.653 DEBUG:boto:Token: None 2024-03-04T17:53:45.653 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.653 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo= 2024-03-04T17:53:45.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo=', 'Content-Length': '0'} 2024-03-04T17:53:45.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e7d9efeb8adeaa4-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.696 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.696 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.696 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.696 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.696 DEBUG:boto:Method: GET 2024-03-04T17:53:45.697 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.697 DEBUG:boto:Data: 2024-03-04T17:53:45.697 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.697 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.697 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.697 DEBUG:boto:Params: {} 2024-03-04T17:53:45.697 DEBUG:boto:Token: None 2024-03-04T17:53:45.697 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.697 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe0a8e8e5603f768-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.702 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.703 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.703 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.703 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.703 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.703 DEBUG:boto:Method: GET 2024-03-04T17:53:45.703 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.703 DEBUG:boto:Data: 2024-03-04T17:53:45.703 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.703 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.703 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.703 DEBUG:boto:Params: {} 2024-03-04T17:53:45.704 DEBUG:boto:Token: None 2024-03-04T17:53:45.704 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.704 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed252a521169f716-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.752 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:45.753 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.753 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.753 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.753 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.753 DEBUG:boto:Method: GET 2024-03-04T17:53:45.753 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.753 DEBUG:boto:Data: 2024-03-04T17:53:45.753 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.753 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.753 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.753 DEBUG:boto:Params: {} 2024-03-04T17:53:45.754 DEBUG:boto:Token: None 2024-03-04T17:53:45.754 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.754 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zSfP1ZGs305B2eXUewGGxBZx2Hg= 2024-03-04T17:53:45.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zSfP1ZGs305B2eXUewGGxBZx2Hg=', 'Content-Length': '0'} 2024-03-04T17:53:45.757 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000facd429a086de1e8-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.757 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.757 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.757 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.758 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.758 DEBUG:boto:Method: GET 2024-03-04T17:53:45.758 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.758 DEBUG:boto:Data: 2024-03-04T17:53:45.758 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.758 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.758 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.758 DEBUG:boto:Params: {} 2024-03-04T17:53:45.758 DEBUG:boto:Token: None 2024-03-04T17:53:45.758 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.759 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zSfP1ZGs305B2eXUewGGxBZx2Hg= 2024-03-04T17:53:45.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zSfP1ZGs305B2eXUewGGxBZx2Hg=', 'Content-Length': '0'} 2024-03-04T17:53:45.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098dc46eee7fb366c-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.762 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.762 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.762 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.762 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.762 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.762 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.762 DEBUG:boto:Data: 2024-03-04T17:53:45.763 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.763 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.763 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.763 DEBUG:boto:Params: {} 2024-03-04T17:53:45.763 DEBUG:boto:Token: None 2024-03-04T17:53:45.763 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.763 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:59EE8D0wRBEktNutempxMHfa8Ko= 2024-03-04T17:53:45.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:59EE8D0wRBEktNutempxMHfa8Ko=', 'Content-Length': '0'} 2024-03-04T17:53:45.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000485a6a602e93308f-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.766 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.766 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.767 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.767 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.767 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.767 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.767 DEBUG:boto:Data: 2024-03-04T17:53:45.767 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.767 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.767 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.767 DEBUG:boto:Params: {} 2024-03-04T17:53:45.767 DEBUG:boto:Token: None 2024-03-04T17:53:45.767 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.768 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:59EE8D0wRBEktNutempxMHfa8Ko= 2024-03-04T17:53:45.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:59EE8D0wRBEktNutempxMHfa8Ko=', 'Content-Length': '0'} 2024-03-04T17:53:45.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfda0bbd090c2f42-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:45.771 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.771 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.771 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.771 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.771 DEBUG:boto:Method: GET 2024-03-04T17:53:45.771 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.771 DEBUG:boto:Data: 2024-03-04T17:53:45.771 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.772 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.772 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.772 DEBUG:boto:Params: {} 2024-03-04T17:53:45.772 DEBUG:boto:Token: None 2024-03-04T17:53:45.772 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.772 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zSfP1ZGs305B2eXUewGGxBZx2Hg= 2024-03-04T17:53:45.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zSfP1ZGs305B2eXUewGGxBZx2Hg=', 'Content-Length': '0'} 2024-03-04T17:53:45.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f0c7128d78218f1-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.816 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.816 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.816 DEBUG:boto:path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.816 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.816 DEBUG:boto:Method: GET 2024-03-04T17:53:45.817 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.817 DEBUG:boto:Data: 2024-03-04T17:53:45.817 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.817 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.817 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.817 DEBUG:boto:Params: {} 2024-03-04T17:53:45.817 DEBUG:boto:Token: None 2024-03-04T17:53:45.817 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF 2024-03-04T17:53:45.817 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zSfP1ZGs305B2eXUewGGxBZx2Hg= 2024-03-04T17:53:45.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zSfP1ZGs305B2eXUewGGxBZx2Hg=', 'Content-Length': '0'} 2024-03-04T17:53:45.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zyjvmlB6z-cVoQlri1b6gVvpnPVUCjF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b91a3baa6e2049d7-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.864 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.864 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.864 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.864 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.864 DEBUG:boto:Method: GET 2024-03-04T17:53:45.864 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.865 DEBUG:boto:Data: 2024-03-04T17:53:45.865 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.865 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.865 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.865 DEBUG:boto:Params: {} 2024-03-04T17:53:45.865 DEBUG:boto:Token: None 2024-03-04T17:53:45.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.865 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000940ceee6e9ffa72d-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.871 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.871 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.871 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.871 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.871 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.872 DEBUG:boto:Method: GET 2024-03-04T17:53:45.872 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.872 DEBUG:boto:Data: 2024-03-04T17:53:45.872 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.872 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.872 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.872 DEBUG:boto:Params: {} 2024-03-04T17:53:45.872 DEBUG:boto:Token: None 2024-03-04T17:53:45.872 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.873 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a2853815264db6b-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.924 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.924 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.924 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.924 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.925 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:45.925 DEBUG:boto:Data: 2024-03-04T17:53:45.925 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.925 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.925 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.925 DEBUG:boto:Params: {} 2024-03-04T17:53:45.925 DEBUG:boto:Token: None 2024-03-04T17:53:45.925 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj 2024-03-04T17:53:45.925 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o= 2024-03-04T17:53:45.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o=', 'Content-Length': '0'} 2024-03-04T17:53:45.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000721a6a077fe639f3-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.931 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.932 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.932 DEBUG:boto:Method: HEAD 2024-03-04T17:53:45.932 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:45.932 DEBUG:boto:Data: 2024-03-04T17:53:45.932 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.932 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.932 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.932 DEBUG:boto:Params: {} 2024-03-04T17:53:45.932 DEBUG:boto:Token: None 2024-03-04T17:53:45.932 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj 2024-03-04T17:53:45.933 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o= 2024-03-04T17:53:45.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qELtW87Lpm/kh5r86e9PdjQBl6o=', 'Content-Length': '0'} 2024-03-04T17:53:45.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b3b4cb9b769c689-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:45.939 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.939 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.939 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.939 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.939 DEBUG:boto:Method: GET 2024-03-04T17:53:45.940 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.940 DEBUG:boto:Data: 2024-03-04T17:53:45.940 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.940 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.940 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.940 DEBUG:boto:Params: {} 2024-03-04T17:53:45.940 DEBUG:boto:Token: None 2024-03-04T17:53:45.940 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.940 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo= 2024-03-04T17:53:45.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo=', 'Content-Length': '0'} 2024-03-04T17:53:45.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008214543600fa4a94-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.944 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:45.944 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:45.944 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.944 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.944 DEBUG:boto:Method: GET 2024-03-04T17:53:45.944 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.944 DEBUG:boto:Data: 2024-03-04T17:53:45.944 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.945 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.945 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.945 DEBUG:boto:Params: {} 2024-03-04T17:53:45.945 DEBUG:boto:Token: None 2024-03-04T17:53:45.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:45.945 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo= 2024-03-04T17:53:45.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dv8KflwtKeMqY0HQjFQ6v0ZoJMo=', 'Content-Length': '0'} 2024-03-04T17:53:45.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e397583e3c04ddf-0065e60aa9-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.992 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.992 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.992 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.992 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.992 DEBUG:boto:Method: GET 2024-03-04T17:53:45.992 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.993 DEBUG:boto:Data: 2024-03-04T17:53:45.993 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.993 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:45.993 DEBUG:boto:Port: 8001 2024-03-04T17:53:45.993 DEBUG:boto:Params: {} 2024-03-04T17:53:45.993 DEBUG:boto:Token: None 2024-03-04T17:53:45.993 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:45 GMT /ppfahn-22/ 2024-03-04T17:53:45.993 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc= 2024-03-04T17:53:45.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Au2QaT5UrxjBUU2zPMaZkl+pYlc=', 'Content-Length': '0'} 2024-03-04T17:53:45.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcc6d66e1137fa64-0065e60aa9-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:45.998 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:45.998 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:45.999 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:45.999 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.999 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:45.999 DEBUG:boto:Method: GET 2024-03-04T17:53:45.999 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:45.999 DEBUG:boto:Data: 2024-03-04T17:53:45.999 DEBUG:boto:Headers: {} 2024-03-04T17:53:45.999 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:45.999 DEBUG:boto:Port: 8000 2024-03-04T17:53:45.999 DEBUG:boto:Params: {} 2024-03-04T17:53:45.999 DEBUG:boto:Token: None 2024-03-04T17:53:46.000 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/ 2024-03-04T17:53:46.000 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw= 2024-03-04T17:53:46.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw=', 'Content-Length': '0'} 2024-03-04T17:53:46.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eec3729b0848ca3-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.048 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:46.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:46.048 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.049 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.049 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.049 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.049 DEBUG:boto:Method: GET 2024-03-04T17:53:46.049 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.049 DEBUG:boto:Data: 2024-03-04T17:53:46.049 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.049 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.049 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.049 DEBUG:boto:Params: {} 2024-03-04T17:53:46.049 DEBUG:boto:Token: None 2024-03-04T17:53:46.050 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.050 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NBkXVykGPKrCP3av5yVpGtDMYFY= 2024-03-04T17:53:46.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NBkXVykGPKrCP3av5yVpGtDMYFY=', 'Content-Length': '0'} 2024-03-04T17:53:46.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000704f6cc81ba18541-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.096 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.096 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.096 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.096 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.096 DEBUG:boto:Method: GET 2024-03-04T17:53:46.096 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.097 DEBUG:boto:Data: 2024-03-04T17:53:46.097 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.097 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.097 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.097 DEBUG:boto:Params: {} 2024-03-04T17:53:46.097 DEBUG:boto:Token: None 2024-03-04T17:53:46.097 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.097 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NBkXVykGPKrCP3av5yVpGtDMYFY= 2024-03-04T17:53:46.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NBkXVykGPKrCP3av5yVpGtDMYFY=', 'Content-Length': '0'} 2024-03-04T17:53:46.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d5def5b2ea37322-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.100 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.100 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.101 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.101 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.101 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.101 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.101 DEBUG:boto:Data: 2024-03-04T17:53:46.101 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.101 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.101 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.101 DEBUG:boto:Params: {} 2024-03-04T17:53:46.101 DEBUG:boto:Token: None 2024-03-04T17:53:46.101 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.102 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zzjJTeDm/p1ZveUwwwpVpGWzvag= 2024-03-04T17:53:46.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zzjJTeDm/p1ZveUwwwpVpGWzvag=', 'Content-Length': '0'} 2024-03-04T17:53:46.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006030dbedceabc60f-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.105 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.105 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.105 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.105 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.105 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.105 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.105 DEBUG:boto:Data: 2024-03-04T17:53:46.106 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.106 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.106 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.106 DEBUG:boto:Params: {} 2024-03-04T17:53:46.106 DEBUG:boto:Token: None 2024-03-04T17:53:46.106 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.106 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zzjJTeDm/p1ZveUwwwpVpGWzvag= 2024-03-04T17:53:46.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zzjJTeDm/p1ZveUwwwpVpGWzvag=', 'Content-Length': '0'} 2024-03-04T17:53:46.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e6f597686f98b55-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:46.110 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.110 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.110 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.110 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.110 DEBUG:boto:Method: GET 2024-03-04T17:53:46.110 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.110 DEBUG:boto:Data: 2024-03-04T17:53:46.110 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.110 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.110 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.110 DEBUG:boto:Params: {} 2024-03-04T17:53:46.111 DEBUG:boto:Token: None 2024-03-04T17:53:46.111 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.111 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NBkXVykGPKrCP3av5yVpGtDMYFY= 2024-03-04T17:53:46.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NBkXVykGPKrCP3av5yVpGtDMYFY=', 'Content-Length': '0'} 2024-03-04T17:53:46.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d6726b9e8a4183a-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.160 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.160 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.160 DEBUG:boto:path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.160 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.160 DEBUG:boto:Method: GET 2024-03-04T17:53:46.160 DEBUG:boto:Path: /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.161 DEBUG:boto:Data: 2024-03-04T17:53:46.161 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.161 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.161 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.161 DEBUG:boto:Params: {} 2024-03-04T17:53:46.161 DEBUG:boto:Token: None 2024-03-04T17:53:46.161 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=6BBhxVS-A79T2MI6AHE.THWDO21iftf 2024-03-04T17:53:46.161 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NBkXVykGPKrCP3av5yVpGtDMYFY= 2024-03-04T17:53:46.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NBkXVykGPKrCP3av5yVpGtDMYFY=', 'Content-Length': '0'} 2024-03-04T17:53:46.164 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '6BBhxVS-A79T2MI6AHE.THWDO21iftf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000325906da9c3e8199-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.164 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:46.165 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:46.165 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.165 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.165 DEBUG:boto:Method: GET 2024-03-04T17:53:46.165 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:46.165 DEBUG:boto:Data: 2024-03-04T17:53:46.165 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.165 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.165 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.165 DEBUG:boto:Params: {} 2024-03-04T17:53:46.166 DEBUG:boto:Token: None 2024-03-04T17:53:46.166 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/ 2024-03-04T17:53:46.166 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw= 2024-03-04T17:53:46.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw=', 'Content-Length': '0'} 2024-03-04T17:53:46.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001451d0fdbd98b9ea-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.171 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:46.171 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:46.171 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:46.172 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.172 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.172 DEBUG:boto:Method: GET 2024-03-04T17:53:46.172 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:46.172 DEBUG:boto:Data: 2024-03-04T17:53:46.172 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.172 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.172 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.172 DEBUG:boto:Params: {} 2024-03-04T17:53:46.172 DEBUG:boto:Token: None 2024-03-04T17:53:46.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/ 2024-03-04T17:53:46.173 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw= 2024-03-04T17:53:46.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw=', 'Content-Length': '0'} 2024-03-04T17:53:46.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df7c303e2699634b-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.179 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:46.180 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.180 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.180 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.180 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:46.180 DEBUG:boto:Data: 2024-03-04T17:53:46.180 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.180 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.180 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.180 DEBUG:boto:Params: {} 2024-03-04T17:53:46.181 DEBUG:boto:Token: None 2024-03-04T17:53:46.181 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj 2024-03-04T17:53:46.181 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IP7D7TQrcJNNo3HoMWauyIMLHnE= 2024-03-04T17:53:46.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IP7D7TQrcJNNo3HoMWauyIMLHnE=', 'Content-Length': '0'} 2024-03-04T17:53:46.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000505ae772beb81943-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.187 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.187 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.187 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.187 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:46.187 DEBUG:boto:Data: 2024-03-04T17:53:46.187 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.187 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.187 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.188 DEBUG:boto:Params: {} 2024-03-04T17:53:46.188 DEBUG:boto:Token: None 2024-03-04T17:53:46.188 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj 2024-03-04T17:53:46.188 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IP7D7TQrcJNNo3HoMWauyIMLHnE= 2024-03-04T17:53:46.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IP7D7TQrcJNNo3HoMWauyIMLHnE=', 'Content-Length': '0'} 2024-03-04T17:53:46.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee8c40c5b9b39069-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:46.194 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.194 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.194 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.194 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.195 DEBUG:boto:Method: GET 2024-03-04T17:53:46.195 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.195 DEBUG:boto:Data: 2024-03-04T17:53:46.195 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.195 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.195 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.195 DEBUG:boto:Params: {} 2024-03-04T17:53:46.195 DEBUG:boto:Token: None 2024-03-04T17:53:46.195 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.195 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yik+4LGHUd0AX44NGM8CiX8+poU= 2024-03-04T17:53:46.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yik+4LGHUd0AX44NGM8CiX8+poU=', 'Content-Length': '0'} 2024-03-04T17:53:46.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a09d7695013ffd3-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.240 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.240 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.240 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.240 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.240 DEBUG:boto:Method: GET 2024-03-04T17:53:46.241 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.241 DEBUG:boto:Data: 2024-03-04T17:53:46.241 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.241 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.241 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.241 DEBUG:boto:Params: {} 2024-03-04T17:53:46.241 DEBUG:boto:Token: None 2024-03-04T17:53:46.241 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.241 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yik+4LGHUd0AX44NGM8CiX8+poU= 2024-03-04T17:53:46.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yik+4LGHUd0AX44NGM8CiX8+poU=', 'Content-Length': '0'} 2024-03-04T17:53:46.245 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000faf6b5ccca5c47d1-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.288 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:46.288 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:46.288 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.288 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.288 DEBUG:boto:Method: GET 2024-03-04T17:53:46.289 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:46.289 DEBUG:boto:Data: 2024-03-04T17:53:46.289 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.289 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.289 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.289 DEBUG:boto:Params: {} 2024-03-04T17:53:46.289 DEBUG:boto:Token: None 2024-03-04T17:53:46.289 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/ 2024-03-04T17:53:46.289 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw= 2024-03-04T17:53:46.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw=', 'Content-Length': '0'} 2024-03-04T17:53:46.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba27bfb1269809f2-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.295 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:46.295 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:46.296 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:46.296 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.296 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.296 DEBUG:boto:Method: GET 2024-03-04T17:53:46.296 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:46.296 DEBUG:boto:Data: 2024-03-04T17:53:46.296 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.296 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.296 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.296 DEBUG:boto:Params: {} 2024-03-04T17:53:46.297 DEBUG:boto:Token: None 2024-03-04T17:53:46.297 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/ 2024-03-04T17:53:46.297 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw= 2024-03-04T17:53:46.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw=', 'Content-Length': '0'} 2024-03-04T17:53:46.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c402af4e087ef017-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.301 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:46.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:46.302 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.302 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.302 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.302 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.302 DEBUG:boto:Method: GET 2024-03-04T17:53:46.302 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.302 DEBUG:boto:Data: 2024-03-04T17:53:46.303 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.303 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.303 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.303 DEBUG:boto:Params: {} 2024-03-04T17:53:46.303 DEBUG:boto:Token: None 2024-03-04T17:53:46.303 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.303 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WlH4EiN0dpLB3dId8amocNxfj7A= 2024-03-04T17:53:46.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WlH4EiN0dpLB3dId8amocNxfj7A=', 'Content-Length': '0'} 2024-03-04T17:53:46.306 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c58303b4a2c7cac2-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.306 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.307 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.307 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.307 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.307 DEBUG:boto:Method: GET 2024-03-04T17:53:46.307 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.307 DEBUG:boto:Data: 2024-03-04T17:53:46.307 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.307 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.307 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.307 DEBUG:boto:Params: {} 2024-03-04T17:53:46.307 DEBUG:boto:Token: None 2024-03-04T17:53:46.308 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.308 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WlH4EiN0dpLB3dId8amocNxfj7A= 2024-03-04T17:53:46.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WlH4EiN0dpLB3dId8amocNxfj7A=', 'Content-Length': '0'} 2024-03-04T17:53:46.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c49e7072695a311e-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.352 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.352 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.352 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.352 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.352 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.353 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.353 DEBUG:boto:Data: 2024-03-04T17:53:46.353 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.353 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.353 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.353 DEBUG:boto:Params: {} 2024-03-04T17:53:46.353 DEBUG:boto:Token: None 2024-03-04T17:53:46.353 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.353 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nRCRNEpT4t7JC3Dk3/IEAN5oWxY= 2024-03-04T17:53:46.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nRCRNEpT4t7JC3Dk3/IEAN5oWxY=', 'Content-Length': '0'} 2024-03-04T17:53:46.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044d9a7ae47b9cf83-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.357 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.357 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.357 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.357 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.357 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.357 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.357 DEBUG:boto:Data: 2024-03-04T17:53:46.357 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.357 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.358 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.358 DEBUG:boto:Params: {} 2024-03-04T17:53:46.358 DEBUG:boto:Token: None 2024-03-04T17:53:46.358 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.358 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nRCRNEpT4t7JC3Dk3/IEAN5oWxY= 2024-03-04T17:53:46.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nRCRNEpT4t7JC3Dk3/IEAN5oWxY=', 'Content-Length': '0'} 2024-03-04T17:53:46.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000aad074ea0c194ca-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:46.361 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.361 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.361 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.361 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.361 DEBUG:boto:Method: GET 2024-03-04T17:53:46.361 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.362 DEBUG:boto:Data: 2024-03-04T17:53:46.362 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.362 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.362 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.362 DEBUG:boto:Params: {} 2024-03-04T17:53:46.362 DEBUG:boto:Token: None 2024-03-04T17:53:46.362 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.362 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WlH4EiN0dpLB3dId8amocNxfj7A= 2024-03-04T17:53:46.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WlH4EiN0dpLB3dId8amocNxfj7A=', 'Content-Length': '0'} 2024-03-04T17:53:46.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d3ba67b796fb1d5-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.365 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.365 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.366 DEBUG:boto:path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.366 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.366 DEBUG:boto:Method: GET 2024-03-04T17:53:46.366 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.366 DEBUG:boto:Data: 2024-03-04T17:53:46.366 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.366 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.366 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.366 DEBUG:boto:Params: {} 2024-03-04T17:53:46.366 DEBUG:boto:Token: None 2024-03-04T17:53:46.366 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k 2024-03-04T17:53:46.367 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WlH4EiN0dpLB3dId8amocNxfj7A= 2024-03-04T17:53:46.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WlH4EiN0dpLB3dId8amocNxfj7A=', 'Content-Length': '0'} 2024-03-04T17:53:46.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9JNqYfSZ1lpLA2AqZrdOTAeSn8.W.7k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058e97ee0af6d31d6-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.416 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:46.416 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:46.416 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.416 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.417 DEBUG:boto:Method: GET 2024-03-04T17:53:46.417 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:46.417 DEBUG:boto:Data: 2024-03-04T17:53:46.417 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.417 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.417 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.417 DEBUG:boto:Params: {} 2024-03-04T17:53:46.417 DEBUG:boto:Token: None 2024-03-04T17:53:46.417 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/ 2024-03-04T17:53:46.417 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw= 2024-03-04T17:53:46.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw=', 'Content-Length': '0'} 2024-03-04T17:53:46.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cfa1da21d7880e9-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.422 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:46.422 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:46.422 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:46.423 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.423 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.423 DEBUG:boto:Method: GET 2024-03-04T17:53:46.423 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:46.423 DEBUG:boto:Data: 2024-03-04T17:53:46.423 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.423 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.423 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.423 DEBUG:boto:Params: {} 2024-03-04T17:53:46.423 DEBUG:boto:Token: None 2024-03-04T17:53:46.423 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/ 2024-03-04T17:53:46.424 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw= 2024-03-04T17:53:46.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw=', 'Content-Length': '0'} 2024-03-04T17:53:46.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ecdc99bb5a77c57-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.429 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:46.429 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.429 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.430 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.430 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:46.430 DEBUG:boto:Data: 2024-03-04T17:53:46.430 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.430 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.430 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.430 DEBUG:boto:Params: {} 2024-03-04T17:53:46.430 DEBUG:boto:Token: None 2024-03-04T17:53:46.430 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj 2024-03-04T17:53:46.430 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IP7D7TQrcJNNo3HoMWauyIMLHnE= 2024-03-04T17:53:46.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IP7D7TQrcJNNo3HoMWauyIMLHnE=', 'Content-Length': '0'} 2024-03-04T17:53:46.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000234e7688db56c477-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.436 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.437 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.437 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.437 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:46.437 DEBUG:boto:Data: 2024-03-04T17:53:46.437 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.437 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.437 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.437 DEBUG:boto:Params: {} 2024-03-04T17:53:46.437 DEBUG:boto:Token: None 2024-03-04T17:53:46.437 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj 2024-03-04T17:53:46.438 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IP7D7TQrcJNNo3HoMWauyIMLHnE= 2024-03-04T17:53:46.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IP7D7TQrcJNNo3HoMWauyIMLHnE=', 'Content-Length': '0'} 2024-03-04T17:53:46.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d913b095d335a69-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:46.445 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.445 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.445 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.445 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.445 DEBUG:boto:Method: GET 2024-03-04T17:53:46.445 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.445 DEBUG:boto:Data: 2024-03-04T17:53:46.445 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.445 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.445 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.445 DEBUG:boto:Params: {} 2024-03-04T17:53:46.446 DEBUG:boto:Token: None 2024-03-04T17:53:46.446 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.446 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yik+4LGHUd0AX44NGM8CiX8+poU= 2024-03-04T17:53:46.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yik+4LGHUd0AX44NGM8CiX8+poU=', 'Content-Length': '0'} 2024-03-04T17:53:46.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c72f2ea43884c434-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.492 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.492 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.492 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.492 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.492 DEBUG:boto:Method: GET 2024-03-04T17:53:46.492 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.492 DEBUG:boto:Data: 2024-03-04T17:53:46.493 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.493 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.493 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.493 DEBUG:boto:Params: {} 2024-03-04T17:53:46.493 DEBUG:boto:Token: None 2024-03-04T17:53:46.493 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.493 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yik+4LGHUd0AX44NGM8CiX8+poU= 2024-03-04T17:53:46.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yik+4LGHUd0AX44NGM8CiX8+poU=', 'Content-Length': '0'} 2024-03-04T17:53:46.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001535e610b64f8b52-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.540 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:46.540 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:46.540 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.540 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.541 DEBUG:boto:Method: GET 2024-03-04T17:53:46.541 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:46.541 DEBUG:boto:Data: 2024-03-04T17:53:46.541 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.541 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.541 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.541 DEBUG:boto:Params: {} 2024-03-04T17:53:46.541 DEBUG:boto:Token: None 2024-03-04T17:53:46.541 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/ 2024-03-04T17:53:46.541 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw= 2024-03-04T17:53:46.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw=', 'Content-Length': '0'} 2024-03-04T17:53:46.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d60b24dd93196c91-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.546 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:46.547 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:46.547 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:46.547 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.547 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.547 DEBUG:boto:Method: GET 2024-03-04T17:53:46.547 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:46.547 DEBUG:boto:Data: 2024-03-04T17:53:46.547 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.548 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.548 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.548 DEBUG:boto:Params: {} 2024-03-04T17:53:46.548 DEBUG:boto:Token: None 2024-03-04T17:53:46.548 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/ 2024-03-04T17:53:46.548 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw= 2024-03-04T17:53:46.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw=', 'Content-Length': '0'} 2024-03-04T17:53:46.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d9ec01bec3394c7-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.553 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:46.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:46.553 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.553 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.554 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.554 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.554 DEBUG:boto:Method: GET 2024-03-04T17:53:46.554 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.554 DEBUG:boto:Data: 2024-03-04T17:53:46.554 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.554 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.554 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.554 DEBUG:boto:Params: {} 2024-03-04T17:53:46.554 DEBUG:boto:Token: None 2024-03-04T17:53:46.555 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.555 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pmeK2hBsZCA42RGwf4Q62NFUzok= 2024-03-04T17:53:46.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pmeK2hBsZCA42RGwf4Q62NFUzok=', 'Content-Length': '0'} 2024-03-04T17:53:46.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000663768e2e6b1ea0f-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.558 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.558 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.558 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.558 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.558 DEBUG:boto:Method: GET 2024-03-04T17:53:46.558 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.559 DEBUG:boto:Data: 2024-03-04T17:53:46.559 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.559 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.559 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.559 DEBUG:boto:Params: {} 2024-03-04T17:53:46.559 DEBUG:boto:Token: None 2024-03-04T17:53:46.559 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.559 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pmeK2hBsZCA42RGwf4Q62NFUzok= 2024-03-04T17:53:46.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pmeK2hBsZCA42RGwf4Q62NFUzok=', 'Content-Length': '0'} 2024-03-04T17:53:46.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026f6188c31b367ba-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.563 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.563 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.563 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.563 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.563 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.563 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.563 DEBUG:boto:Data: 2024-03-04T17:53:46.563 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.563 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.564 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.564 DEBUG:boto:Params: {} 2024-03-04T17:53:46.564 DEBUG:boto:Token: None 2024-03-04T17:53:46.564 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.564 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K3CwmC+asVSgBTXMIAryxdoyEOA= 2024-03-04T17:53:46.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K3CwmC+asVSgBTXMIAryxdoyEOA=', 'Content-Length': '0'} 2024-03-04T17:53:46.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009600567c085cd93f-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.567 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.567 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.567 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.567 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.568 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.568 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.568 DEBUG:boto:Data: 2024-03-04T17:53:46.568 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.568 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.568 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.568 DEBUG:boto:Params: {} 2024-03-04T17:53:46.568 DEBUG:boto:Token: None 2024-03-04T17:53:46.568 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.569 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:K3CwmC+asVSgBTXMIAryxdoyEOA= 2024-03-04T17:53:46.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:K3CwmC+asVSgBTXMIAryxdoyEOA=', 'Content-Length': '0'} 2024-03-04T17:53:46.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085b9eb67cb365720-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:46.572 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.572 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.572 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.572 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.573 DEBUG:boto:Method: GET 2024-03-04T17:53:46.573 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.573 DEBUG:boto:Data: 2024-03-04T17:53:46.573 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.573 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.573 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.573 DEBUG:boto:Params: {} 2024-03-04T17:53:46.573 DEBUG:boto:Token: None 2024-03-04T17:53:46.573 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.573 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pmeK2hBsZCA42RGwf4Q62NFUzok= 2024-03-04T17:53:46.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pmeK2hBsZCA42RGwf4Q62NFUzok=', 'Content-Length': '0'} 2024-03-04T17:53:46.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000898973d898059eb3-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.620 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.620 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.620 DEBUG:boto:path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.620 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.620 DEBUG:boto:Method: GET 2024-03-04T17:53:46.621 DEBUG:boto:Path: /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.621 DEBUG:boto:Data: 2024-03-04T17:53:46.621 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.621 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.621 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.621 DEBUG:boto:Params: {} 2024-03-04T17:53:46.621 DEBUG:boto:Token: None 2024-03-04T17:53:46.621 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx 2024-03-04T17:53:46.621 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pmeK2hBsZCA42RGwf4Q62NFUzok= 2024-03-04T17:53:46.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pmeK2hBsZCA42RGwf4Q62NFUzok=', 'Content-Length': '0'} 2024-03-04T17:53:46.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'i.eNnYbM.4yY91CfDbCxKIqa-Uec-fx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f16aa0d0d921392e-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.668 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:46.668 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:46.668 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.668 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.668 DEBUG:boto:Method: GET 2024-03-04T17:53:46.669 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:46.669 DEBUG:boto:Data: 2024-03-04T17:53:46.669 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.669 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.669 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.669 DEBUG:boto:Params: {} 2024-03-04T17:53:46.669 DEBUG:boto:Token: None 2024-03-04T17:53:46.669 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/ 2024-03-04T17:53:46.669 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw= 2024-03-04T17:53:46.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw=', 'Content-Length': '0'} 2024-03-04T17:53:46.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4e7c929e6f4717c-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.716 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:46.716 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:46.716 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:46.716 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.717 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.717 DEBUG:boto:Method: GET 2024-03-04T17:53:46.717 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:46.717 DEBUG:boto:Data: 2024-03-04T17:53:46.717 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.717 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.717 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.717 DEBUG:boto:Params: {} 2024-03-04T17:53:46.717 DEBUG:boto:Token: None 2024-03-04T17:53:46.717 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/ 2024-03-04T17:53:46.717 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw= 2024-03-04T17:53:46.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw=', 'Content-Length': '0'} 2024-03-04T17:53:46.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001af4d1dc12289ae4-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.764 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:46.764 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.764 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.765 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.765 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:46.765 DEBUG:boto:Data: 2024-03-04T17:53:46.765 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.765 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.765 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.765 DEBUG:boto:Params: {} 2024-03-04T17:53:46.765 DEBUG:boto:Token: None 2024-03-04T17:53:46.765 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj 2024-03-04T17:53:46.766 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IP7D7TQrcJNNo3HoMWauyIMLHnE= 2024-03-04T17:53:46.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IP7D7TQrcJNNo3HoMWauyIMLHnE=', 'Content-Length': '0'} 2024-03-04T17:53:46.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047f3faeca5eede43-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.772 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.772 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.772 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.772 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:46.772 DEBUG:boto:Data: 2024-03-04T17:53:46.772 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.772 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.772 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.772 DEBUG:boto:Params: {} 2024-03-04T17:53:46.773 DEBUG:boto:Token: None 2024-03-04T17:53:46.773 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj 2024-03-04T17:53:46.773 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IP7D7TQrcJNNo3HoMWauyIMLHnE= 2024-03-04T17:53:46.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IP7D7TQrcJNNo3HoMWauyIMLHnE=', 'Content-Length': '0'} 2024-03-04T17:53:46.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fc83cc1c6f98a63-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:46.783 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.783 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.783 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.783 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.783 DEBUG:boto:Method: GET 2024-03-04T17:53:46.783 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.783 DEBUG:boto:Data: 2024-03-04T17:53:46.783 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.783 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.783 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.783 DEBUG:boto:Params: {} 2024-03-04T17:53:46.783 DEBUG:boto:Token: None 2024-03-04T17:53:46.784 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.784 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yik+4LGHUd0AX44NGM8CiX8+poU= 2024-03-04T17:53:46.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yik+4LGHUd0AX44NGM8CiX8+poU=', 'Content-Length': '0'} 2024-03-04T17:53:46.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000453995be7a47122b-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.787 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.788 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.788 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.788 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.788 DEBUG:boto:Method: GET 2024-03-04T17:53:46.788 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.788 DEBUG:boto:Data: 2024-03-04T17:53:46.788 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.788 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.788 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.788 DEBUG:boto:Params: {} 2024-03-04T17:53:46.789 DEBUG:boto:Token: None 2024-03-04T17:53:46.789 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:46.789 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yik+4LGHUd0AX44NGM8CiX8+poU= 2024-03-04T17:53:46.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yik+4LGHUd0AX44NGM8CiX8+poU=', 'Content-Length': '0'} 2024-03-04T17:53:46.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a608602cc7a549db-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.836 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:46.836 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:46.836 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.836 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.836 DEBUG:boto:Method: GET 2024-03-04T17:53:46.836 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:46.837 DEBUG:boto:Data: 2024-03-04T17:53:46.837 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.837 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.837 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.837 DEBUG:boto:Params: {} 2024-03-04T17:53:46.837 DEBUG:boto:Token: None 2024-03-04T17:53:46.837 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/ 2024-03-04T17:53:46.837 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw= 2024-03-04T17:53:46.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw=', 'Content-Length': '0'} 2024-03-04T17:53:46.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092d9d2d669dd17ad-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.842 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:46.843 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:46.843 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:46.843 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.843 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:46.843 DEBUG:boto:Method: GET 2024-03-04T17:53:46.843 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:46.843 DEBUG:boto:Data: 2024-03-04T17:53:46.843 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.843 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.843 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.844 DEBUG:boto:Params: {} 2024-03-04T17:53:46.844 DEBUG:boto:Token: None 2024-03-04T17:53:46.844 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/ 2024-03-04T17:53:46.844 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw= 2024-03-04T17:53:46.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7Tx+Nq3Pl3xHJojzphHgkb/ycSw=', 'Content-Length': '0'} 2024-03-04T17:53:46.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc0a7816767dac32-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.892 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:46.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:46.892 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.893 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.893 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.893 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.893 DEBUG:boto:Method: GET 2024-03-04T17:53:46.893 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.893 DEBUG:boto:Data: 2024-03-04T17:53:46.893 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.893 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.893 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.893 DEBUG:boto:Params: {} 2024-03-04T17:53:46.893 DEBUG:boto:Token: None 2024-03-04T17:53:46.894 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.894 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GKjY68V7KcqUhaIOpi/jPIeWr5g= 2024-03-04T17:53:46.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GKjY68V7KcqUhaIOpi/jPIeWr5g=', 'Content-Length': '0'} 2024-03-04T17:53:46.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7a379de9ca1d58d-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.940 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.940 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.940 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.940 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.940 DEBUG:boto:Method: GET 2024-03-04T17:53:46.940 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.940 DEBUG:boto:Data: 2024-03-04T17:53:46.941 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.941 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.941 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.941 DEBUG:boto:Params: {} 2024-03-04T17:53:46.941 DEBUG:boto:Token: None 2024-03-04T17:53:46.941 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.941 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GKjY68V7KcqUhaIOpi/jPIeWr5g= 2024-03-04T17:53:46.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GKjY68V7KcqUhaIOpi/jPIeWr5g=', 'Content-Length': '0'} 2024-03-04T17:53:46.944 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b560a4c06240e861-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.945 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.945 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.945 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.945 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.945 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.945 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.945 DEBUG:boto:Data: 2024-03-04T17:53:46.945 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.945 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.946 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.946 DEBUG:boto:Params: {} 2024-03-04T17:53:46.946 DEBUG:boto:Token: None 2024-03-04T17:53:46.946 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.946 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KRfHGp8ACIaK8Sr/7/XQFHo+jyI= 2024-03-04T17:53:46.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KRfHGp8ACIaK8Sr/7/XQFHo+jyI=', 'Content-Length': '0'} 2024-03-04T17:53:46.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d80333ea14a7a4f-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.950 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.950 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.950 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.950 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.950 DEBUG:boto:Method: HEAD 2024-03-04T17:53:46.950 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.951 DEBUG:boto:Data: 2024-03-04T17:53:46.951 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.951 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:46.951 DEBUG:boto:Port: 8000 2024-03-04T17:53:46.951 DEBUG:boto:Params: {} 2024-03-04T17:53:46.951 DEBUG:boto:Token: None 2024-03-04T17:53:46.951 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.951 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KRfHGp8ACIaK8Sr/7/XQFHo+jyI= 2024-03-04T17:53:46.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KRfHGp8ACIaK8Sr/7/XQFHo+jyI=', 'Content-Length': '0'} 2024-03-04T17:53:46.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007663f1275d4cad99-0065e60aaa-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:46.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:46.954 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:46.954 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:46.954 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.954 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.954 DEBUG:boto:Method: GET 2024-03-04T17:53:46.955 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.955 DEBUG:boto:Data: 2024-03-04T17:53:46.955 DEBUG:boto:Headers: {} 2024-03-04T17:53:46.955 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:46.955 DEBUG:boto:Port: 8001 2024-03-04T17:53:46.955 DEBUG:boto:Params: {} 2024-03-04T17:53:46.955 DEBUG:boto:Token: None 2024-03-04T17:53:46.955 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:46 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:46.955 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GKjY68V7KcqUhaIOpi/jPIeWr5g= 2024-03-04T17:53:46.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GKjY68V7KcqUhaIOpi/jPIeWr5g=', 'Content-Length': '0'} 2024-03-04T17:53:46.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005aaaa04c855adf35-0065e60aaa-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.000 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.000 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.000 DEBUG:boto:path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:47.000 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:47.000 DEBUG:boto:Method: GET 2024-03-04T17:53:47.001 DEBUG:boto:Path: /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:47.001 DEBUG:boto:Data: 2024-03-04T17:53:47.001 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.001 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.001 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.001 DEBUG:boto:Params: {} 2024-03-04T17:53:47.001 DEBUG:boto:Token: None 2024-03-04T17:53:47.001 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU 2024-03-04T17:53:47.001 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:emAQKs6dZd+Fvuxl48AsxbAM5kA= 2024-03-04T17:53:47.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:emAQKs6dZd+Fvuxl48AsxbAM5kA=', 'Content-Length': '0'} 2024-03-04T17:53:47.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'E8qLWRxKZb8m2Q9tz4kxLbIJ0tw0zkU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075f1f5ae68ab112a-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.048 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.048 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.048 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.048 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.048 DEBUG:boto:Method: GET 2024-03-04T17:53:47.049 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.049 DEBUG:boto:Data: 2024-03-04T17:53:47.049 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.049 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.049 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.049 DEBUG:boto:Params: {} 2024-03-04T17:53:47.049 DEBUG:boto:Token: None 2024-03-04T17:53:47.049 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.049 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f03379c7c6379c9-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.054 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.054 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.054 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.054 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.054 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.055 DEBUG:boto:Method: GET 2024-03-04T17:53:47.055 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.055 DEBUG:boto:Data: 2024-03-04T17:53:47.055 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.055 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.055 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.055 DEBUG:boto:Params: {} 2024-03-04T17:53:47.055 DEBUG:boto:Token: None 2024-03-04T17:53:47.055 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.055 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000814d413368b441e9-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.059 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.060 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.060 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.060 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.060 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:47.060 DEBUG:boto:Data: 2024-03-04T17:53:47.060 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.060 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.060 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.061 DEBUG:boto:Params: {} 2024-03-04T17:53:47.061 DEBUG:boto:Token: None 2024-03-04T17:53:47.061 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj 2024-03-04T17:53:47.061 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8= 2024-03-04T17:53:47.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8=', 'Content-Length': '0'} 2024-03-04T17:53:47.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b083577a3bbd7982-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.073 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.073 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.073 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.073 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:47.073 DEBUG:boto:Data: 2024-03-04T17:53:47.074 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.074 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.074 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.074 DEBUG:boto:Params: {} 2024-03-04T17:53:47.074 DEBUG:boto:Token: None 2024-03-04T17:53:47.074 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj 2024-03-04T17:53:47.074 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8= 2024-03-04T17:53:47.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8=', 'Content-Length': '0'} 2024-03-04T17:53:47.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f81326a38267ad8-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:47.089 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.089 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.089 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.089 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.089 DEBUG:boto:Method: GET 2024-03-04T17:53:47.089 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.089 DEBUG:boto:Data: 2024-03-04T17:53:47.089 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.089 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.090 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.090 DEBUG:boto:Params: {} 2024-03-04T17:53:47.090 DEBUG:boto:Token: None 2024-03-04T17:53:47.090 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.090 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y= 2024-03-04T17:53:47.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y=', 'Content-Length': '0'} 2024-03-04T17:53:47.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006ab934db9f98564-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.092 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.093 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.093 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.093 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.093 DEBUG:boto:Method: GET 2024-03-04T17:53:47.093 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.093 DEBUG:boto:Data: 2024-03-04T17:53:47.093 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.093 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.093 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.093 DEBUG:boto:Params: {} 2024-03-04T17:53:47.094 DEBUG:boto:Token: None 2024-03-04T17:53:47.094 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.094 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y= 2024-03-04T17:53:47.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y=', 'Content-Length': '0'} 2024-03-04T17:53:47.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd156dca5f63cf4c-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.140 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.140 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.140 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.140 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.140 DEBUG:boto:Method: GET 2024-03-04T17:53:47.140 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.141 DEBUG:boto:Data: 2024-03-04T17:53:47.141 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.141 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.141 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.141 DEBUG:boto:Params: {} 2024-03-04T17:53:47.141 DEBUG:boto:Token: None 2024-03-04T17:53:47.141 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.141 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002220953409c56355-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.145 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.145 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.145 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.145 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.145 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.146 DEBUG:boto:Method: GET 2024-03-04T17:53:47.146 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.146 DEBUG:boto:Data: 2024-03-04T17:53:47.146 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.146 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.146 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.146 DEBUG:boto:Params: {} 2024-03-04T17:53:47.146 DEBUG:boto:Token: None 2024-03-04T17:53:47.146 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.146 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee6f1dabc877940-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.151 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:47.151 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.151 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.151 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.151 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.152 DEBUG:boto:Method: GET 2024-03-04T17:53:47.152 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.152 DEBUG:boto:Data: 2024-03-04T17:53:47.152 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.152 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.152 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.152 DEBUG:boto:Params: {} 2024-03-04T17:53:47.152 DEBUG:boto:Token: None 2024-03-04T17:53:47.152 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.152 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TFXLNbazr99dmLnl2SXyMFmOhNw= 2024-03-04T17:53:47.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TFXLNbazr99dmLnl2SXyMFmOhNw=', 'Content-Length': '0'} 2024-03-04T17:53:47.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f907f1eb0d20e66-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.196 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.196 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.196 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.196 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.196 DEBUG:boto:Method: GET 2024-03-04T17:53:47.197 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.197 DEBUG:boto:Data: 2024-03-04T17:53:47.197 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.197 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.197 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.197 DEBUG:boto:Params: {} 2024-03-04T17:53:47.197 DEBUG:boto:Token: None 2024-03-04T17:53:47.197 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.197 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TFXLNbazr99dmLnl2SXyMFmOhNw= 2024-03-04T17:53:47.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TFXLNbazr99dmLnl2SXyMFmOhNw=', 'Content-Length': '0'} 2024-03-04T17:53:47.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087cd2311ba406e65-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.201 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.201 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.201 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.201 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.201 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.202 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.202 DEBUG:boto:Data: 2024-03-04T17:53:47.202 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.202 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.202 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.202 DEBUG:boto:Params: {} 2024-03-04T17:53:47.202 DEBUG:boto:Token: None 2024-03-04T17:53:47.202 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.202 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KpegeTPJICoYKI7lakqKlVIWqj4= 2024-03-04T17:53:47.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KpegeTPJICoYKI7lakqKlVIWqj4=', 'Content-Length': '0'} 2024-03-04T17:53:47.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2f93500a22c4e98-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.206 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.206 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.206 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.206 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.206 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.206 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.206 DEBUG:boto:Data: 2024-03-04T17:53:47.207 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.207 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.207 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.207 DEBUG:boto:Params: {} 2024-03-04T17:53:47.207 DEBUG:boto:Token: None 2024-03-04T17:53:47.207 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.207 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KpegeTPJICoYKI7lakqKlVIWqj4= 2024-03-04T17:53:47.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KpegeTPJICoYKI7lakqKlVIWqj4=', 'Content-Length': '0'} 2024-03-04T17:53:47.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2f70f124823e9ac-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:47.211 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.211 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.211 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.211 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.211 DEBUG:boto:Method: GET 2024-03-04T17:53:47.211 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.211 DEBUG:boto:Data: 2024-03-04T17:53:47.211 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.211 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.211 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.211 DEBUG:boto:Params: {} 2024-03-04T17:53:47.212 DEBUG:boto:Token: None 2024-03-04T17:53:47.212 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.212 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TFXLNbazr99dmLnl2SXyMFmOhNw= 2024-03-04T17:53:47.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TFXLNbazr99dmLnl2SXyMFmOhNw=', 'Content-Length': '0'} 2024-03-04T17:53:47.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000148cd6f582b61a31-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.260 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.260 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.260 DEBUG:boto:path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.260 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.260 DEBUG:boto:Method: GET 2024-03-04T17:53:47.260 DEBUG:boto:Path: /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.261 DEBUG:boto:Data: 2024-03-04T17:53:47.261 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.261 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.261 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.261 DEBUG:boto:Params: {} 2024-03-04T17:53:47.261 DEBUG:boto:Token: None 2024-03-04T17:53:47.261 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=e9K3WlwCsFUmkYneucDfMv9rNQO6cdp 2024-03-04T17:53:47.261 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TFXLNbazr99dmLnl2SXyMFmOhNw= 2024-03-04T17:53:47.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TFXLNbazr99dmLnl2SXyMFmOhNw=', 'Content-Length': '0'} 2024-03-04T17:53:47.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'e9K3WlwCsFUmkYneucDfMv9rNQO6cdp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017328d0b92ca0144-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.265 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.265 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.265 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.265 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.265 DEBUG:boto:Method: GET 2024-03-04T17:53:47.265 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.265 DEBUG:boto:Data: 2024-03-04T17:53:47.265 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.265 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.265 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.266 DEBUG:boto:Params: {} 2024-03-04T17:53:47.266 DEBUG:boto:Token: None 2024-03-04T17:53:47.266 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.266 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b6e5719247900b9-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.312 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.312 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.312 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.313 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.313 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.313 DEBUG:boto:Method: GET 2024-03-04T17:53:47.313 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.313 DEBUG:boto:Data: 2024-03-04T17:53:47.313 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.313 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.313 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.313 DEBUG:boto:Params: {} 2024-03-04T17:53:47.313 DEBUG:boto:Token: None 2024-03-04T17:53:47.314 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.314 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003167c545e1f28e35-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.319 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.319 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.319 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.320 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.320 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:47.320 DEBUG:boto:Data: 2024-03-04T17:53:47.320 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.320 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.320 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.320 DEBUG:boto:Params: {} 2024-03-04T17:53:47.320 DEBUG:boto:Token: None 2024-03-04T17:53:47.320 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj 2024-03-04T17:53:47.320 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8= 2024-03-04T17:53:47.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8=', 'Content-Length': '0'} 2024-03-04T17:53:47.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6a1b10e885a6236-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.327 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.327 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.327 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.327 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:47.327 DEBUG:boto:Data: 2024-03-04T17:53:47.327 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.327 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.328 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.328 DEBUG:boto:Params: {} 2024-03-04T17:53:47.328 DEBUG:boto:Token: None 2024-03-04T17:53:47.328 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj 2024-03-04T17:53:47.328 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8= 2024-03-04T17:53:47.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8=', 'Content-Length': '0'} 2024-03-04T17:53:47.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6c1ac448f2635a9-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:47.334 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.335 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.335 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.335 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.335 DEBUG:boto:Method: GET 2024-03-04T17:53:47.335 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.335 DEBUG:boto:Data: 2024-03-04T17:53:47.335 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.335 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.335 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.335 DEBUG:boto:Params: {} 2024-03-04T17:53:47.336 DEBUG:boto:Token: None 2024-03-04T17:53:47.336 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.336 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y= 2024-03-04T17:53:47.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y=', 'Content-Length': '0'} 2024-03-04T17:53:47.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008885c6ff6b829bef-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.339 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.339 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.339 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.339 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.339 DEBUG:boto:Method: GET 2024-03-04T17:53:47.340 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.340 DEBUG:boto:Data: 2024-03-04T17:53:47.340 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.340 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.340 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.340 DEBUG:boto:Params: {} 2024-03-04T17:53:47.340 DEBUG:boto:Token: None 2024-03-04T17:53:47.340 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.340 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y= 2024-03-04T17:53:47.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y=', 'Content-Length': '0'} 2024-03-04T17:53:47.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee38a9e7db569b8d-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.388 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.388 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.388 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.388 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.388 DEBUG:boto:Method: GET 2024-03-04T17:53:47.389 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.389 DEBUG:boto:Data: 2024-03-04T17:53:47.389 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.389 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.389 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.389 DEBUG:boto:Params: {} 2024-03-04T17:53:47.389 DEBUG:boto:Token: None 2024-03-04T17:53:47.389 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.389 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fcb8572d61ece81-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.394 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.394 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.394 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.395 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.395 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.395 DEBUG:boto:Method: GET 2024-03-04T17:53:47.395 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.395 DEBUG:boto:Data: 2024-03-04T17:53:47.395 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.395 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.395 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.395 DEBUG:boto:Params: {} 2024-03-04T17:53:47.395 DEBUG:boto:Token: None 2024-03-04T17:53:47.396 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.396 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049d77e7e1e065cc2-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.401 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:47.401 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.401 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.401 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.401 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.401 DEBUG:boto:Method: GET 2024-03-04T17:53:47.402 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.402 DEBUG:boto:Data: 2024-03-04T17:53:47.402 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.402 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.402 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.402 DEBUG:boto:Params: {} 2024-03-04T17:53:47.402 DEBUG:boto:Token: None 2024-03-04T17:53:47.402 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.402 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BFY0DMVIWhPyYrRlXYVPiCc93yk= 2024-03-04T17:53:47.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BFY0DMVIWhPyYrRlXYVPiCc93yk=', 'Content-Length': '0'} 2024-03-04T17:53:47.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026c0c225d711fa62-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.452 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.452 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.452 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.452 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.452 DEBUG:boto:Method: GET 2024-03-04T17:53:47.453 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.453 DEBUG:boto:Data: 2024-03-04T17:53:47.453 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.453 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.453 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.453 DEBUG:boto:Params: {} 2024-03-04T17:53:47.453 DEBUG:boto:Token: None 2024-03-04T17:53:47.453 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.453 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BFY0DMVIWhPyYrRlXYVPiCc93yk= 2024-03-04T17:53:47.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BFY0DMVIWhPyYrRlXYVPiCc93yk=', 'Content-Length': '0'} 2024-03-04T17:53:47.456 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f070e5614606fb73-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.457 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.457 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.457 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.457 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.457 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.457 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.457 DEBUG:boto:Data: 2024-03-04T17:53:47.457 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.458 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.458 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.458 DEBUG:boto:Params: {} 2024-03-04T17:53:47.458 DEBUG:boto:Token: None 2024-03-04T17:53:47.458 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.458 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jcarDS4ZgAuzEbQXF5K2GwvFbCc= 2024-03-04T17:53:47.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jcarDS4ZgAuzEbQXF5K2GwvFbCc=', 'Content-Length': '0'} 2024-03-04T17:53:47.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab8747ff4ef509d7-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.461 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.462 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.462 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.462 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.462 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.462 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.462 DEBUG:boto:Data: 2024-03-04T17:53:47.462 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.462 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.462 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.462 DEBUG:boto:Params: {} 2024-03-04T17:53:47.463 DEBUG:boto:Token: None 2024-03-04T17:53:47.463 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.463 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jcarDS4ZgAuzEbQXF5K2GwvFbCc= 2024-03-04T17:53:47.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jcarDS4ZgAuzEbQXF5K2GwvFbCc=', 'Content-Length': '0'} 2024-03-04T17:53:47.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9434d55be0acc34-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:47.466 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.466 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.466 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.466 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.467 DEBUG:boto:Method: GET 2024-03-04T17:53:47.467 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.467 DEBUG:boto:Data: 2024-03-04T17:53:47.467 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.467 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.467 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.467 DEBUG:boto:Params: {} 2024-03-04T17:53:47.467 DEBUG:boto:Token: None 2024-03-04T17:53:47.467 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.468 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BFY0DMVIWhPyYrRlXYVPiCc93yk= 2024-03-04T17:53:47.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BFY0DMVIWhPyYrRlXYVPiCc93yk=', 'Content-Length': '0'} 2024-03-04T17:53:47.470 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc92d49b45402821-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.471 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.471 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.471 DEBUG:boto:path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.471 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.471 DEBUG:boto:Method: GET 2024-03-04T17:53:47.471 DEBUG:boto:Path: /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.471 DEBUG:boto:Data: 2024-03-04T17:53:47.471 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.471 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.471 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.471 DEBUG:boto:Params: {} 2024-03-04T17:53:47.472 DEBUG:boto:Token: None 2024-03-04T17:53:47.472 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT 2024-03-04T17:53:47.472 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BFY0DMVIWhPyYrRlXYVPiCc93yk= 2024-03-04T17:53:47.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BFY0DMVIWhPyYrRlXYVPiCc93yk=', 'Content-Length': '0'} 2024-03-04T17:53:47.475 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'mO1G.dvjFKYGfDq4Jm4.6d7BwjH6gRT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007033b59664e03c56-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.520 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.520 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.520 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.520 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.520 DEBUG:boto:Method: GET 2024-03-04T17:53:47.521 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.521 DEBUG:boto:Data: 2024-03-04T17:53:47.521 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.521 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.521 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.521 DEBUG:boto:Params: {} 2024-03-04T17:53:47.521 DEBUG:boto:Token: None 2024-03-04T17:53:47.521 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.521 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016ed8e28b06a6c05-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.526 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.526 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.526 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.526 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.527 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.527 DEBUG:boto:Method: GET 2024-03-04T17:53:47.527 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.527 DEBUG:boto:Data: 2024-03-04T17:53:47.527 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.527 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.527 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.527 DEBUG:boto:Params: {} 2024-03-04T17:53:47.527 DEBUG:boto:Token: None 2024-03-04T17:53:47.528 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.528 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf63d4076870aa8-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.532 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.533 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.533 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.533 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.533 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:47.533 DEBUG:boto:Data: 2024-03-04T17:53:47.533 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.533 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.533 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.534 DEBUG:boto:Params: {} 2024-03-04T17:53:47.534 DEBUG:boto:Token: None 2024-03-04T17:53:47.534 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj 2024-03-04T17:53:47.534 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8= 2024-03-04T17:53:47.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8=', 'Content-Length': '0'} 2024-03-04T17:53:47.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010c0d43784561160-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.540 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.540 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.540 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.540 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:47.541 DEBUG:boto:Data: 2024-03-04T17:53:47.541 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.541 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.541 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.541 DEBUG:boto:Params: {} 2024-03-04T17:53:47.541 DEBUG:boto:Token: None 2024-03-04T17:53:47.541 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj 2024-03-04T17:53:47.541 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8= 2024-03-04T17:53:47.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8=', 'Content-Length': '0'} 2024-03-04T17:53:47.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068d0f9eb3c4d32af-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:47.547 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.548 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.548 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.548 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.548 DEBUG:boto:Method: GET 2024-03-04T17:53:47.548 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.548 DEBUG:boto:Data: 2024-03-04T17:53:47.548 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.548 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.548 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.548 DEBUG:boto:Params: {} 2024-03-04T17:53:47.549 DEBUG:boto:Token: None 2024-03-04T17:53:47.549 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.549 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y= 2024-03-04T17:53:47.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y=', 'Content-Length': '0'} 2024-03-04T17:53:47.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000309a05f1dd91431a-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.600 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.600 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.600 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.600 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.600 DEBUG:boto:Method: GET 2024-03-04T17:53:47.600 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.601 DEBUG:boto:Data: 2024-03-04T17:53:47.601 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.601 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.601 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.601 DEBUG:boto:Params: {} 2024-03-04T17:53:47.601 DEBUG:boto:Token: None 2024-03-04T17:53:47.601 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.601 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y= 2024-03-04T17:53:47.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y=', 'Content-Length': '0'} 2024-03-04T17:53:47.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6d817180e4b5163-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.648 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.648 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.648 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.648 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.648 DEBUG:boto:Method: GET 2024-03-04T17:53:47.649 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.649 DEBUG:boto:Data: 2024-03-04T17:53:47.649 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.649 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.649 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.649 DEBUG:boto:Params: {} 2024-03-04T17:53:47.649 DEBUG:boto:Token: None 2024-03-04T17:53:47.649 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.649 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f2036547b1d5380-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.654 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.654 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.654 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.655 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.655 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.655 DEBUG:boto:Method: GET 2024-03-04T17:53:47.655 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.655 DEBUG:boto:Data: 2024-03-04T17:53:47.655 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.655 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.655 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.655 DEBUG:boto:Params: {} 2024-03-04T17:53:47.655 DEBUG:boto:Token: None 2024-03-04T17:53:47.656 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.656 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d27dd9e9bee7240a-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.660 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:47.661 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.661 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.661 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.661 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.661 DEBUG:boto:Method: GET 2024-03-04T17:53:47.661 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.661 DEBUG:boto:Data: 2024-03-04T17:53:47.662 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.662 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.662 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.662 DEBUG:boto:Params: {} 2024-03-04T17:53:47.662 DEBUG:boto:Token: None 2024-03-04T17:53:47.662 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.662 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D9lHNd9WurWsl/NpZuLKjD+A/wE= 2024-03-04T17:53:47.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D9lHNd9WurWsl/NpZuLKjD+A/wE=', 'Content-Length': '0'} 2024-03-04T17:53:47.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f897daef90f1a8db-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.665 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.666 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.666 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.666 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.666 DEBUG:boto:Method: GET 2024-03-04T17:53:47.666 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.666 DEBUG:boto:Data: 2024-03-04T17:53:47.666 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.666 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.666 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.666 DEBUG:boto:Params: {} 2024-03-04T17:53:47.667 DEBUG:boto:Token: None 2024-03-04T17:53:47.667 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.667 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D9lHNd9WurWsl/NpZuLKjD+A/wE= 2024-03-04T17:53:47.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D9lHNd9WurWsl/NpZuLKjD+A/wE=', 'Content-Length': '0'} 2024-03-04T17:53:47.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cfd4ff9c127d4b7-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.670 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.670 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.670 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.670 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.671 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.671 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.671 DEBUG:boto:Data: 2024-03-04T17:53:47.671 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.671 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.671 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.671 DEBUG:boto:Params: {} 2024-03-04T17:53:47.671 DEBUG:boto:Token: None 2024-03-04T17:53:47.671 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.671 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qMH4valngme8NdhSErLMeM3Zw+0= 2024-03-04T17:53:47.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qMH4valngme8NdhSErLMeM3Zw+0=', 'Content-Length': '0'} 2024-03-04T17:53:47.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be869985737a81b5-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.674 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.675 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.675 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.675 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.675 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.675 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.675 DEBUG:boto:Data: 2024-03-04T17:53:47.675 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.675 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.675 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.676 DEBUG:boto:Params: {} 2024-03-04T17:53:47.676 DEBUG:boto:Token: None 2024-03-04T17:53:47.676 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.676 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qMH4valngme8NdhSErLMeM3Zw+0= 2024-03-04T17:53:47.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qMH4valngme8NdhSErLMeM3Zw+0=', 'Content-Length': '0'} 2024-03-04T17:53:47.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088ab38e01f9b687a-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:47.679 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.679 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.679 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.679 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.680 DEBUG:boto:Method: GET 2024-03-04T17:53:47.680 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.680 DEBUG:boto:Data: 2024-03-04T17:53:47.680 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.680 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.680 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.680 DEBUG:boto:Params: {} 2024-03-04T17:53:47.680 DEBUG:boto:Token: None 2024-03-04T17:53:47.680 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.680 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D9lHNd9WurWsl/NpZuLKjD+A/wE= 2024-03-04T17:53:47.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D9lHNd9WurWsl/NpZuLKjD+A/wE=', 'Content-Length': '0'} 2024-03-04T17:53:47.683 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007de056082ee10151-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.724 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.724 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.724 DEBUG:boto:path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.724 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.724 DEBUG:boto:Method: GET 2024-03-04T17:53:47.725 DEBUG:boto:Path: /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.725 DEBUG:boto:Data: 2024-03-04T17:53:47.725 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.725 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.725 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.725 DEBUG:boto:Params: {} 2024-03-04T17:53:47.725 DEBUG:boto:Token: None 2024-03-04T17:53:47.725 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT 2024-03-04T17:53:47.725 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D9lHNd9WurWsl/NpZuLKjD+A/wE= 2024-03-04T17:53:47.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D9lHNd9WurWsl/NpZuLKjD+A/wE=', 'Content-Length': '0'} 2024-03-04T17:53:47.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'gRxtOPcYyW7cm-IBYtAu0zyYNScRyoT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee14288d2297a619-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.729 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.729 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.729 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.729 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.729 DEBUG:boto:Method: GET 2024-03-04T17:53:47.729 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.729 DEBUG:boto:Data: 2024-03-04T17:53:47.729 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.729 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.729 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.730 DEBUG:boto:Params: {} 2024-03-04T17:53:47.730 DEBUG:boto:Token: None 2024-03-04T17:53:47.730 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.730 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5dd0a972e54649f-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.776 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.776 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.776 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.777 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.777 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.777 DEBUG:boto:Method: GET 2024-03-04T17:53:47.777 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.777 DEBUG:boto:Data: 2024-03-04T17:53:47.777 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.777 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.777 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.777 DEBUG:boto:Params: {} 2024-03-04T17:53:47.777 DEBUG:boto:Token: None 2024-03-04T17:53:47.778 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.778 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efae6a6f88e59ab7-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.782 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.783 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.783 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.783 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.783 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:47.783 DEBUG:boto:Data: 2024-03-04T17:53:47.783 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.783 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.783 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.783 DEBUG:boto:Params: {} 2024-03-04T17:53:47.784 DEBUG:boto:Token: None 2024-03-04T17:53:47.784 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj 2024-03-04T17:53:47.784 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8= 2024-03-04T17:53:47.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8=', 'Content-Length': '0'} 2024-03-04T17:53:47.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c577770c3676e881-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.790 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.790 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.790 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.790 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:47.790 DEBUG:boto:Data: 2024-03-04T17:53:47.790 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.791 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.791 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.791 DEBUG:boto:Params: {} 2024-03-04T17:53:47.791 DEBUG:boto:Token: None 2024-03-04T17:53:47.791 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj 2024-03-04T17:53:47.791 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8= 2024-03-04T17:53:47.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:AtYVtV9/60VzA1dAWIlDV5jEK/8=', 'Content-Length': '0'} 2024-03-04T17:53:47.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfa31012765f8251-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:47.798 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.798 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.798 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.798 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.798 DEBUG:boto:Method: GET 2024-03-04T17:53:47.798 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.798 DEBUG:boto:Data: 2024-03-04T17:53:47.798 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.798 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.799 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.799 DEBUG:boto:Params: {} 2024-03-04T17:53:47.799 DEBUG:boto:Token: None 2024-03-04T17:53:47.799 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.799 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y= 2024-03-04T17:53:47.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y=', 'Content-Length': '0'} 2024-03-04T17:53:47.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000746cc41bcfccc3c2-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.802 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.802 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.802 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.803 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.803 DEBUG:boto:Method: GET 2024-03-04T17:53:47.803 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.803 DEBUG:boto:Data: 2024-03-04T17:53:47.803 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.803 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.803 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.803 DEBUG:boto:Params: {} 2024-03-04T17:53:47.803 DEBUG:boto:Token: None 2024-03-04T17:53:47.803 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:47.804 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y= 2024-03-04T17:53:47.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LyXhacalatSe/+LYJXI3uiqGW3Y=', 'Content-Length': '0'} 2024-03-04T17:53:47.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9609f6807841dfe-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.848 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.848 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.848 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.848 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.848 DEBUG:boto:Method: GET 2024-03-04T17:53:47.849 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.849 DEBUG:boto:Data: 2024-03-04T17:53:47.849 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.849 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.849 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.849 DEBUG:boto:Params: {} 2024-03-04T17:53:47.849 DEBUG:boto:Token: None 2024-03-04T17:53:47.849 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.849 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000271817b9211b31df-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.855 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.855 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:47.855 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:47.855 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.855 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:47.855 DEBUG:boto:Method: GET 2024-03-04T17:53:47.856 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:47.856 DEBUG:boto:Data: 2024-03-04T17:53:47.856 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.856 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.856 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.856 DEBUG:boto:Params: {} 2024-03-04T17:53:47.856 DEBUG:boto:Token: None 2024-03-04T17:53:47.856 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/ 2024-03-04T17:53:47.856 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o= 2024-03-04T17:53:47.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ya0wDh0NiwcX+cW2vD2w3MfYo/o=', 'Content-Length': '0'} 2024-03-04T17:53:47.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8fdbfb6d276dad7-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.904 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:47.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:47.904 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.905 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.905 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.905 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.905 DEBUG:boto:Method: GET 2024-03-04T17:53:47.905 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.905 DEBUG:boto:Data: 2024-03-04T17:53:47.905 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.905 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.905 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.905 DEBUG:boto:Params: {} 2024-03-04T17:53:47.905 DEBUG:boto:Token: None 2024-03-04T17:53:47.906 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.906 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yGj1tOkotAFlBCqIse+TDz+Fc1o= 2024-03-04T17:53:47.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yGj1tOkotAFlBCqIse+TDz+Fc1o=', 'Content-Length': '0'} 2024-03-04T17:53:47.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb112a03db7a04d2-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.952 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.952 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.952 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.952 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.952 DEBUG:boto:Method: GET 2024-03-04T17:53:47.953 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.953 DEBUG:boto:Data: 2024-03-04T17:53:47.953 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.953 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.953 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.953 DEBUG:boto:Params: {} 2024-03-04T17:53:47.953 DEBUG:boto:Token: None 2024-03-04T17:53:47.953 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.953 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yGj1tOkotAFlBCqIse+TDz+Fc1o= 2024-03-04T17:53:47.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yGj1tOkotAFlBCqIse+TDz+Fc1o=', 'Content-Length': '0'} 2024-03-04T17:53:47.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ce258cb2cde2c9a-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.957 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.957 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.957 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.957 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.957 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.957 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.957 DEBUG:boto:Data: 2024-03-04T17:53:47.957 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.957 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.958 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.958 DEBUG:boto:Params: {} 2024-03-04T17:53:47.958 DEBUG:boto:Token: None 2024-03-04T17:53:47.958 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.958 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UPyyH7kpc5J34kkzaqp4imIhXBQ= 2024-03-04T17:53:47.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UPyyH7kpc5J34kkzaqp4imIhXBQ=', 'Content-Length': '0'} 2024-03-04T17:53:47.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bff6e17ed2dd1f5-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.961 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.961 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.962 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.962 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.962 DEBUG:boto:Method: HEAD 2024-03-04T17:53:47.962 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.962 DEBUG:boto:Data: 2024-03-04T17:53:47.962 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.962 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:47.962 DEBUG:boto:Port: 8000 2024-03-04T17:53:47.962 DEBUG:boto:Params: {} 2024-03-04T17:53:47.962 DEBUG:boto:Token: None 2024-03-04T17:53:47.963 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.963 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UPyyH7kpc5J34kkzaqp4imIhXBQ= 2024-03-04T17:53:47.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UPyyH7kpc5J34kkzaqp4imIhXBQ=', 'Content-Length': '0'} 2024-03-04T17:53:47.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d99288c1e05a3dbd-0065e60aab-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:47.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:47.966 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:47.966 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:47.966 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.966 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.966 DEBUG:boto:Method: GET 2024-03-04T17:53:47.966 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.967 DEBUG:boto:Data: 2024-03-04T17:53:47.967 DEBUG:boto:Headers: {} 2024-03-04T17:53:47.967 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:47.967 DEBUG:boto:Port: 8001 2024-03-04T17:53:47.967 DEBUG:boto:Params: {} 2024-03-04T17:53:47.967 DEBUG:boto:Token: None 2024-03-04T17:53:47.967 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:47 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:47.967 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yGj1tOkotAFlBCqIse+TDz+Fc1o= 2024-03-04T17:53:47.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yGj1tOkotAFlBCqIse+TDz+Fc1o=', 'Content-Length': '0'} 2024-03-04T17:53:47.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b264e80233e6dac0-0065e60aab-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.012 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.012 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.012 DEBUG:boto:path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:48.012 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:48.012 DEBUG:boto:Method: GET 2024-03-04T17:53:48.013 DEBUG:boto:Path: /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:48.013 DEBUG:boto:Data: 2024-03-04T17:53:48.013 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.013 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.013 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.013 DEBUG:boto:Params: {} 2024-03-04T17:53:48.013 DEBUG:boto:Token: None 2024-03-04T17:53:48.013 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR 2024-03-04T17:53:48.013 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3Bc48O32VUAiEH0tYe0m5oSI1sI= 2024-03-04T17:53:48.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3Bc48O32VUAiEH0tYe0m5oSI1sI=', 'Content-Length': '0'} 2024-03-04T17:53:48.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'me6.y9cJ8bkVcZ04PNMrwIXjDQUGzXR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072300230930130e3-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.060 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.060 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.060 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.060 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.060 DEBUG:boto:Method: GET 2024-03-04T17:53:48.061 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.061 DEBUG:boto:Data: 2024-03-04T17:53:48.061 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.061 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.061 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.061 DEBUG:boto:Params: {} 2024-03-04T17:53:48.061 DEBUG:boto:Token: None 2024-03-04T17:53:48.061 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.061 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b781e2488271a87-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.066 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.066 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.066 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.067 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.067 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.067 DEBUG:boto:Method: GET 2024-03-04T17:53:48.067 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.067 DEBUG:boto:Data: 2024-03-04T17:53:48.067 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.067 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.067 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.067 DEBUG:boto:Params: {} 2024-03-04T17:53:48.067 DEBUG:boto:Token: None 2024-03-04T17:53:48.068 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.068 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002867d8c6d1f2e99e-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.072 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.072 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.073 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.073 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.073 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:48.073 DEBUG:boto:Data: 2024-03-04T17:53:48.073 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.073 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.073 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.073 DEBUG:boto:Params: {} 2024-03-04T17:53:48.073 DEBUG:boto:Token: None 2024-03-04T17:53:48.074 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj 2024-03-04T17:53:48.074 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM= 2024-03-04T17:53:48.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM=', 'Content-Length': '0'} 2024-03-04T17:53:48.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b62a3d8e45dd2a7-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.080 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.080 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.080 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.080 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:48.080 DEBUG:boto:Data: 2024-03-04T17:53:48.080 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.080 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.080 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.080 DEBUG:boto:Params: {} 2024-03-04T17:53:48.080 DEBUG:boto:Token: None 2024-03-04T17:53:48.081 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj 2024-03-04T17:53:48.081 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM= 2024-03-04T17:53:48.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM=', 'Content-Length': '0'} 2024-03-04T17:53:48.087 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054a5958fb367490c-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:48.087 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.087 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.088 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.088 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.088 DEBUG:boto:Method: GET 2024-03-04T17:53:48.088 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.088 DEBUG:boto:Data: 2024-03-04T17:53:48.088 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.088 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.088 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.088 DEBUG:boto:Params: {} 2024-03-04T17:53:48.088 DEBUG:boto:Token: None 2024-03-04T17:53:48.089 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.089 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo= 2024-03-04T17:53:48.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo=', 'Content-Length': '0'} 2024-03-04T17:53:48.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b660cf98c23a157d-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.092 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.092 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.092 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.092 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.092 DEBUG:boto:Method: GET 2024-03-04T17:53:48.092 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.092 DEBUG:boto:Data: 2024-03-04T17:53:48.093 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.093 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.093 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.093 DEBUG:boto:Params: {} 2024-03-04T17:53:48.093 DEBUG:boto:Token: None 2024-03-04T17:53:48.093 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.093 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo= 2024-03-04T17:53:48.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo=', 'Content-Length': '0'} 2024-03-04T17:53:48.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f995979131d116b4-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.140 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.140 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.140 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.140 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.140 DEBUG:boto:Method: GET 2024-03-04T17:53:48.140 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.141 DEBUG:boto:Data: 2024-03-04T17:53:48.141 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.141 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.141 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.141 DEBUG:boto:Params: {} 2024-03-04T17:53:48.141 DEBUG:boto:Token: None 2024-03-04T17:53:48.141 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.141 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000850f31b2bcd5091f-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.147 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.147 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.147 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.147 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.147 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.148 DEBUG:boto:Method: GET 2024-03-04T17:53:48.148 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.148 DEBUG:boto:Data: 2024-03-04T17:53:48.148 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.148 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.148 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.148 DEBUG:boto:Params: {} 2024-03-04T17:53:48.148 DEBUG:boto:Token: None 2024-03-04T17:53:48.148 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.148 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3ad9f0d88974c04-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.154 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:48.154 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.154 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.154 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.154 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.155 DEBUG:boto:Method: GET 2024-03-04T17:53:48.155 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.155 DEBUG:boto:Data: 2024-03-04T17:53:48.155 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.155 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.155 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.155 DEBUG:boto:Params: {} 2024-03-04T17:53:48.155 DEBUG:boto:Token: None 2024-03-04T17:53:48.155 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.155 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kabIvMFBhXGZUNgdgcEV5xCPEBM= 2024-03-04T17:53:48.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kabIvMFBhXGZUNgdgcEV5xCPEBM=', 'Content-Length': '0'} 2024-03-04T17:53:48.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049af41997b3bd50c-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.200 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.200 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.200 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.200 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.200 DEBUG:boto:Method: GET 2024-03-04T17:53:48.200 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.201 DEBUG:boto:Data: 2024-03-04T17:53:48.201 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.201 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.201 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.201 DEBUG:boto:Params: {} 2024-03-04T17:53:48.201 DEBUG:boto:Token: None 2024-03-04T17:53:48.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.201 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kabIvMFBhXGZUNgdgcEV5xCPEBM= 2024-03-04T17:53:48.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kabIvMFBhXGZUNgdgcEV5xCPEBM=', 'Content-Length': '0'} 2024-03-04T17:53:48.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005876413b9591af08-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.205 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.205 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.205 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.206 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.206 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.206 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.206 DEBUG:boto:Data: 2024-03-04T17:53:48.206 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.206 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.206 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.206 DEBUG:boto:Params: {} 2024-03-04T17:53:48.206 DEBUG:boto:Token: None 2024-03-04T17:53:48.206 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.207 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bOr5UxKSBiU6Qb748ocFAwCQMzc= 2024-03-04T17:53:48.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bOr5UxKSBiU6Qb748ocFAwCQMzc=', 'Content-Length': '0'} 2024-03-04T17:53:48.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc5a91cd6acdb001-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.210 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.210 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.210 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.210 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.210 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.210 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.210 DEBUG:boto:Data: 2024-03-04T17:53:48.210 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.211 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.211 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.211 DEBUG:boto:Params: {} 2024-03-04T17:53:48.211 DEBUG:boto:Token: None 2024-03-04T17:53:48.211 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.211 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bOr5UxKSBiU6Qb748ocFAwCQMzc= 2024-03-04T17:53:48.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bOr5UxKSBiU6Qb748ocFAwCQMzc=', 'Content-Length': '0'} 2024-03-04T17:53:48.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f6ad22ade6e7bf5-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:48.214 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.215 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.215 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.215 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.215 DEBUG:boto:Method: GET 2024-03-04T17:53:48.215 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.215 DEBUG:boto:Data: 2024-03-04T17:53:48.215 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.215 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.215 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.215 DEBUG:boto:Params: {} 2024-03-04T17:53:48.215 DEBUG:boto:Token: None 2024-03-04T17:53:48.216 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.216 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kabIvMFBhXGZUNgdgcEV5xCPEBM= 2024-03-04T17:53:48.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kabIvMFBhXGZUNgdgcEV5xCPEBM=', 'Content-Length': '0'} 2024-03-04T17:53:48.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005212b6aacce3a936-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.219 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.220 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.220 DEBUG:boto:path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.220 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.220 DEBUG:boto:Method: GET 2024-03-04T17:53:48.220 DEBUG:boto:Path: /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.220 DEBUG:boto:Data: 2024-03-04T17:53:48.220 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.220 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.220 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.220 DEBUG:boto:Params: {} 2024-03-04T17:53:48.221 DEBUG:boto:Token: None 2024-03-04T17:53:48.221 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc 2024-03-04T17:53:48.221 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:kabIvMFBhXGZUNgdgcEV5xCPEBM= 2024-03-04T17:53:48.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kabIvMFBhXGZUNgdgcEV5xCPEBM=', 'Content-Length': '0'} 2024-03-04T17:53:48.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'oHs8pGuS1bc-GYUbjIaqPQPwQGDukkc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b05ce784855b5b3-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.264 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.264 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.264 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.264 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.264 DEBUG:boto:Method: GET 2024-03-04T17:53:48.264 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.265 DEBUG:boto:Data: 2024-03-04T17:53:48.265 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.265 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.265 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.265 DEBUG:boto:Params: {} 2024-03-04T17:53:48.265 DEBUG:boto:Token: None 2024-03-04T17:53:48.265 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.265 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3961b78b975a160-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.270 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.271 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.271 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.271 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.271 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.271 DEBUG:boto:Method: GET 2024-03-04T17:53:48.271 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.271 DEBUG:boto:Data: 2024-03-04T17:53:48.271 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.271 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.272 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.272 DEBUG:boto:Params: {} 2024-03-04T17:53:48.272 DEBUG:boto:Token: None 2024-03-04T17:53:48.272 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.272 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045318f3c5ca068e9-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.277 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.277 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.277 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.277 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.278 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:48.278 DEBUG:boto:Data: 2024-03-04T17:53:48.278 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.278 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.278 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.278 DEBUG:boto:Params: {} 2024-03-04T17:53:48.278 DEBUG:boto:Token: None 2024-03-04T17:53:48.278 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj 2024-03-04T17:53:48.278 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM= 2024-03-04T17:53:48.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM=', 'Content-Length': '0'} 2024-03-04T17:53:48.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa1b888762d6a059-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.284 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.284 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.284 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.284 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:48.285 DEBUG:boto:Data: 2024-03-04T17:53:48.285 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.285 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.285 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.285 DEBUG:boto:Params: {} 2024-03-04T17:53:48.285 DEBUG:boto:Token: None 2024-03-04T17:53:48.285 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj 2024-03-04T17:53:48.285 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM= 2024-03-04T17:53:48.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM=', 'Content-Length': '0'} 2024-03-04T17:53:48.291 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d0f490738ca5220-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:48.291 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.291 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.291 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.292 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.292 DEBUG:boto:Method: GET 2024-03-04T17:53:48.292 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.292 DEBUG:boto:Data: 2024-03-04T17:53:48.292 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.292 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.292 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.292 DEBUG:boto:Params: {} 2024-03-04T17:53:48.292 DEBUG:boto:Token: None 2024-03-04T17:53:48.293 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.293 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo= 2024-03-04T17:53:48.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo=', 'Content-Length': '0'} 2024-03-04T17:53:48.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2a5ed912c8b370f-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.336 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.336 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.336 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.336 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.336 DEBUG:boto:Method: GET 2024-03-04T17:53:48.337 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.337 DEBUG:boto:Data: 2024-03-04T17:53:48.337 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.337 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.337 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.337 DEBUG:boto:Params: {} 2024-03-04T17:53:48.337 DEBUG:boto:Token: None 2024-03-04T17:53:48.337 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.337 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo= 2024-03-04T17:53:48.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo=', 'Content-Length': '0'} 2024-03-04T17:53:48.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004042fd99db242361-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.384 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.384 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.384 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.384 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.384 DEBUG:boto:Method: GET 2024-03-04T17:53:48.384 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.385 DEBUG:boto:Data: 2024-03-04T17:53:48.385 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.385 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.385 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.385 DEBUG:boto:Params: {} 2024-03-04T17:53:48.385 DEBUG:boto:Token: None 2024-03-04T17:53:48.385 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.385 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d1a4aa39dcd47f9-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.391 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.391 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.391 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.391 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.391 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.392 DEBUG:boto:Method: GET 2024-03-04T17:53:48.392 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.392 DEBUG:boto:Data: 2024-03-04T17:53:48.392 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.392 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.392 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.392 DEBUG:boto:Params: {} 2024-03-04T17:53:48.392 DEBUG:boto:Token: None 2024-03-04T17:53:48.392 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.393 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008713971f1200e20e-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.397 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:48.397 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.398 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.398 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.398 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.398 DEBUG:boto:Method: GET 2024-03-04T17:53:48.398 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.398 DEBUG:boto:Data: 2024-03-04T17:53:48.398 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.398 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.398 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.399 DEBUG:boto:Params: {} 2024-03-04T17:53:48.399 DEBUG:boto:Token: None 2024-03-04T17:53:48.399 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.399 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hmxw+5jGIDZX6U1JOlPYjzBXRKQ= 2024-03-04T17:53:48.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hmxw+5jGIDZX6U1JOlPYjzBXRKQ=', 'Content-Length': '0'} 2024-03-04T17:53:48.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046d8b783608e6421-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.444 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.444 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.444 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.444 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.444 DEBUG:boto:Method: GET 2024-03-04T17:53:48.445 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.445 DEBUG:boto:Data: 2024-03-04T17:53:48.445 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.445 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.445 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.445 DEBUG:boto:Params: {} 2024-03-04T17:53:48.445 DEBUG:boto:Token: None 2024-03-04T17:53:48.445 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.445 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hmxw+5jGIDZX6U1JOlPYjzBXRKQ= 2024-03-04T17:53:48.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hmxw+5jGIDZX6U1JOlPYjzBXRKQ=', 'Content-Length': '0'} 2024-03-04T17:53:48.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000635ac9ffad9dd082-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.449 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.449 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.449 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.449 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.449 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.450 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.450 DEBUG:boto:Data: 2024-03-04T17:53:48.450 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.450 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.450 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.450 DEBUG:boto:Params: {} 2024-03-04T17:53:48.450 DEBUG:boto:Token: None 2024-03-04T17:53:48.450 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.450 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D6Nwx0VxKE+ai9sW7oauVNTzDM8= 2024-03-04T17:53:48.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D6Nwx0VxKE+ai9sW7oauVNTzDM8=', 'Content-Length': '0'} 2024-03-04T17:53:48.453 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000295114156eb77c64-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.454 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.454 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.454 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.454 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.454 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.454 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.454 DEBUG:boto:Data: 2024-03-04T17:53:48.454 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.454 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.455 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.455 DEBUG:boto:Params: {} 2024-03-04T17:53:48.455 DEBUG:boto:Token: None 2024-03-04T17:53:48.455 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.455 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D6Nwx0VxKE+ai9sW7oauVNTzDM8= 2024-03-04T17:53:48.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D6Nwx0VxKE+ai9sW7oauVNTzDM8=', 'Content-Length': '0'} 2024-03-04T17:53:48.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f98fc21eb3ea9e3-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:48.458 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.458 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.459 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.459 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.459 DEBUG:boto:Method: GET 2024-03-04T17:53:48.459 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.459 DEBUG:boto:Data: 2024-03-04T17:53:48.459 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.459 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.459 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.459 DEBUG:boto:Params: {} 2024-03-04T17:53:48.459 DEBUG:boto:Token: None 2024-03-04T17:53:48.460 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.460 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hmxw+5jGIDZX6U1JOlPYjzBXRKQ= 2024-03-04T17:53:48.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hmxw+5jGIDZX6U1JOlPYjzBXRKQ=', 'Content-Length': '0'} 2024-03-04T17:53:48.463 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d39e279099f9cad2-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.504 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.504 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.504 DEBUG:boto:path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.504 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.504 DEBUG:boto:Method: GET 2024-03-04T17:53:48.505 DEBUG:boto:Path: /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.505 DEBUG:boto:Data: 2024-03-04T17:53:48.505 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.505 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.505 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.505 DEBUG:boto:Params: {} 2024-03-04T17:53:48.505 DEBUG:boto:Token: None 2024-03-04T17:53:48.505 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=160IY3UeXb5qH4Rh4JB09PNigxsTa5g 2024-03-04T17:53:48.505 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hmxw+5jGIDZX6U1JOlPYjzBXRKQ= 2024-03-04T17:53:48.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hmxw+5jGIDZX6U1JOlPYjzBXRKQ=', 'Content-Length': '0'} 2024-03-04T17:53:48.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '160IY3UeXb5qH4Rh4JB09PNigxsTa5g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6cb8410db6c5d23-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.509 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.509 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.509 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.509 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.509 DEBUG:boto:Method: GET 2024-03-04T17:53:48.509 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.510 DEBUG:boto:Data: 2024-03-04T17:53:48.510 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.510 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.510 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.510 DEBUG:boto:Params: {} 2024-03-04T17:53:48.510 DEBUG:boto:Token: None 2024-03-04T17:53:48.510 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.510 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8cd38d14f6819bf-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.515 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.515 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.516 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.516 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.516 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.516 DEBUG:boto:Method: GET 2024-03-04T17:53:48.516 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.516 DEBUG:boto:Data: 2024-03-04T17:53:48.516 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.516 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.516 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.516 DEBUG:boto:Params: {} 2024-03-04T17:53:48.517 DEBUG:boto:Token: None 2024-03-04T17:53:48.517 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.517 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000377f413ea92cd56f-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.564 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.564 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.564 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.564 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.565 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:48.565 DEBUG:boto:Data: 2024-03-04T17:53:48.565 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.565 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.565 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.565 DEBUG:boto:Params: {} 2024-03-04T17:53:48.565 DEBUG:boto:Token: None 2024-03-04T17:53:48.565 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj 2024-03-04T17:53:48.565 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM= 2024-03-04T17:53:48.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM=', 'Content-Length': '0'} 2024-03-04T17:53:48.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f47b9daded6f6b35-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.572 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.572 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.572 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.572 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:48.572 DEBUG:boto:Data: 2024-03-04T17:53:48.573 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.573 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.573 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.573 DEBUG:boto:Params: {} 2024-03-04T17:53:48.573 DEBUG:boto:Token: None 2024-03-04T17:53:48.573 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj 2024-03-04T17:53:48.573 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM= 2024-03-04T17:53:48.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM=', 'Content-Length': '0'} 2024-03-04T17:53:48.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c43fce546109fa11-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:48.580 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.580 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.580 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.580 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.580 DEBUG:boto:Method: GET 2024-03-04T17:53:48.580 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.580 DEBUG:boto:Data: 2024-03-04T17:53:48.580 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.581 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.581 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.581 DEBUG:boto:Params: {} 2024-03-04T17:53:48.581 DEBUG:boto:Token: None 2024-03-04T17:53:48.581 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.581 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo= 2024-03-04T17:53:48.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo=', 'Content-Length': '0'} 2024-03-04T17:53:48.584 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eacc38925fff14d3-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.584 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.584 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.584 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.585 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.585 DEBUG:boto:Method: GET 2024-03-04T17:53:48.585 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.585 DEBUG:boto:Data: 2024-03-04T17:53:48.585 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.585 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.585 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.585 DEBUG:boto:Params: {} 2024-03-04T17:53:48.585 DEBUG:boto:Token: None 2024-03-04T17:53:48.586 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.586 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo= 2024-03-04T17:53:48.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo=', 'Content-Length': '0'} 2024-03-04T17:53:48.589 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073f4764a1d33ce92-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.632 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.632 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.632 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.632 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.632 DEBUG:boto:Method: GET 2024-03-04T17:53:48.633 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.633 DEBUG:boto:Data: 2024-03-04T17:53:48.633 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.633 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.633 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.633 DEBUG:boto:Params: {} 2024-03-04T17:53:48.633 DEBUG:boto:Token: None 2024-03-04T17:53:48.633 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.633 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065c2aded89487fc6-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.639 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.639 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.639 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.639 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.639 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.639 DEBUG:boto:Method: GET 2024-03-04T17:53:48.639 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.640 DEBUG:boto:Data: 2024-03-04T17:53:48.640 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.640 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.640 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.640 DEBUG:boto:Params: {} 2024-03-04T17:53:48.640 DEBUG:boto:Token: None 2024-03-04T17:53:48.640 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.640 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c811f29784570400-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.692 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:48.692 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.693 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.693 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.693 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.693 DEBUG:boto:Method: GET 2024-03-04T17:53:48.693 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.693 DEBUG:boto:Data: 2024-03-04T17:53:48.693 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.693 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.693 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.693 DEBUG:boto:Params: {} 2024-03-04T17:53:48.694 DEBUG:boto:Token: None 2024-03-04T17:53:48.694 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.694 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avpkPtfRnX1fY9FxdsLH4NbPZp8= 2024-03-04T17:53:48.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avpkPtfRnX1fY9FxdsLH4NbPZp8=', 'Content-Length': '0'} 2024-03-04T17:53:48.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054fa59822c2d04fc-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.740 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.740 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.740 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.740 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.740 DEBUG:boto:Method: GET 2024-03-04T17:53:48.740 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.741 DEBUG:boto:Data: 2024-03-04T17:53:48.741 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.741 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.741 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.741 DEBUG:boto:Params: {} 2024-03-04T17:53:48.741 DEBUG:boto:Token: None 2024-03-04T17:53:48.741 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.741 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avpkPtfRnX1fY9FxdsLH4NbPZp8= 2024-03-04T17:53:48.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avpkPtfRnX1fY9FxdsLH4NbPZp8=', 'Content-Length': '0'} 2024-03-04T17:53:48.745 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002177ea5a8996b349-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.745 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.745 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.745 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.746 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.746 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.746 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.746 DEBUG:boto:Data: 2024-03-04T17:53:48.746 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.746 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.746 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.746 DEBUG:boto:Params: {} 2024-03-04T17:53:48.746 DEBUG:boto:Token: None 2024-03-04T17:53:48.747 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.747 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IHJevP87/PXURV99Z/oXTh+HAg4= 2024-03-04T17:53:48.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IHJevP87/PXURV99Z/oXTh+HAg4=', 'Content-Length': '0'} 2024-03-04T17:53:48.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000757d08ec71b75a9e-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.750 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.750 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.750 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.750 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.751 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.751 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.751 DEBUG:boto:Data: 2024-03-04T17:53:48.751 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.751 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.751 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.751 DEBUG:boto:Params: {} 2024-03-04T17:53:48.751 DEBUG:boto:Token: None 2024-03-04T17:53:48.751 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.751 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IHJevP87/PXURV99Z/oXTh+HAg4= 2024-03-04T17:53:48.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IHJevP87/PXURV99Z/oXTh+HAg4=', 'Content-Length': '0'} 2024-03-04T17:53:48.754 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000afebc001fd694837-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:48.755 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.755 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.755 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.755 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.755 DEBUG:boto:Method: GET 2024-03-04T17:53:48.755 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.755 DEBUG:boto:Data: 2024-03-04T17:53:48.755 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.755 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.755 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.756 DEBUG:boto:Params: {} 2024-03-04T17:53:48.756 DEBUG:boto:Token: None 2024-03-04T17:53:48.756 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.756 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avpkPtfRnX1fY9FxdsLH4NbPZp8= 2024-03-04T17:53:48.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avpkPtfRnX1fY9FxdsLH4NbPZp8=', 'Content-Length': '0'} 2024-03-04T17:53:48.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045309cd7e050ea5b-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.800 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.800 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.800 DEBUG:boto:path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.800 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.800 DEBUG:boto:Method: GET 2024-03-04T17:53:48.801 DEBUG:boto:Path: /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.801 DEBUG:boto:Data: 2024-03-04T17:53:48.801 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.801 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.801 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.801 DEBUG:boto:Params: {} 2024-03-04T17:53:48.801 DEBUG:boto:Token: None 2024-03-04T17:53:48.801 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=9WE9mNbplYdU449Vkwh8lwrICYxU6v7 2024-03-04T17:53:48.801 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avpkPtfRnX1fY9FxdsLH4NbPZp8= 2024-03-04T17:53:48.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avpkPtfRnX1fY9FxdsLH4NbPZp8=', 'Content-Length': '0'} 2024-03-04T17:53:48.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '9WE9mNbplYdU449Vkwh8lwrICYxU6v7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f83d5b59279512e-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.848 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.848 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.848 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.848 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.848 DEBUG:boto:Method: GET 2024-03-04T17:53:48.848 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.849 DEBUG:boto:Data: 2024-03-04T17:53:48.849 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.849 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.849 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.849 DEBUG:boto:Params: {} 2024-03-04T17:53:48.849 DEBUG:boto:Token: None 2024-03-04T17:53:48.849 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.849 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1a6ea802441fc00-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.854 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.855 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.855 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.855 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.855 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.855 DEBUG:boto:Method: GET 2024-03-04T17:53:48.855 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.855 DEBUG:boto:Data: 2024-03-04T17:53:48.855 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.855 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.855 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.856 DEBUG:boto:Params: {} 2024-03-04T17:53:48.856 DEBUG:boto:Token: None 2024-03-04T17:53:48.856 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.856 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000318e2a5a92c543c9-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.861 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.862 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.862 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.862 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.862 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:48.863 DEBUG:boto:Data: 2024-03-04T17:53:48.863 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.863 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.863 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.863 DEBUG:boto:Params: {} 2024-03-04T17:53:48.864 DEBUG:boto:Token: None 2024-03-04T17:53:48.864 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj 2024-03-04T17:53:48.864 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM= 2024-03-04T17:53:48.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM=', 'Content-Length': '0'} 2024-03-04T17:53:48.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c38689a45d560e73-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.870 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.870 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.870 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.871 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:48.871 DEBUG:boto:Data: 2024-03-04T17:53:48.871 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.871 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.871 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.871 DEBUG:boto:Params: {} 2024-03-04T17:53:48.871 DEBUG:boto:Token: None 2024-03-04T17:53:48.871 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj 2024-03-04T17:53:48.871 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM= 2024-03-04T17:53:48.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nc1mYO0HWnyUUEr86pdix/CMscM=', 'Content-Length': '0'} 2024-03-04T17:53:48.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008353506a983e1088-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:48.878 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.878 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.878 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.878 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.878 DEBUG:boto:Method: GET 2024-03-04T17:53:48.878 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.878 DEBUG:boto:Data: 2024-03-04T17:53:48.878 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.878 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.878 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.879 DEBUG:boto:Params: {} 2024-03-04T17:53:48.879 DEBUG:boto:Token: None 2024-03-04T17:53:48.879 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.879 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo= 2024-03-04T17:53:48.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo=', 'Content-Length': '0'} 2024-03-04T17:53:48.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffba5d1ec989a068-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.882 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.882 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.883 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.883 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.883 DEBUG:boto:Method: GET 2024-03-04T17:53:48.883 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.883 DEBUG:boto:Data: 2024-03-04T17:53:48.883 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.883 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.883 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.883 DEBUG:boto:Params: {} 2024-03-04T17:53:48.883 DEBUG:boto:Token: None 2024-03-04T17:53:48.884 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:48.884 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo= 2024-03-04T17:53:48.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y+yc5/BjGvf6cE/ZPziogu8QZvo=', 'Content-Length': '0'} 2024-03-04T17:53:48.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bb38ac241e9bcea-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.928 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.928 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.928 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.928 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.929 DEBUG:boto:Method: GET 2024-03-04T17:53:48.929 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.929 DEBUG:boto:Data: 2024-03-04T17:53:48.929 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.929 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.929 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.929 DEBUG:boto:Params: {} 2024-03-04T17:53:48.929 DEBUG:boto:Token: None 2024-03-04T17:53:48.929 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.929 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1429899315a206b-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.935 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.935 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:48.935 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:48.935 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.935 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:48.935 DEBUG:boto:Method: GET 2024-03-04T17:53:48.936 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:48.936 DEBUG:boto:Data: 2024-03-04T17:53:48.936 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.936 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.936 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.936 DEBUG:boto:Params: {} 2024-03-04T17:53:48.936 DEBUG:boto:Token: None 2024-03-04T17:53:48.936 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/ 2024-03-04T17:53:48.936 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo= 2024-03-04T17:53:48.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Ogpxaqz2iD2MSTFOcu3/aR5Sqo=', 'Content-Length': '0'} 2024-03-04T17:53:48.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df831820b141524b-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.941 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:48.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:48.941 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.941 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.942 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.942 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.942 DEBUG:boto:Method: GET 2024-03-04T17:53:48.942 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.942 DEBUG:boto:Data: 2024-03-04T17:53:48.942 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.942 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.942 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.942 DEBUG:boto:Params: {} 2024-03-04T17:53:48.942 DEBUG:boto:Token: None 2024-03-04T17:53:48.942 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.943 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eEwinkqjekLuMcp13LBQOsZdj7M= 2024-03-04T17:53:48.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eEwinkqjekLuMcp13LBQOsZdj7M=', 'Content-Length': '0'} 2024-03-04T17:53:48.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cbaa78251ad98ef-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.988 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.988 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.988 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.988 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.988 DEBUG:boto:Method: GET 2024-03-04T17:53:48.989 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.989 DEBUG:boto:Data: 2024-03-04T17:53:48.989 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.989 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.989 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.989 DEBUG:boto:Params: {} 2024-03-04T17:53:48.989 DEBUG:boto:Token: None 2024-03-04T17:53:48.989 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.989 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eEwinkqjekLuMcp13LBQOsZdj7M= 2024-03-04T17:53:48.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eEwinkqjekLuMcp13LBQOsZdj7M=', 'Content-Length': '0'} 2024-03-04T17:53:48.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1b6b8f3df791602-0065e60aac-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.993 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.993 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.993 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.993 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.993 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.993 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.993 DEBUG:boto:Data: 2024-03-04T17:53:48.993 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.993 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:48.993 DEBUG:boto:Port: 8001 2024-03-04T17:53:48.993 DEBUG:boto:Params: {} 2024-03-04T17:53:48.994 DEBUG:boto:Token: None 2024-03-04T17:53:48.994 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.994 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tNIuE8ht2tDTkkzDl/zrpb7f5Cs= 2024-03-04T17:53:48.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tNIuE8ht2tDTkkzDl/zrpb7f5Cs=', 'Content-Length': '0'} 2024-03-04T17:53:48.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2a41e249ee0287a-0065e60aac-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:48.997 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:48.997 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:48.997 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.998 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.998 DEBUG:boto:Method: HEAD 2024-03-04T17:53:48.998 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.998 DEBUG:boto:Data: 2024-03-04T17:53:48.998 DEBUG:boto:Headers: {} 2024-03-04T17:53:48.998 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:48.998 DEBUG:boto:Port: 8000 2024-03-04T17:53:48.998 DEBUG:boto:Params: {} 2024-03-04T17:53:48.998 DEBUG:boto:Token: None 2024-03-04T17:53:48.998 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:48 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:48.998 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tNIuE8ht2tDTkkzDl/zrpb7f5Cs= 2024-03-04T17:53:48.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tNIuE8ht2tDTkkzDl/zrpb7f5Cs=', 'Content-Length': '0'} 2024-03-04T17:53:49.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000283597690bf92be4-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:49.002 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.002 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.002 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:49.002 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:49.002 DEBUG:boto:Method: GET 2024-03-04T17:53:49.002 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:49.002 DEBUG:boto:Data: 2024-03-04T17:53:49.002 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.002 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.002 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.002 DEBUG:boto:Params: {} 2024-03-04T17:53:49.003 DEBUG:boto:Token: None 2024-03-04T17:53:49.003 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:49.003 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S2o+nWOiNi0Yxq7n7N/GrQy42mg= 2024-03-04T17:53:49.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S2o+nWOiNi0Yxq7n7N/GrQy42mg=', 'Content-Length': '0'} 2024-03-04T17:53:49.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bb27af2768251b7-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.006 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.006 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.006 DEBUG:boto:path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:49.006 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:49.007 DEBUG:boto:Method: GET 2024-03-04T17:53:49.007 DEBUG:boto:Path: /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:49.007 DEBUG:boto:Data: 2024-03-04T17:53:49.007 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.007 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.007 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.007 DEBUG:boto:Params: {} 2024-03-04T17:53:49.007 DEBUG:boto:Token: None 2024-03-04T17:53:49.007 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg 2024-03-04T17:53:49.008 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:S2o+nWOiNi0Yxq7n7N/GrQy42mg= 2024-03-04T17:53:49.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S2o+nWOiNi0Yxq7n7N/GrQy42mg=', 'Content-Length': '0'} 2024-03-04T17:53:49.010 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'ZJZPq8HwR1LmdjGaT37eTcVO7gvndKg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4de12986bb936e3-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.052 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.052 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.052 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.052 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.052 DEBUG:boto:Method: GET 2024-03-04T17:53:49.052 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.053 DEBUG:boto:Data: 2024-03-04T17:53:49.053 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.053 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.053 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.053 DEBUG:boto:Params: {} 2024-03-04T17:53:49.053 DEBUG:boto:Token: None 2024-03-04T17:53:49.053 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.053 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000869f7ef0881218a6-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.058 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.059 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.059 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.059 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.059 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.059 DEBUG:boto:Method: GET 2024-03-04T17:53:49.059 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.059 DEBUG:boto:Data: 2024-03-04T17:53:49.059 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.059 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.060 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.060 DEBUG:boto:Params: {} 2024-03-04T17:53:49.060 DEBUG:boto:Token: None 2024-03-04T17:53:49.060 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.060 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cf2284d089fcf11-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.065 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.065 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.065 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.065 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.065 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:49.065 DEBUG:boto:Data: 2024-03-04T17:53:49.065 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.065 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.066 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.066 DEBUG:boto:Params: {} 2024-03-04T17:53:49.066 DEBUG:boto:Token: None 2024-03-04T17:53:49.066 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj 2024-03-04T17:53:49.066 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ= 2024-03-04T17:53:49.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ=', 'Content-Length': '0'} 2024-03-04T17:53:49.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008ef75839e5a8d53-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.072 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.072 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.072 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.072 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:49.073 DEBUG:boto:Data: 2024-03-04T17:53:49.073 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.073 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.073 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.073 DEBUG:boto:Params: {} 2024-03-04T17:53:49.073 DEBUG:boto:Token: None 2024-03-04T17:53:49.073 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj 2024-03-04T17:53:49.073 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ= 2024-03-04T17:53:49.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ=', 'Content-Length': '0'} 2024-03-04T17:53:49.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c262f9b0d664d665-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:49.080 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.080 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.080 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.080 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.080 DEBUG:boto:Method: GET 2024-03-04T17:53:49.081 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.081 DEBUG:boto:Data: 2024-03-04T17:53:49.081 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.081 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.081 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.081 DEBUG:boto:Params: {} 2024-03-04T17:53:49.081 DEBUG:boto:Token: None 2024-03-04T17:53:49.081 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.081 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54= 2024-03-04T17:53:49.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54=', 'Content-Length': '0'} 2024-03-04T17:53:49.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bc0a9df1ca0e469-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.128 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.128 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.128 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.128 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.128 DEBUG:boto:Method: GET 2024-03-04T17:53:49.129 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.129 DEBUG:boto:Data: 2024-03-04T17:53:49.129 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.129 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.129 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.129 DEBUG:boto:Params: {} 2024-03-04T17:53:49.129 DEBUG:boto:Token: None 2024-03-04T17:53:49.129 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.129 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54= 2024-03-04T17:53:49.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54=', 'Content-Length': '0'} 2024-03-04T17:53:49.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c31df7293c26c57a-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.180 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.180 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.180 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.181 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.181 DEBUG:boto:Method: GET 2024-03-04T17:53:49.181 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.181 DEBUG:boto:Data: 2024-03-04T17:53:49.181 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.181 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.181 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.181 DEBUG:boto:Params: {} 2024-03-04T17:53:49.182 DEBUG:boto:Token: None 2024-03-04T17:53:49.182 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.182 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ed2aa1ef5f3cdb2-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.187 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.187 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.187 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.187 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.188 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.188 DEBUG:boto:Method: GET 2024-03-04T17:53:49.188 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.188 DEBUG:boto:Data: 2024-03-04T17:53:49.188 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.188 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.188 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.188 DEBUG:boto:Params: {} 2024-03-04T17:53:49.191 DEBUG:boto:Token: None 2024-03-04T17:53:49.191 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.191 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aac522153d6f556c-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.197 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:49.197 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.197 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.197 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.197 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.197 DEBUG:boto:Method: GET 2024-03-04T17:53:49.198 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.198 DEBUG:boto:Data: 2024-03-04T17:53:49.198 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.198 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.198 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.198 DEBUG:boto:Params: {} 2024-03-04T17:53:49.198 DEBUG:boto:Token: None 2024-03-04T17:53:49.198 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.199 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oNXwNVPd3GiE429ezz4rOPl8seA= 2024-03-04T17:53:49.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oNXwNVPd3GiE429ezz4rOPl8seA=', 'Content-Length': '0'} 2024-03-04T17:53:49.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006964723e5d80b5ea-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.244 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.244 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.244 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.244 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.244 DEBUG:boto:Method: GET 2024-03-04T17:53:49.245 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.245 DEBUG:boto:Data: 2024-03-04T17:53:49.245 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.245 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.245 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.245 DEBUG:boto:Params: {} 2024-03-04T17:53:49.245 DEBUG:boto:Token: None 2024-03-04T17:53:49.245 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.245 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oNXwNVPd3GiE429ezz4rOPl8seA= 2024-03-04T17:53:49.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oNXwNVPd3GiE429ezz4rOPl8seA=', 'Content-Length': '0'} 2024-03-04T17:53:49.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d2ad6c45a4901de-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.249 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.249 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.249 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.249 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.250 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.250 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.250 DEBUG:boto:Data: 2024-03-04T17:53:49.250 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.250 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.250 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.250 DEBUG:boto:Params: {} 2024-03-04T17:53:49.250 DEBUG:boto:Token: None 2024-03-04T17:53:49.250 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.251 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:d2z7lCe3lVZCNc5jZ60hAiLr4+c= 2024-03-04T17:53:49.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:d2z7lCe3lVZCNc5jZ60hAiLr4+c=', 'Content-Length': '0'} 2024-03-04T17:53:49.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c673b08d90cccd73-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.254 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.254 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.254 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.254 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.254 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.255 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.255 DEBUG:boto:Data: 2024-03-04T17:53:49.255 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.255 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.255 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.255 DEBUG:boto:Params: {} 2024-03-04T17:53:49.255 DEBUG:boto:Token: None 2024-03-04T17:53:49.255 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.255 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:d2z7lCe3lVZCNc5jZ60hAiLr4+c= 2024-03-04T17:53:49.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:d2z7lCe3lVZCNc5jZ60hAiLr4+c=', 'Content-Length': '0'} 2024-03-04T17:53:49.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc33ea4237e53115-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:49.259 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.259 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.259 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.260 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.260 DEBUG:boto:Method: GET 2024-03-04T17:53:49.260 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.260 DEBUG:boto:Data: 2024-03-04T17:53:49.260 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.260 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.260 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.260 DEBUG:boto:Params: {} 2024-03-04T17:53:49.261 DEBUG:boto:Token: None 2024-03-04T17:53:49.261 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.261 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oNXwNVPd3GiE429ezz4rOPl8seA= 2024-03-04T17:53:49.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oNXwNVPd3GiE429ezz4rOPl8seA=', 'Content-Length': '0'} 2024-03-04T17:53:49.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f481ad2209183ec3-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.312 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.312 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.312 DEBUG:boto:path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.312 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.312 DEBUG:boto:Method: GET 2024-03-04T17:53:49.313 DEBUG:boto:Path: /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.313 DEBUG:boto:Data: 2024-03-04T17:53:49.313 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.313 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.313 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.313 DEBUG:boto:Params: {} 2024-03-04T17:53:49.313 DEBUG:boto:Token: None 2024-03-04T17:53:49.313 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa 2024-03-04T17:53:49.314 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oNXwNVPd3GiE429ezz4rOPl8seA= 2024-03-04T17:53:49.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oNXwNVPd3GiE429ezz4rOPl8seA=', 'Content-Length': '0'} 2024-03-04T17:53:49.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', '2lwBSkQlMAA54TFAsh7J7MCnzo0gTfa'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079347fd46698757e-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.364 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.364 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.364 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.364 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.365 DEBUG:boto:Method: GET 2024-03-04T17:53:49.365 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.365 DEBUG:boto:Data: 2024-03-04T17:53:49.365 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.365 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.365 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.365 DEBUG:boto:Params: {} 2024-03-04T17:53:49.365 DEBUG:boto:Token: None 2024-03-04T17:53:49.365 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.366 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041eb4a3c7501738d-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.370 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.371 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.371 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.371 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.371 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.371 DEBUG:boto:Method: GET 2024-03-04T17:53:49.371 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.371 DEBUG:boto:Data: 2024-03-04T17:53:49.371 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.372 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.372 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.372 DEBUG:boto:Params: {} 2024-03-04T17:53:49.372 DEBUG:boto:Token: None 2024-03-04T17:53:49.372 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.372 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f76c4106ea9530f3-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.424 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.424 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.424 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.425 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.425 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:49.425 DEBUG:boto:Data: 2024-03-04T17:53:49.425 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.425 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.425 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.425 DEBUG:boto:Params: {} 2024-03-04T17:53:49.425 DEBUG:boto:Token: None 2024-03-04T17:53:49.425 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj 2024-03-04T17:53:49.426 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ= 2024-03-04T17:53:49.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ=', 'Content-Length': '0'} 2024-03-04T17:53:49.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008550ff147bf38db7-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.432 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.432 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.432 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.432 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:49.432 DEBUG:boto:Data: 2024-03-04T17:53:49.432 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.432 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.432 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.432 DEBUG:boto:Params: {} 2024-03-04T17:53:49.433 DEBUG:boto:Token: None 2024-03-04T17:53:49.433 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj 2024-03-04T17:53:49.433 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ= 2024-03-04T17:53:49.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ=', 'Content-Length': '0'} 2024-03-04T17:53:49.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000509a79f0590eaa4b-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:49.440 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.440 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.441 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.441 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.441 DEBUG:boto:Method: GET 2024-03-04T17:53:49.441 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.441 DEBUG:boto:Data: 2024-03-04T17:53:49.441 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.441 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.441 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.441 DEBUG:boto:Params: {} 2024-03-04T17:53:49.441 DEBUG:boto:Token: None 2024-03-04T17:53:49.441 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.442 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54= 2024-03-04T17:53:49.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54=', 'Content-Length': '0'} 2024-03-04T17:53:49.445 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c41f3e4cadfc25d-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.445 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.445 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.445 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.445 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.445 DEBUG:boto:Method: GET 2024-03-04T17:53:49.445 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.446 DEBUG:boto:Data: 2024-03-04T17:53:49.446 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.446 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.446 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.446 DEBUG:boto:Params: {} 2024-03-04T17:53:49.446 DEBUG:boto:Token: None 2024-03-04T17:53:49.446 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.446 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54= 2024-03-04T17:53:49.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54=', 'Content-Length': '0'} 2024-03-04T17:53:49.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c69f28a35bd08456-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.492 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.492 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.492 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.492 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.492 DEBUG:boto:Method: GET 2024-03-04T17:53:49.493 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.493 DEBUG:boto:Data: 2024-03-04T17:53:49.493 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.493 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.493 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.493 DEBUG:boto:Params: {} 2024-03-04T17:53:49.493 DEBUG:boto:Token: None 2024-03-04T17:53:49.493 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.493 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a635ee40a48497d2-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.498 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.499 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.499 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.499 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.499 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.499 DEBUG:boto:Method: GET 2024-03-04T17:53:49.500 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.500 DEBUG:boto:Data: 2024-03-04T17:53:49.500 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.500 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.500 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.500 DEBUG:boto:Params: {} 2024-03-04T17:53:49.500 DEBUG:boto:Token: None 2024-03-04T17:53:49.501 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.501 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000132e32872f360558-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.548 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:49.549 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.549 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.549 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.549 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.549 DEBUG:boto:Method: GET 2024-03-04T17:53:49.549 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.549 DEBUG:boto:Data: 2024-03-04T17:53:49.549 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.549 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.549 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.550 DEBUG:boto:Params: {} 2024-03-04T17:53:49.550 DEBUG:boto:Token: None 2024-03-04T17:53:49.550 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.550 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vsJLddI0I8id2qUqMOTVFzyy4i4= 2024-03-04T17:53:49.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vsJLddI0I8id2qUqMOTVFzyy4i4=', 'Content-Length': '0'} 2024-03-04T17:53:49.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7c6ebaeca1e2bd1-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.553 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.553 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.553 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.553 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.553 DEBUG:boto:Method: GET 2024-03-04T17:53:49.554 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.554 DEBUG:boto:Data: 2024-03-04T17:53:49.554 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.554 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.554 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.554 DEBUG:boto:Params: {} 2024-03-04T17:53:49.554 DEBUG:boto:Token: None 2024-03-04T17:53:49.554 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.554 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vsJLddI0I8id2qUqMOTVFzyy4i4= 2024-03-04T17:53:49.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vsJLddI0I8id2qUqMOTVFzyy4i4=', 'Content-Length': '0'} 2024-03-04T17:53:49.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099a386a4972d822f-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.558 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.558 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.558 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.558 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.558 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.558 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.558 DEBUG:boto:Data: 2024-03-04T17:53:49.558 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.558 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.558 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.559 DEBUG:boto:Params: {} 2024-03-04T17:53:49.559 DEBUG:boto:Token: None 2024-03-04T17:53:49.559 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.559 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v/PqtbV54pnmruN0w9DwiDfN0so= 2024-03-04T17:53:49.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v/PqtbV54pnmruN0w9DwiDfN0so=', 'Content-Length': '0'} 2024-03-04T17:53:49.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000affc066e56272925-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.562 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.562 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.562 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.562 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.562 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.562 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.563 DEBUG:boto:Data: 2024-03-04T17:53:49.563 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.563 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.563 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.563 DEBUG:boto:Params: {} 2024-03-04T17:53:49.563 DEBUG:boto:Token: None 2024-03-04T17:53:49.563 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.563 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v/PqtbV54pnmruN0w9DwiDfN0so= 2024-03-04T17:53:49.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v/PqtbV54pnmruN0w9DwiDfN0so=', 'Content-Length': '0'} 2024-03-04T17:53:49.566 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a631d71c0b7e39c1-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:49.566 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.566 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.567 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.567 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.567 DEBUG:boto:Method: GET 2024-03-04T17:53:49.567 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.567 DEBUG:boto:Data: 2024-03-04T17:53:49.567 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.567 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.567 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.567 DEBUG:boto:Params: {} 2024-03-04T17:53:49.567 DEBUG:boto:Token: None 2024-03-04T17:53:49.567 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.568 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vsJLddI0I8id2qUqMOTVFzyy4i4= 2024-03-04T17:53:49.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vsJLddI0I8id2qUqMOTVFzyy4i4=', 'Content-Length': '0'} 2024-03-04T17:53:49.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000325992f4e41890c3-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.612 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.612 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.612 DEBUG:boto:path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.612 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.612 DEBUG:boto:Method: GET 2024-03-04T17:53:49.613 DEBUG:boto:Path: /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.613 DEBUG:boto:Data: 2024-03-04T17:53:49.613 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.613 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.613 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.613 DEBUG:boto:Params: {} 2024-03-04T17:53:49.613 DEBUG:boto:Token: None 2024-03-04T17:53:49.613 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE 2024-03-04T17:53:49.613 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vsJLddI0I8id2qUqMOTVFzyy4i4= 2024-03-04T17:53:49.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vsJLddI0I8id2qUqMOTVFzyy4i4=', 'Content-Length': '0'} 2024-03-04T17:53:49.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'q4h-uWPuCpU7FK9SfN-LHLv-vtuKwuE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb963ccbba8d16a3-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.664 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.664 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.664 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.664 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.664 DEBUG:boto:Method: GET 2024-03-04T17:53:49.665 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.665 DEBUG:boto:Data: 2024-03-04T17:53:49.665 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.665 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.665 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.665 DEBUG:boto:Params: {} 2024-03-04T17:53:49.665 DEBUG:boto:Token: None 2024-03-04T17:53:49.665 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.665 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006863d0ee40c5e21d-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.671 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.671 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.671 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.671 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.671 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.672 DEBUG:boto:Method: GET 2024-03-04T17:53:49.672 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.672 DEBUG:boto:Data: 2024-03-04T17:53:49.672 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.672 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.672 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.672 DEBUG:boto:Params: {} 2024-03-04T17:53:49.672 DEBUG:boto:Token: None 2024-03-04T17:53:49.672 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.672 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000158d110f6a8234bc-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.724 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.724 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.725 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.725 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.725 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:49.725 DEBUG:boto:Data: 2024-03-04T17:53:49.725 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.725 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.725 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.725 DEBUG:boto:Params: {} 2024-03-04T17:53:49.725 DEBUG:boto:Token: None 2024-03-04T17:53:49.726 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj 2024-03-04T17:53:49.726 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ= 2024-03-04T17:53:49.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ=', 'Content-Length': '0'} 2024-03-04T17:53:49.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7b73262b9bfb131-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.731 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.731 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.732 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.732 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:49.732 DEBUG:boto:Data: 2024-03-04T17:53:49.732 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.732 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.732 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.732 DEBUG:boto:Params: {} 2024-03-04T17:53:49.732 DEBUG:boto:Token: None 2024-03-04T17:53:49.732 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj 2024-03-04T17:53:49.732 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ= 2024-03-04T17:53:49.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ=', 'Content-Length': '0'} 2024-03-04T17:53:49.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b37bc0a6358254a-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:49.739 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.739 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.739 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.739 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.739 DEBUG:boto:Method: GET 2024-03-04T17:53:49.739 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.739 DEBUG:boto:Data: 2024-03-04T17:53:49.739 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.739 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.739 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.740 DEBUG:boto:Params: {} 2024-03-04T17:53:49.740 DEBUG:boto:Token: None 2024-03-04T17:53:49.740 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.740 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54= 2024-03-04T17:53:49.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54=', 'Content-Length': '0'} 2024-03-04T17:53:49.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092b87ea55225ae61-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.743 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.743 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.743 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.744 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.744 DEBUG:boto:Method: GET 2024-03-04T17:53:49.744 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.744 DEBUG:boto:Data: 2024-03-04T17:53:49.744 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.744 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.744 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.744 DEBUG:boto:Params: {} 2024-03-04T17:53:49.744 DEBUG:boto:Token: None 2024-03-04T17:53:49.744 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.745 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54= 2024-03-04T17:53:49.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54=', 'Content-Length': '0'} 2024-03-04T17:53:49.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000acc3f3fb3680b5f4-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.788 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.788 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.788 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.788 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.788 DEBUG:boto:Method: GET 2024-03-04T17:53:49.789 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.789 DEBUG:boto:Data: 2024-03-04T17:53:49.789 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.789 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.789 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.789 DEBUG:boto:Params: {} 2024-03-04T17:53:49.789 DEBUG:boto:Token: None 2024-03-04T17:53:49.789 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.789 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfadd3a51b905444-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.795 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.795 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.795 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.796 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.796 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.796 DEBUG:boto:Method: GET 2024-03-04T17:53:49.796 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.796 DEBUG:boto:Data: 2024-03-04T17:53:49.796 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.796 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.796 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.796 DEBUG:boto:Params: {} 2024-03-04T17:53:49.796 DEBUG:boto:Token: None 2024-03-04T17:53:49.796 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.797 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ddcfc7d5715e71e-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.848 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:49.848 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.849 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.849 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.849 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.849 DEBUG:boto:Method: GET 2024-03-04T17:53:49.849 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.849 DEBUG:boto:Data: 2024-03-04T17:53:49.849 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.849 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.849 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.849 DEBUG:boto:Params: {} 2024-03-04T17:53:49.850 DEBUG:boto:Token: None 2024-03-04T17:53:49.850 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.850 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pXrcM7C6Za1n5TEfcFJoWqFyF1I= 2024-03-04T17:53:49.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pXrcM7C6Za1n5TEfcFJoWqFyF1I=', 'Content-Length': '0'} 2024-03-04T17:53:49.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000891c0fec7e44f80b-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.900 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.900 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.900 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.900 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.901 DEBUG:boto:Method: GET 2024-03-04T17:53:49.901 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.901 DEBUG:boto:Data: 2024-03-04T17:53:49.901 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.901 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.901 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.901 DEBUG:boto:Params: {} 2024-03-04T17:53:49.901 DEBUG:boto:Token: None 2024-03-04T17:53:49.901 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.901 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pXrcM7C6Za1n5TEfcFJoWqFyF1I= 2024-03-04T17:53:49.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pXrcM7C6Za1n5TEfcFJoWqFyF1I=', 'Content-Length': '0'} 2024-03-04T17:53:49.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000afdeef42eae333f4-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.905 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.905 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.905 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.906 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.906 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.906 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.906 DEBUG:boto:Data: 2024-03-04T17:53:49.906 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.906 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.906 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.906 DEBUG:boto:Params: {} 2024-03-04T17:53:49.906 DEBUG:boto:Token: None 2024-03-04T17:53:49.906 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.907 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:N9VHWnoXubhqyAA5tjprCGveyjM= 2024-03-04T17:53:49.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:N9VHWnoXubhqyAA5tjprCGveyjM=', 'Content-Length': '0'} 2024-03-04T17:53:49.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bf2e317fe6ca3e8-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.910 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.910 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.910 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.910 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.910 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.910 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.910 DEBUG:boto:Data: 2024-03-04T17:53:49.911 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.911 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.911 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.911 DEBUG:boto:Params: {} 2024-03-04T17:53:49.911 DEBUG:boto:Token: None 2024-03-04T17:53:49.911 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.911 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:N9VHWnoXubhqyAA5tjprCGveyjM= 2024-03-04T17:53:49.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:N9VHWnoXubhqyAA5tjprCGveyjM=', 'Content-Length': '0'} 2024-03-04T17:53:49.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000937037b80af5a5f6-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:49.915 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.915 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.915 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.915 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.915 DEBUG:boto:Method: GET 2024-03-04T17:53:49.915 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.915 DEBUG:boto:Data: 2024-03-04T17:53:49.915 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.915 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.916 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.916 DEBUG:boto:Params: {} 2024-03-04T17:53:49.916 DEBUG:boto:Token: None 2024-03-04T17:53:49.916 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.916 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pXrcM7C6Za1n5TEfcFJoWqFyF1I= 2024-03-04T17:53:49.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pXrcM7C6Za1n5TEfcFJoWqFyF1I=', 'Content-Length': '0'} 2024-03-04T17:53:49.919 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058ddf90346982743-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.964 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.964 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.964 DEBUG:boto:path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.964 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.965 DEBUG:boto:Method: GET 2024-03-04T17:53:49.965 DEBUG:boto:Path: /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.965 DEBUG:boto:Data: 2024-03-04T17:53:49.965 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.965 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.965 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.965 DEBUG:boto:Params: {} 2024-03-04T17:53:49.965 DEBUG:boto:Token: None 2024-03-04T17:53:49.965 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=zKZVhMV59rWXQPf05XlhVIimSaZTr40 2024-03-04T17:53:49.965 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pXrcM7C6Za1n5TEfcFJoWqFyF1I= 2024-03-04T17:53:49.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pXrcM7C6Za1n5TEfcFJoWqFyF1I=', 'Content-Length': '0'} 2024-03-04T17:53:49.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'zKZVhMV59rWXQPf05XlhVIimSaZTr40'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0f2c80a828e88f0-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.969 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.969 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.969 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.969 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.970 DEBUG:boto:Method: GET 2024-03-04T17:53:49.970 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.970 DEBUG:boto:Data: 2024-03-04T17:53:49.970 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.970 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.970 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.970 DEBUG:boto:Params: {} 2024-03-04T17:53:49.970 DEBUG:boto:Token: None 2024-03-04T17:53:49.970 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.971 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052cd7f01176594d1-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.975 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.976 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:49.976 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:49.976 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.976 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:49.976 DEBUG:boto:Method: GET 2024-03-04T17:53:49.976 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:49.976 DEBUG:boto:Data: 2024-03-04T17:53:49.976 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.977 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.977 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.977 DEBUG:boto:Params: {} 2024-03-04T17:53:49.977 DEBUG:boto:Token: None 2024-03-04T17:53:49.977 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/ 2024-03-04T17:53:49.977 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc= 2024-03-04T17:53:49.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:68ka/1vHVAbK9ehvrDGlKhxHqZc=', 'Content-Length': '0'} 2024-03-04T17:53:49.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091f2e5bf7bc9c91a-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.982 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:49.982 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.982 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.983 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.983 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:49.983 DEBUG:boto:Data: 2024-03-04T17:53:49.983 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.983 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.983 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.983 DEBUG:boto:Params: {} 2024-03-04T17:53:49.983 DEBUG:boto:Token: None 2024-03-04T17:53:49.983 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj 2024-03-04T17:53:49.983 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ= 2024-03-04T17:53:49.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ=', 'Content-Length': '0'} 2024-03-04T17:53:49.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029023b97ff24af4c-0065e60aad-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.989 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.989 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.990 DEBUG:boto:Method: HEAD 2024-03-04T17:53:49.990 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:49.990 DEBUG:boto:Data: 2024-03-04T17:53:49.990 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.990 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:49.990 DEBUG:boto:Port: 8000 2024-03-04T17:53:49.990 DEBUG:boto:Params: {} 2024-03-04T17:53:49.990 DEBUG:boto:Token: None 2024-03-04T17:53:49.990 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj 2024-03-04T17:53:49.990 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ= 2024-03-04T17:53:49.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:KYqtJlmFDyhFA4pfCtxvo+y+5JQ=', 'Content-Length': '0'} 2024-03-04T17:53:49.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d60db0f761873a18-0065e60aad-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:49.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:49.996 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:49.996 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:49.997 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.997 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.997 DEBUG:boto:Method: GET 2024-03-04T17:53:49.997 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.997 DEBUG:boto:Data: 2024-03-04T17:53:49.997 DEBUG:boto:Headers: {} 2024-03-04T17:53:49.997 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:49.997 DEBUG:boto:Port: 8001 2024-03-04T17:53:49.997 DEBUG:boto:Params: {} 2024-03-04T17:53:49.997 DEBUG:boto:Token: None 2024-03-04T17:53:49.998 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:49 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:49.998 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54= 2024-03-04T17:53:49.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:49 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CkmK22r5km+hMZ3dpj6/6c7eR54=', 'Content-Length': '0'} 2024-03-04T17:53:50.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031d67e2e2a92b9b6-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.044 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.044 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.044 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.044 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.045 DEBUG:boto:Method: GET 2024-03-04T17:53:50.045 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.045 DEBUG:boto:Data: 2024-03-04T17:53:50.045 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.045 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.045 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.045 DEBUG:boto:Params: {} 2024-03-04T17:53:50.045 DEBUG:boto:Token: None 2024-03-04T17:53:50.045 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.046 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M= 2024-03-04T17:53:50.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M=', 'Content-Length': '0'} 2024-03-04T17:53:50.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000178821caf6a4ff77-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.092 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.092 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.092 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.092 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.093 DEBUG:boto:Method: GET 2024-03-04T17:53:50.093 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.093 DEBUG:boto:Data: 2024-03-04T17:53:50.093 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.093 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.093 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.093 DEBUG:boto:Params: {} 2024-03-04T17:53:50.093 DEBUG:boto:Token: None 2024-03-04T17:53:50.093 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.094 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f62a1f90b658218d-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.098 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:50.099 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.099 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.099 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.099 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.099 DEBUG:boto:Method: GET 2024-03-04T17:53:50.099 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.099 DEBUG:boto:Data: 2024-03-04T17:53:50.099 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.099 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.099 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.099 DEBUG:boto:Params: {} 2024-03-04T17:53:50.100 DEBUG:boto:Token: None 2024-03-04T17:53:50.100 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.100 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027286aad00460878-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.105 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:50.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:50.105 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.105 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.105 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.106 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.106 DEBUG:boto:Method: GET 2024-03-04T17:53:50.106 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.106 DEBUG:boto:Data: 2024-03-04T17:53:50.106 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.106 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.106 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.106 DEBUG:boto:Params: {} 2024-03-04T17:53:50.106 DEBUG:boto:Token: None 2024-03-04T17:53:50.106 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.107 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yKMvs2CYd18EMrDRJN7Tcq/ci9U= 2024-03-04T17:53:50.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yKMvs2CYd18EMrDRJN7Tcq/ci9U=', 'Content-Length': '0'} 2024-03-04T17:53:50.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031c085b66613c095-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.156 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.156 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.156 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.156 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.157 DEBUG:boto:Method: GET 2024-03-04T17:53:50.157 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.157 DEBUG:boto:Data: 2024-03-04T17:53:50.157 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.157 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.157 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.157 DEBUG:boto:Params: {} 2024-03-04T17:53:50.157 DEBUG:boto:Token: None 2024-03-04T17:53:50.157 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.158 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yKMvs2CYd18EMrDRJN7Tcq/ci9U= 2024-03-04T17:53:50.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yKMvs2CYd18EMrDRJN7Tcq/ci9U=', 'Content-Length': '0'} 2024-03-04T17:53:50.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d19ca37e3ef46590-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.161 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.162 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.162 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.162 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.162 DEBUG:boto:Method: HEAD 2024-03-04T17:53:50.162 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.162 DEBUG:boto:Data: 2024-03-04T17:53:50.162 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.162 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.162 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.162 DEBUG:boto:Params: {} 2024-03-04T17:53:50.162 DEBUG:boto:Token: None 2024-03-04T17:53:50.163 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.163 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iXIOnyZpM918g9qzK2O4qf0GDWQ= 2024-03-04T17:53:50.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iXIOnyZpM918g9qzK2O4qf0GDWQ=', 'Content-Length': '0'} 2024-03-04T17:53:50.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7075ea00d90f95c-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.165 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.165 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.166 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.166 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.166 DEBUG:boto:Method: HEAD 2024-03-04T17:53:50.166 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.166 DEBUG:boto:Data: 2024-03-04T17:53:50.166 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.166 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.166 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.166 DEBUG:boto:Params: {} 2024-03-04T17:53:50.166 DEBUG:boto:Token: None 2024-03-04T17:53:50.167 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.167 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iXIOnyZpM918g9qzK2O4qf0GDWQ= 2024-03-04T17:53:50.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iXIOnyZpM918g9qzK2O4qf0GDWQ=', 'Content-Length': '0'} 2024-03-04T17:53:50.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e468b38b9a4854f-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:50.170 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.170 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.170 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.170 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.170 DEBUG:boto:Method: GET 2024-03-04T17:53:50.170 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.170 DEBUG:boto:Data: 2024-03-04T17:53:50.171 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.171 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.171 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.171 DEBUG:boto:Params: {} 2024-03-04T17:53:50.171 DEBUG:boto:Token: None 2024-03-04T17:53:50.171 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.171 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yKMvs2CYd18EMrDRJN7Tcq/ci9U= 2024-03-04T17:53:50.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yKMvs2CYd18EMrDRJN7Tcq/ci9U=', 'Content-Length': '0'} 2024-03-04T17:53:50.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc2b5faa1680b4fd-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.216 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.216 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.216 DEBUG:boto:path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.216 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.217 DEBUG:boto:Method: GET 2024-03-04T17:53:50.217 DEBUG:boto:Path: /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.217 DEBUG:boto:Data: 2024-03-04T17:53:50.217 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.217 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.217 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.217 DEBUG:boto:Params: {} 2024-03-04T17:53:50.217 DEBUG:boto:Token: None 2024-03-04T17:53:50.217 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU 2024-03-04T17:53:50.218 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yKMvs2CYd18EMrDRJN7Tcq/ci9U= 2024-03-04T17:53:50.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yKMvs2CYd18EMrDRJN7Tcq/ci9U=', 'Content-Length': '0'} 2024-03-04T17:53:50.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'Acrzzw6Ded5-7UcCqpx-w2RGP5BSbOU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005984d03a56f382d4-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.221 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.221 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.221 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.221 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.221 DEBUG:boto:Method: GET 2024-03-04T17:53:50.221 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.222 DEBUG:boto:Data: 2024-03-04T17:53:50.222 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.222 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.222 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.222 DEBUG:boto:Params: {} 2024-03-04T17:53:50.222 DEBUG:boto:Token: None 2024-03-04T17:53:50.222 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.222 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077f36c833ce66bcb-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.227 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:50.228 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.228 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.228 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.228 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.228 DEBUG:boto:Method: GET 2024-03-04T17:53:50.228 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.228 DEBUG:boto:Data: 2024-03-04T17:53:50.229 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.229 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.229 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.229 DEBUG:boto:Params: {} 2024-03-04T17:53:50.229 DEBUG:boto:Token: None 2024-03-04T17:53:50.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.229 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be7135d67b492409-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.276 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:50.276 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.277 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.277 DEBUG:boto:Method: HEAD 2024-03-04T17:53:50.277 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:50.277 DEBUG:boto:Data: 2024-03-04T17:53:50.277 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.277 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.277 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.277 DEBUG:boto:Params: {} 2024-03-04T17:53:50.277 DEBUG:boto:Token: None 2024-03-04T17:53:50.278 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj 2024-03-04T17:53:50.278 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yu7KXdkAOB6yLu3p13MEoWs08I4= 2024-03-04T17:53:50.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yu7KXdkAOB6yLu3p13MEoWs08I4=', 'Content-Length': '0'} 2024-03-04T17:53:50.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060e4d3b89a7e4876-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.284 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.284 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.284 DEBUG:boto:Method: HEAD 2024-03-04T17:53:50.284 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:50.284 DEBUG:boto:Data: 2024-03-04T17:53:50.284 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.284 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.284 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.285 DEBUG:boto:Params: {} 2024-03-04T17:53:50.285 DEBUG:boto:Token: None 2024-03-04T17:53:50.285 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj 2024-03-04T17:53:50.285 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yu7KXdkAOB6yLu3p13MEoWs08I4= 2024-03-04T17:53:50.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yu7KXdkAOB6yLu3p13MEoWs08I4=', 'Content-Length': '0'} 2024-03-04T17:53:50.291 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb3ad29e98cbc0bc-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:50.291 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.291 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.291 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.291 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.291 DEBUG:boto:Method: GET 2024-03-04T17:53:50.291 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.292 DEBUG:boto:Data: 2024-03-04T17:53:50.292 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.292 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.292 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.292 DEBUG:boto:Params: {} 2024-03-04T17:53:50.292 DEBUG:boto:Token: None 2024-03-04T17:53:50.292 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.292 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M= 2024-03-04T17:53:50.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M=', 'Content-Length': '0'} 2024-03-04T17:53:50.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf71e30e9e1f20e6-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.295 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.295 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.295 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.295 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.296 DEBUG:boto:Method: GET 2024-03-04T17:53:50.296 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.296 DEBUG:boto:Data: 2024-03-04T17:53:50.296 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.296 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.296 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.296 DEBUG:boto:Params: {} 2024-03-04T17:53:50.296 DEBUG:boto:Token: None 2024-03-04T17:53:50.296 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.296 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M= 2024-03-04T17:53:50.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M=', 'Content-Length': '0'} 2024-03-04T17:53:50.300 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9db851dee959017-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.344 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.344 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.344 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.345 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.345 DEBUG:boto:Method: GET 2024-03-04T17:53:50.345 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.345 DEBUG:boto:Data: 2024-03-04T17:53:50.345 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.345 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.345 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.345 DEBUG:boto:Params: {} 2024-03-04T17:53:50.345 DEBUG:boto:Token: None 2024-03-04T17:53:50.345 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.346 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017178054b376f0e8-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.351 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:50.351 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.351 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.351 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.351 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.352 DEBUG:boto:Method: GET 2024-03-04T17:53:50.352 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.352 DEBUG:boto:Data: 2024-03-04T17:53:50.352 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.352 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.352 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.352 DEBUG:boto:Params: {} 2024-03-04T17:53:50.352 DEBUG:boto:Token: None 2024-03-04T17:53:50.352 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.353 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000463cdbdc721ab1e1-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.400 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:50.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:50.401 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.401 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.401 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.401 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.401 DEBUG:boto:Method: GET 2024-03-04T17:53:50.401 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.401 DEBUG:boto:Data: 2024-03-04T17:53:50.401 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.401 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.402 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.402 DEBUG:boto:Params: {} 2024-03-04T17:53:50.402 DEBUG:boto:Token: None 2024-03-04T17:53:50.402 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.402 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LJwRWkPGY3fe9uX03xynklPflYw= 2024-03-04T17:53:50.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LJwRWkPGY3fe9uX03xynklPflYw=', 'Content-Length': '0'} 2024-03-04T17:53:50.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dab40b4359d23cd5-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.448 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.448 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.448 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.448 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.449 DEBUG:boto:Method: GET 2024-03-04T17:53:50.449 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.449 DEBUG:boto:Data: 2024-03-04T17:53:50.449 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.449 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.449 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.449 DEBUG:boto:Params: {} 2024-03-04T17:53:50.449 DEBUG:boto:Token: None 2024-03-04T17:53:50.449 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.449 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LJwRWkPGY3fe9uX03xynklPflYw= 2024-03-04T17:53:50.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LJwRWkPGY3fe9uX03xynklPflYw=', 'Content-Length': '0'} 2024-03-04T17:53:50.453 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004375a2da3257ecce-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.453 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.453 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.453 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.453 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.453 DEBUG:boto:Method: HEAD 2024-03-04T17:53:50.453 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.453 DEBUG:boto:Data: 2024-03-04T17:53:50.454 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.454 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.454 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.454 DEBUG:boto:Params: {} 2024-03-04T17:53:50.454 DEBUG:boto:Token: None 2024-03-04T17:53:50.454 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.454 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:n2+ecvhrkOia7rs+5EIBFF/dO1c= 2024-03-04T17:53:50.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:n2+ecvhrkOia7rs+5EIBFF/dO1c=', 'Content-Length': '0'} 2024-03-04T17:53:50.457 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae2fc79dccf84d88-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.458 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.458 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.458 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.458 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.458 DEBUG:boto:Method: HEAD 2024-03-04T17:53:50.458 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.458 DEBUG:boto:Data: 2024-03-04T17:53:50.458 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.458 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.459 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.459 DEBUG:boto:Params: {} 2024-03-04T17:53:50.459 DEBUG:boto:Token: None 2024-03-04T17:53:50.459 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.459 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:n2+ecvhrkOia7rs+5EIBFF/dO1c= 2024-03-04T17:53:50.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:n2+ecvhrkOia7rs+5EIBFF/dO1c=', 'Content-Length': '0'} 2024-03-04T17:53:50.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000054d0506f1d6501c-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:50.462 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.462 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.463 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.463 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.463 DEBUG:boto:Method: GET 2024-03-04T17:53:50.463 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.463 DEBUG:boto:Data: 2024-03-04T17:53:50.463 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.463 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.463 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.463 DEBUG:boto:Params: {} 2024-03-04T17:53:50.463 DEBUG:boto:Token: None 2024-03-04T17:53:50.463 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.464 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LJwRWkPGY3fe9uX03xynklPflYw= 2024-03-04T17:53:50.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LJwRWkPGY3fe9uX03xynklPflYw=', 'Content-Length': '0'} 2024-03-04T17:53:50.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdac88de1dec3b90-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.512 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.512 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.512 DEBUG:boto:path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.512 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.513 DEBUG:boto:Method: GET 2024-03-04T17:53:50.513 DEBUG:boto:Path: /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.513 DEBUG:boto:Data: 2024-03-04T17:53:50.513 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.513 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.513 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.513 DEBUG:boto:Params: {} 2024-03-04T17:53:50.513 DEBUG:boto:Token: None 2024-03-04T17:53:50.513 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=dVzfZSb65W93TGSPX8waGA5tm4N9lW6 2024-03-04T17:53:50.513 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LJwRWkPGY3fe9uX03xynklPflYw= 2024-03-04T17:53:50.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LJwRWkPGY3fe9uX03xynklPflYw=', 'Content-Length': '0'} 2024-03-04T17:53:50.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'dVzfZSb65W93TGSPX8waGA5tm4N9lW6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bcfa727248f87ba1-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.560 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.560 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.561 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.561 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.561 DEBUG:boto:Method: GET 2024-03-04T17:53:50.561 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.561 DEBUG:boto:Data: 2024-03-04T17:53:50.561 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.561 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.561 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.561 DEBUG:boto:Params: {} 2024-03-04T17:53:50.561 DEBUG:boto:Token: None 2024-03-04T17:53:50.562 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.562 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b93cc621a50e081-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.567 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:50.567 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.567 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.567 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.567 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.568 DEBUG:boto:Method: GET 2024-03-04T17:53:50.568 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.568 DEBUG:boto:Data: 2024-03-04T17:53:50.568 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.568 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.568 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.568 DEBUG:boto:Params: {} 2024-03-04T17:53:50.568 DEBUG:boto:Token: None 2024-03-04T17:53:50.568 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.569 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd0e43a4f2bfcbce-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.574 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:50.574 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.574 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.574 DEBUG:boto:Method: HEAD 2024-03-04T17:53:50.575 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:50.575 DEBUG:boto:Data: 2024-03-04T17:53:50.575 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.575 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.575 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.575 DEBUG:boto:Params: {} 2024-03-04T17:53:50.575 DEBUG:boto:Token: None 2024-03-04T17:53:50.575 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj 2024-03-04T17:53:50.575 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yu7KXdkAOB6yLu3p13MEoWs08I4= 2024-03-04T17:53:50.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yu7KXdkAOB6yLu3p13MEoWs08I4=', 'Content-Length': '0'} 2024-03-04T17:53:50.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e878885ce0a2c279-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.581 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.581 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.581 DEBUG:boto:Method: HEAD 2024-03-04T17:53:50.581 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:50.581 DEBUG:boto:Data: 2024-03-04T17:53:50.582 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.582 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.582 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.582 DEBUG:boto:Params: {} 2024-03-04T17:53:50.582 DEBUG:boto:Token: None 2024-03-04T17:53:50.582 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj 2024-03-04T17:53:50.582 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yu7KXdkAOB6yLu3p13MEoWs08I4= 2024-03-04T17:53:50.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yu7KXdkAOB6yLu3p13MEoWs08I4=', 'Content-Length': '0'} 2024-03-04T17:53:50.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e52e0c10cf80b21-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:50.588 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.589 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.589 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.589 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.589 DEBUG:boto:Method: GET 2024-03-04T17:53:50.589 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.589 DEBUG:boto:Data: 2024-03-04T17:53:50.589 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.589 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.589 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.589 DEBUG:boto:Params: {} 2024-03-04T17:53:50.589 DEBUG:boto:Token: None 2024-03-04T17:53:50.590 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.590 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M= 2024-03-04T17:53:50.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M=', 'Content-Length': '0'} 2024-03-04T17:53:50.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0f1eba25184b51a-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.593 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.593 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.593 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.593 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.593 DEBUG:boto:Method: GET 2024-03-04T17:53:50.593 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.593 DEBUG:boto:Data: 2024-03-04T17:53:50.593 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.594 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.594 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.594 DEBUG:boto:Params: {} 2024-03-04T17:53:50.594 DEBUG:boto:Token: None 2024-03-04T17:53:50.594 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.594 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M= 2024-03-04T17:53:50.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M=', 'Content-Length': '0'} 2024-03-04T17:53:50.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9ddb21f4c3eb982-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.640 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.640 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.640 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.640 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.640 DEBUG:boto:Method: GET 2024-03-04T17:53:50.641 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.641 DEBUG:boto:Data: 2024-03-04T17:53:50.641 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.641 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.641 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.641 DEBUG:boto:Params: {} 2024-03-04T17:53:50.641 DEBUG:boto:Token: None 2024-03-04T17:53:50.641 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.641 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000682533f9245d72e-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.647 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:50.647 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.648 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.648 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.648 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.648 DEBUG:boto:Method: GET 2024-03-04T17:53:50.648 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.648 DEBUG:boto:Data: 2024-03-04T17:53:50.648 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.648 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.648 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.648 DEBUG:boto:Params: {} 2024-03-04T17:53:50.648 DEBUG:boto:Token: None 2024-03-04T17:53:50.649 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.649 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf3a7c36347a9c98-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.654 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:50.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:50.654 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.654 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.654 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.655 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.655 DEBUG:boto:Method: GET 2024-03-04T17:53:50.655 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.655 DEBUG:boto:Data: 2024-03-04T17:53:50.655 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.655 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.655 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.655 DEBUG:boto:Params: {} 2024-03-04T17:53:50.655 DEBUG:boto:Token: None 2024-03-04T17:53:50.656 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.656 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bUHDCHMgoMQx0N3k8hKoT6UJhxU= 2024-03-04T17:53:50.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bUHDCHMgoMQx0N3k8hKoT6UJhxU=', 'Content-Length': '0'} 2024-03-04T17:53:50.659 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a7ec0848529ddd0-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.704 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.704 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.704 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.704 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.704 DEBUG:boto:Method: GET 2024-03-04T17:53:50.705 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.705 DEBUG:boto:Data: 2024-03-04T17:53:50.705 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.705 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.705 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.705 DEBUG:boto:Params: {} 2024-03-04T17:53:50.705 DEBUG:boto:Token: None 2024-03-04T17:53:50.705 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.705 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bUHDCHMgoMQx0N3k8hKoT6UJhxU= 2024-03-04T17:53:50.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bUHDCHMgoMQx0N3k8hKoT6UJhxU=', 'Content-Length': '0'} 2024-03-04T17:53:50.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003329c06ca789dca3-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.709 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.709 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.709 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.710 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.710 DEBUG:boto:Method: HEAD 2024-03-04T17:53:50.710 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.710 DEBUG:boto:Data: 2024-03-04T17:53:50.710 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.710 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.710 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.710 DEBUG:boto:Params: {} 2024-03-04T17:53:50.710 DEBUG:boto:Token: None 2024-03-04T17:53:50.710 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.710 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bC7P90qta4IIrirQVfmUd6HJhy8= 2024-03-04T17:53:50.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bC7P90qta4IIrirQVfmUd6HJhy8=', 'Content-Length': '0'} 2024-03-04T17:53:50.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d1ec4228942a894-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.714 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.714 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.714 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.714 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.715 DEBUG:boto:Method: HEAD 2024-03-04T17:53:50.715 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.715 DEBUG:boto:Data: 2024-03-04T17:53:50.715 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.715 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.715 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.715 DEBUG:boto:Params: {} 2024-03-04T17:53:50.715 DEBUG:boto:Token: None 2024-03-04T17:53:50.715 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.715 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bC7P90qta4IIrirQVfmUd6HJhy8= 2024-03-04T17:53:50.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bC7P90qta4IIrirQVfmUd6HJhy8=', 'Content-Length': '0'} 2024-03-04T17:53:50.719 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091cc4a09aef2a456-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:50.719 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.719 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.719 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.719 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.720 DEBUG:boto:Method: GET 2024-03-04T17:53:50.720 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.720 DEBUG:boto:Data: 2024-03-04T17:53:50.720 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.720 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.720 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.720 DEBUG:boto:Params: {} 2024-03-04T17:53:50.720 DEBUG:boto:Token: None 2024-03-04T17:53:50.720 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.721 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bUHDCHMgoMQx0N3k8hKoT6UJhxU= 2024-03-04T17:53:50.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bUHDCHMgoMQx0N3k8hKoT6UJhxU=', 'Content-Length': '0'} 2024-03-04T17:53:50.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000960409676dffa45-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.724 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.724 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.724 DEBUG:boto:path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.724 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.724 DEBUG:boto:Method: GET 2024-03-04T17:53:50.724 DEBUG:boto:Path: /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.725 DEBUG:boto:Data: 2024-03-04T17:53:50.725 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.725 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.725 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.725 DEBUG:boto:Params: {} 2024-03-04T17:53:50.725 DEBUG:boto:Token: None 2024-03-04T17:53:50.725 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e 2024-03-04T17:53:50.725 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bUHDCHMgoMQx0N3k8hKoT6UJhxU= 2024-03-04T17:53:50.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bUHDCHMgoMQx0N3k8hKoT6UJhxU=', 'Content-Length': '0'} 2024-03-04T17:53:50.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:30 GMT'), ('x-amz-version-id', 'lwBZf1YBiU2Wqy250Elw-2.f6FuQF0e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e902e3c7d568af4f-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.776 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.776 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.776 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.776 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.776 DEBUG:boto:Method: GET 2024-03-04T17:53:50.777 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.777 DEBUG:boto:Data: 2024-03-04T17:53:50.777 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.777 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.777 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.777 DEBUG:boto:Params: {} 2024-03-04T17:53:50.777 DEBUG:boto:Token: None 2024-03-04T17:53:50.777 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.777 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018fbb860881d08ab-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.782 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:50.783 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.783 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.783 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.783 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.783 DEBUG:boto:Method: GET 2024-03-04T17:53:50.783 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.783 DEBUG:boto:Data: 2024-03-04T17:53:50.783 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.784 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.784 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.784 DEBUG:boto:Params: {} 2024-03-04T17:53:50.784 DEBUG:boto:Token: None 2024-03-04T17:53:50.784 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.784 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b13d59bc20a9e755-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.832 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:50.832 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.832 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.833 DEBUG:boto:Method: HEAD 2024-03-04T17:53:50.833 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:50.833 DEBUG:boto:Data: 2024-03-04T17:53:50.833 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.833 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.833 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.833 DEBUG:boto:Params: {} 2024-03-04T17:53:50.833 DEBUG:boto:Token: None 2024-03-04T17:53:50.834 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj 2024-03-04T17:53:50.834 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yu7KXdkAOB6yLu3p13MEoWs08I4= 2024-03-04T17:53:50.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yu7KXdkAOB6yLu3p13MEoWs08I4=', 'Content-Length': '0'} 2024-03-04T17:53:50.840 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e63bdfc930d64965-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.840 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.841 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.841 DEBUG:boto:Method: HEAD 2024-03-04T17:53:50.841 DEBUG:boto:Path: /ppfahn-22/obj 2024-03-04T17:53:50.841 DEBUG:boto:Data: 2024-03-04T17:53:50.841 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.841 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.841 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.841 DEBUG:boto:Params: {} 2024-03-04T17:53:50.841 DEBUG:boto:Token: None 2024-03-04T17:53:50.842 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj 2024-03-04T17:53:50.842 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Yu7KXdkAOB6yLu3p13MEoWs08I4= 2024-03-04T17:53:50.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Yu7KXdkAOB6yLu3p13MEoWs08I4=', 'Content-Length': '0'} 2024-03-04T17:53:50.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a94b481cf5ac6a39-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:53:50.849 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.849 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.849 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.849 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.849 DEBUG:boto:Method: GET 2024-03-04T17:53:50.849 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.849 DEBUG:boto:Data: 2024-03-04T17:53:50.850 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.850 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.850 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.850 DEBUG:boto:Params: {} 2024-03-04T17:53:50.850 DEBUG:boto:Token: None 2024-03-04T17:53:50.850 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.850 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M= 2024-03-04T17:53:50.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M=', 'Content-Length': '0'} 2024-03-04T17:53:50.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f698498535c2884-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.896 DEBUG:boto:path=/ppfahn-22/obj 2024-03-04T17:53:50.896 DEBUG:boto:auth_path=/ppfahn-22/obj 2024-03-04T17:53:50.896 DEBUG:boto:path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.896 DEBUG:boto:auth_path=/ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.897 DEBUG:boto:Method: GET 2024-03-04T17:53:50.897 DEBUG:boto:Path: /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.897 DEBUG:boto:Data: 2024-03-04T17:53:50.897 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.897 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.897 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.897 DEBUG:boto:Params: {} 2024-03-04T17:53:50.897 DEBUG:boto:Token: None 2024-03-04T17:53:50.897 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/obj?versionId=G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w 2024-03-04T17:53:50.898 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M= 2024-03-04T17:53:50.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EH2z6/lKBQuAXevaRx2GoOUtL1M=', 'Content-Length': '0'} 2024-03-04T17:53:50.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:51:31 GMT'), ('x-amz-version-id', 'G6tTJPArSLLFYQGx5Z9J94Qp3btVq8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-22:d96636ac-054d-40e3-90f3-e3912da784b2.4349.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005411adbd5905cfc-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.944 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.944 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.944 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.945 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.945 DEBUG:boto:Method: GET 2024-03-04T17:53:50.945 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.945 DEBUG:boto:Data: 2024-03-04T17:53:50.945 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.945 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:53:50.945 DEBUG:boto:Port: 8001 2024-03-04T17:53:50.945 DEBUG:boto:Params: {} 2024-03-04T17:53:50.945 DEBUG:boto:Token: None 2024-03-04T17:53:50.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.946 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002afda21b37455c32-0065e60aae-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:50.951 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:50.952 DEBUG:boto:path=/ppfahn-22/ 2024-03-04T17:53:50.952 DEBUG:boto:auth_path=/ppfahn-22/ 2024-03-04T17:53:50.952 DEBUG:boto:path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.952 DEBUG:boto:auth_path=/ppfahn-22/?prefix=obj 2024-03-04T17:53:50.952 DEBUG:boto:Method: GET 2024-03-04T17:53:50.952 DEBUG:boto:Path: /ppfahn-22/?prefix=obj 2024-03-04T17:53:50.953 DEBUG:boto:Data: 2024-03-04T17:53:50.953 DEBUG:boto:Headers: {} 2024-03-04T17:53:50.953 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:53:50.953 DEBUG:boto:Port: 8000 2024-03-04T17:53:50.953 DEBUG:boto:Params: {} 2024-03-04T17:53:50.953 DEBUG:boto:Token: None 2024-03-04T17:53:50.953 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:53:50 GMT /ppfahn-22/ 2024-03-04T17:53:50.953 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M= 2024-03-04T17:53:50.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:50 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QEXjxUIi0Q7WfHhe73nU6yl2d6M=', 'Content-Length': '0'} 2024-03-04T17:53:50.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080183a1d6cbff250-0065e60aae-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:51.000 DEBUG:boto:b'ppfahn-22obj1000falseobj2024-03-04T17:51:31.108Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:53:51.001 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-22 zones={test-zone3, test-zone2} 2024-03-04T17:53:51.001 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-03-04T17:53:51.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.001 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:51.127 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"00000000000000000000:00000000000000003760","last_update":"2024-03-04T17:41:19.567751Z"},{"marker":"00000000000000000000:00000000000000003896","last_update":"2024-03-04T17:47:11.591249Z"},{"marker":"00000000000000000000:00000000000000004037","last_update":"2024-03-04T17:53:03.620968Z"},{"marker":"00000000000000000000:00000000000000009113","last_update":"2024-03-04T17:51:35.609966Z"}] 2024-03-04T17:53:51.127 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000003896', 2: '00000000000000000000:00000000000000004037', 3: '00000000000000000000:00000000000000009113'} 2024-03-04T17:53:51.127 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-03-04T17:53:51.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.128 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:51.173 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:51.173 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6459291056959814556},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:41:19.567751Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003896","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:47:11.591249Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004037","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:03.620968Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009113","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:51:35.609966Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T17:53:51Z"} 2024-03-04T17:53:51.256 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6459291056959814556},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:41:19.567751Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003896","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:47:11.591249Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004037","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:03.620968Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009113","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:51:35.609966Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T17:53:51Z"} 2024-03-04T17:53:51.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003760', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:41:19.567751Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:47:11.591249Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004037', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:53:03.620968Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009113', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:51:35.609966Z'}}] 2024-03-04T17:53:51.256 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000003896', 2: '00000000000000000000:00000000000000004037', 3: '00000000000000000000:00000000000000009113'} 2024-03-04T17:53:51.256 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000003896', 2: '00000000000000000000:00000000000000004037', 3: '00000000000000000000:00000000000000009113'} 2024-03-04T17:53:51.256 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-03-04T17:53:51.256 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2024-03-04T17:53:51.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.257 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:51.298 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:51.298 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:51.391 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"00000000000000000000:00000000000000003760","last_update":"2024-03-04T17:41:19.567751Z"},{"marker":"00000000000000000000:00000000000000003896","last_update":"2024-03-04T17:47:11.591249Z"},{"marker":"00000000000000000000:00000000000000004037","last_update":"2024-03-04T17:53:03.620968Z"},{"marker":"00000000000000000000:00000000000000009113","last_update":"2024-03-04T17:51:35.609966Z"}] 2024-03-04T17:53:51.391 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000003896', 2: '00000000000000000000:00000000000000004037', 3: '00000000000000000000:00000000000000009113'} 2024-03-04T17:53:51.391 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-03-04T17:53:51.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.392 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:51.476 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:51.476 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:51.555 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8128840547189210959},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:41:19.567751Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003896","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:47:11.591249Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004037","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:03.620968Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009113","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:51:35.609966Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T17:53:51Z"} 2024-03-04T17:53:51.556 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8128840547189210959},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:41:19.567751Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003896","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:47:11.591249Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004037","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:03.620968Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009113","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:51:35.609966Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T17:53:51Z"} 2024-03-04T17:53:51.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003760', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:41:19.567751Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:47:11.591249Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004037', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:53:03.620968Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009113', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:51:35.609966Z'}}] 2024-03-04T17:53:51.556 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000003896', 2: '00000000000000000000:00000000000000004037', 3: '00000000000000000000:00000000000000009113'} 2024-03-04T17:53:51.556 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000003896', 2: '00000000000000000000:00000000000000004037', 3: '00000000000000000000:00000000000000009113'} 2024-03-04T17:53:51.556 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-03-04T17:53:51.556 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-03-04T17:53:51.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.557 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:51.599 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:51.599 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:51.685 INFO:teuthology.orchestra.run.smithi153.stdout:[{"marker":"00000000000000000000:00000000000000003760","last_update":"2024-03-04T17:41:19.523439Z"},{"marker":"00000000000000000000:00000000000000004178","last_update":"2024-03-04T17:48:25.140633Z"},{"marker":"00000000000000000000:00000000000000004319","last_update":"2024-03-04T17:53:25.577583Z"},{"marker":"00000000000000000000:00000000000000008831","last_update":"2024-03-04T17:51:35.561996Z"}] 2024-03-04T17:53:51.686 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000004178', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000008831'} 2024-03-04T17:53:51.686 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-03-04T17:53:51.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.686 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:51.734 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:51.734 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:51.833 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1353568530578591994},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:41:19.523439Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:48:25.140633Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004319","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:25.577583Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008831","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:51:35.561996Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T17:53:51Z"} 2024-03-04T17:53:51.834 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1353568530578591994},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:41:19.523439Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:48:25.140633Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004319","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:25.577583Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008831","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:51:35.561996Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T17:53:51Z"} 2024-03-04T17:53:51.834 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003760', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:41:19.523439Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:48:25.140633Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:53:25.577583Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008831', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:51:35.561996Z'}}] 2024-03-04T17:53:51.834 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000004178', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000008831'} 2024-03-04T17:53:51.834 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000004178', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000008831'} 2024-03-04T17:53:51.834 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-03-04T17:53:51.834 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2024-03-04T17:53:51.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.835 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:51.878 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:51.879 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:51.963 INFO:teuthology.orchestra.run.smithi153.stdout:[{"marker":"00000000000000000000:00000000000000003760","last_update":"2024-03-04T17:41:19.523439Z"},{"marker":"00000000000000000000:00000000000000004178","last_update":"2024-03-04T17:48:25.140633Z"},{"marker":"00000000000000000000:00000000000000004319","last_update":"2024-03-04T17:53:25.577583Z"},{"marker":"00000000000000000000:00000000000000008831","last_update":"2024-03-04T17:51:35.561996Z"}] 2024-03-04T17:53:51.963 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000004178', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000008831'} 2024-03-04T17:53:51.963 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-03-04T17:53:51.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:51.964 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:52.008 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:52.009 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:52.088 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3757128659127286239},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:41:19.523439Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:48:25.140633Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004319","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:25.577583Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008831","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:51:35.561996Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T17:53:52Z"} 2024-03-04T17:53:52.088 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3757128659127286239},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:41:19.523439Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:48:25.140633Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004319","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:25.577583Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008831","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:51:35.561996Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T17:53:52Z"} 2024-03-04T17:53:52.088 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003760', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:41:19.523439Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:48:25.140633Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:53:25.577583Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008831', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:51:35.561996Z'}}] 2024-03-04T17:53:52.088 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000004178', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000008831'} 2024-03-04T17:53:52.088 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000004178', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000008831'} 2024-03-04T17:53:52.088 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-03-04T17:53:52.089 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2024-03-04T17:53:52.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:52.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:52.089 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:52.169 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:52.169 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:52.251 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"00000000000000000000:00000000000000003760","last_update":"2024-03-04T17:41:03.651607Z"},{"marker":"00000000000000000000:00000000000000004178","last_update":"2024-03-04T17:47:28.197879Z"},{"marker":"00000000000000000000:00000000000000004319","last_update":"2024-03-04T17:53:09.707816Z"},{"marker":"00000000000000000000:00000000000000008690","last_update":"2024-03-04T17:51:41.697162Z"}] 2024-03-04T17:53:52.251 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000004178', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000008690'} 2024-03-04T17:53:52.252 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-03-04T17:53:52.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:52.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:52.252 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:52.332 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:52.332 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:52.409 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2656409305820931233},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:41:03.651607Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:47:28.197879Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004319","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:09.707816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008690","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:51:41.697162Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T17:53:52Z"} 2024-03-04T17:53:52.409 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2656409305820931233},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:41:03.651607Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:47:28.197879Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004319","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:09.707816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008690","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:51:41.697162Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T17:53:52Z"} 2024-03-04T17:53:52.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003760', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:41:03.651607Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:47:28.197879Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:53:09.707816Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008690', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:51:41.697162Z'}}] 2024-03-04T17:53:52.409 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000004178', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000008690'} 2024-03-04T17:53:52.409 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000004178', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000008690'} 2024-03-04T17:53:52.409 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-03-04T17:53:52.410 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2024-03-04T17:53:52.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:52.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:52.410 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:52.483 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:52.483 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:52.559 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"00000000000000000000:00000000000000003760","last_update":"2024-03-04T17:41:03.651607Z"},{"marker":"00000000000000000000:00000000000000004178","last_update":"2024-03-04T17:47:28.197879Z"},{"marker":"00000000000000000000:00000000000000004319","last_update":"2024-03-04T17:53:09.707816Z"},{"marker":"00000000000000000000:00000000000000008690","last_update":"2024-03-04T17:51:41.697162Z"}] 2024-03-04T17:53:52.560 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000004178', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000008690'} 2024-03-04T17:53:52.560 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-03-04T17:53:52.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:52.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:52.560 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:52.687 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8447256778796353175},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:41:03.651607Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:47:28.197879Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004319","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:09.707816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008690","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:51:41.697162Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T17:53:52Z"} 2024-03-04T17:53:52.688 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8447256778796353175},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:41:03.651607Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:47:28.197879Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004319","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:09.707816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008690","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:51:41.697162Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T17:53:52Z"} 2024-03-04T17:53:52.688 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003760', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:41:03.651607Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:47:28.197879Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:53:09.707816Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008690', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:51:41.697162Z'}}] 2024-03-04T17:53:52.688 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000004178', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000008690'} 2024-03-04T17:53:52.688 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003760', 1: '00000000000000000000:00000000000000004178', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000008690'} 2024-03-04T17:53:52.688 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-03-04T17:53:52.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2024-03-04T17:53:52.691 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:53:52.691 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:53:52.691 DEBUG:boto:Method: PUT 2024-03-04T17:53:52.691 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:53:52.691 DEBUG:boto:Data: 2024-03-04T17:53:52.691 DEBUG:boto:Headers: {} 2024-03-04T17:53:52.691 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:52.691 DEBUG:boto:Port: 8000 2024-03-04T17:53:52.691 DEBUG:boto:Params: {} 2024-03-04T17:53:52.692 DEBUG:boto:Token: None 2024-03-04T17:53:52.692 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:53:52 GMT /ppfahn-23/ 2024-03-04T17:53:52.692 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cTxFnjxHMogNabmkHH28sDQ0Jfo= 2024-03-04T17:53:52.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cTxFnjxHMogNabmkHH28sDQ0Jfo=', 'Content-Length': '0'} 2024-03-04T17:53:52.723 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ae40db6f0f0bfb03-0065e60ab0-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:53:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:52.723 DEBUG:rgw_multi.tests:created bucket=ppfahn-23 2024-03-04T17:53:52.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:52.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:52.724 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:52.763 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:52.763 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:52.816 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:53:52.838 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709574832.724592_190.1","last_update":"2024-03-04T17:53:52.724592Z"}] 2024-03-04T17:53:52.838 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574832.724592_190.1'} 2024-03-04T17:53:52.838 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:53:52.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:52.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:52.839 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:52.885 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:52.885 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:52.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:53:52.945+0000 7f7e10046980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:53:52.945+0000 7f7e10046980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:52.964 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574832.709979_187.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:52.709979Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:52Z"} 2024-03-04T17:53:52.964 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574832.709979_187.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:52.709979Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:52Z"} 2024-03-04T17:53:52.964 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574832.709979_187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:53:52.709979Z', 'realm_epoch': 2}}] 2024-03-04T17:53:52.964 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574832.724592_190.1'} 2024-03-04T17:53:52.965 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574832.709979_187.1'} 2024-03-04T17:53:52.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709574832.724592_190.1 target=1_1709574832.709979_187.1 2024-03-04T17:53:57.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:57.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:57.966 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:58.010 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:58.010 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:58.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:53:58.068+0000 7fb6de9ff980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:58.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:53:58.068+0000 7fb6de9ff980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:58.091 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574832.724592_190.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:52.724592Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:58Z"} 2024-03-04T17:53:58.092 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574832.724592_190.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:52.724592Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:58Z"} 2024-03-04T17:53:58.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574832.724592_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:53:52.724592Z', 'realm_epoch': 2}}] 2024-03-04T17:53:58.092 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574832.724592_190.1'} 2024-03-04T17:53:58.092 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574832.724592_190.1'} 2024-03-04T17:53:58.092 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:53:58.092 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:53:58.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:58.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:58.093 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:58.134 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:58.135 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:58.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:53:58.191+0000 7fead5d83980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:58.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:53:58.191+0000 7fead5d83980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:58.212 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574832.724592_190.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:52.724592Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:58Z"} 2024-03-04T17:53:58.212 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574832.724592_190.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:52.724592Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:58Z"} 2024-03-04T17:53:58.212 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574832.724592_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:53:52.724592Z', 'realm_epoch': 2}}] 2024-03-04T17:53:58.213 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574832.724592_190.1'} 2024-03-04T17:53:58.213 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574832.724592_190.1'} 2024-03-04T17:53:58.213 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:53:58.213 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:53:58.213 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:53:58.213 DEBUG:boto:Method: HEAD 2024-03-04T17:53:58.213 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:53:58.213 DEBUG:boto:Data: 2024-03-04T17:53:58.213 DEBUG:boto:Headers: {} 2024-03-04T17:53:58.214 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:58.214 DEBUG:boto:Port: 8000 2024-03-04T17:53:58.214 DEBUG:boto:Params: {} 2024-03-04T17:53:58.214 DEBUG:boto:Token: None 2024-03-04T17:53:58.214 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:53:58 GMT /ppfahn-23/ 2024-03-04T17:53:58.214 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ecLPVx/kpFyNcaj4GXq48lfjzF0= 2024-03-04T17:53:58.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:53:58 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ecLPVx/kpFyNcaj4GXq48lfjzF0=', 'Content-Length': '0'} 2024-03-04T17:53:58.217 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b0c9972242c0b1f-0065e60ab6-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:58.217 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:53:58.217 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:53:58.217 DEBUG:boto:Method: PUT 2024-03-04T17:53:58.217 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:53:58.217 DEBUG:boto:Data: 2024-03-04T17:53:58.217 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T17:53:58.218 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:53:58.218 DEBUG:boto:Port: 8000 2024-03-04T17:53:58.218 DEBUG:boto:Params: {} 2024-03-04T17:53:58.218 DEBUG:boto:Token: None 2024-03-04T17:53:58.218 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 17:53:58 GMT /ppfahn-23/obj 2024-03-04T17:53:58.218 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DUpkUPXtwAWiBZ1gRaJXeqB8oKg= 2024-03-04T17:53:58.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:53:58 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DUpkUPXtwAWiBZ1gRaJXeqB8oKg='} 2024-03-04T17:53:58.225 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a658b78ffb967771-0065e60ab6-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:53:58.225 DEBUG:rgw_multi.tests:created initial version id=None 2024-03-04T17:53:58.225 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-23 2024-03-04T17:53:58.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:53:58.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:53:58.226 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:53:58.277 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:58.277 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:53:58.344+0000 7f50b18c8980 1 waiting to reach incremental sync.. 2024-03-04T17:54:03.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:03.344+0000 7f50b18c8980 1 waiting to reach incremental sync.. 2024-03-04T17:54:08.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:08.344+0000 7f50b18c8980 1 waiting to reach incremental sync.. 2024-03-04T17:54:13.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:13.348+0000 7f50b18c8980 1 bucket sync caught up with source: 2024-03-04T17:54:13.347 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000001.27.6, , ] 2024-03-04T17:54:13.348 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000001.27.6, , ] 2024-03-04T17:54:13.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:13.348+0000 7f50b18c8980 0 bucket checkpoint complete 2024-03-04T17:54:13.385 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-23 2024-03-04T17:54:13.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:13.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:13.386 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:54:13.437 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:13.437 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:13.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:13.502+0000 7fde98329980 1 bucket sync caught up with source: 2024-03-04T17:54:13.500 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.27.6, , ] 2024-03-04T17:54:13.500 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000001.27.6, , ] 2024-03-04T17:54:13.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:13.502+0000 7fde98329980 0 bucket checkpoint complete 2024-03-04T17:54:13.522 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-23 2024-03-04T17:54:13.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:13.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:13.523 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:54:13.563 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:13.563 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:13.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:13.622+0000 7fb9bca0a980 1 waiting to reach incremental sync.. 2024-03-04T17:54:18.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:18.622+0000 7fb9bca0a980 1 waiting to reach incremental sync.. 2024-03-04T17:54:23.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:23.626+0000 7fb9bca0a980 1 bucket sync caught up with source: 2024-03-04T17:54:23.626 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.38.6, , ] 2024-03-04T17:54:23.627 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000001.38.6, , ] 2024-03-04T17:54:23.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:23.626+0000 7fb9bca0a980 0 bucket checkpoint complete 2024-03-04T17:54:23.667 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-23 2024-03-04T17:54:23.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:23.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:23.668 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:54:23.774 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:23.774 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:23.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:23.838+0000 7fdedd4e6980 1 waiting to reach incremental sync.. 2024-03-04T17:54:28.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:28.838+0000 7fdedd4e6980 1 bucket sync caught up with source: 2024-03-04T17:54:28.839 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.38.6, , ] 2024-03-04T17:54:28.839 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000001.38.6, , ] 2024-03-04T17:54:28.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:28.838+0000 7fdedd4e6980 0 bucket checkpoint complete 2024-03-04T17:54:28.881 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-23 2024-03-04T17:54:28.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:28.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:28.882 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:54:28.930 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:28.931 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:28.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:28.990+0000 7f62d8d5f980 1 bucket sync caught up with source: 2024-03-04T17:54:28.991 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.44.6, , ] 2024-03-04T17:54:28.991 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000001.44.6, , ] 2024-03-04T17:54:28.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:28.990+0000 7f62d8d5f980 0 bucket checkpoint complete 2024-03-04T17:54:29.012 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-23 2024-03-04T17:54:29.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:29.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:29.013 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:54:29.064 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:29.065 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:29.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:29.128+0000 7f8849355980 1 bucket sync caught up with source: 2024-03-04T17:54:29.128 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000001.44.6, , ] 2024-03-04T17:54:29.128 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000001.44.6, , ] 2024-03-04T17:54:29.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:29.128+0000 7f8849355980 0 bucket checkpoint complete 2024-03-04T17:54:29.150 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-23 zones={test-zone2, test-zone1} 2024-03-04T17:54:29.151 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.151 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.151 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.151 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:54:29.151 DEBUG:boto:Data: 2024-03-04T17:54:29.151 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.151 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.151 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.151 DEBUG:boto:Params: {} 2024-03-04T17:54:29.152 DEBUG:boto:Token: None 2024-03-04T17:54:29.152 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.152 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zlIqRhk9QNmhC49OX7iPnIXt0qs= 2024-03-04T17:54:29.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zlIqRhk9QNmhC49OX7iPnIXt0qs=', 'Content-Length': '0'} 2024-03-04T17:54:29.155 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f80fbfbec3affacb-0065e60ad5-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.155 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.155 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.155 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.155 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:54:29.155 DEBUG:boto:Data: 2024-03-04T17:54:29.155 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.155 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.156 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.156 DEBUG:boto:Params: {} 2024-03-04T17:54:29.156 DEBUG:boto:Token: None 2024-03-04T17:54:29.156 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.156 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zlIqRhk9QNmhC49OX7iPnIXt0qs= 2024-03-04T17:54:29.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zlIqRhk9QNmhC49OX7iPnIXt0qs=', 'Content-Length': '0'} 2024-03-04T17:54:29.160 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006cb3763bc1625770-0065e60ad5-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.160 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:54:29.160 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.160 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.160 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.160 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.161 DEBUG:boto:Method: GET 2024-03-04T17:54:29.161 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.161 DEBUG:boto:Data: 2024-03-04T17:54:29.161 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.161 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.161 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.161 DEBUG:boto:Params: {} 2024-03-04T17:54:29.161 DEBUG:boto:Token: None 2024-03-04T17:54:29.161 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/?versions 2024-03-04T17:54:29.161 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo= 2024-03-04T17:54:29.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo=', 'Content-Length': '0'} 2024-03-04T17:54:29.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001947fb5d246fdd1c-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.164 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:54:29.165 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:54:29.165 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.165 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.165 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.165 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.165 DEBUG:boto:Method: GET 2024-03-04T17:54:29.166 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.166 DEBUG:boto:Data: 2024-03-04T17:54:29.166 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.166 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.166 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.166 DEBUG:boto:Params: {} 2024-03-04T17:54:29.166 DEBUG:boto:Token: None 2024-03-04T17:54:29.166 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/?versions 2024-03-04T17:54:29.166 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo= 2024-03-04T17:54:29.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo=', 'Content-Length': '0'} 2024-03-04T17:54:29.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ad5316639e9d36e-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.170 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:54:29.171 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.171 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.171 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.171 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.171 DEBUG:boto:Method: GET 2024-03-04T17:54:29.171 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.171 DEBUG:boto:Data: 2024-03-04T17:54:29.172 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.172 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.172 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.172 DEBUG:boto:Params: {} 2024-03-04T17:54:29.172 DEBUG:boto:Token: None 2024-03-04T17:54:29.172 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/?versions 2024-03-04T17:54:29.172 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo= 2024-03-04T17:54:29.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo=', 'Content-Length': '0'} 2024-03-04T17:54:29.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a26673c6dedebea5-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.174 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.175 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.175 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.175 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.175 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.175 DEBUG:boto:Method: GET 2024-03-04T17:54:29.175 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.175 DEBUG:boto:Data: 2024-03-04T17:54:29.175 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.175 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.175 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.175 DEBUG:boto:Params: {} 2024-03-04T17:54:29.176 DEBUG:boto:Token: None 2024-03-04T17:54:29.176 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/?versions 2024-03-04T17:54:29.176 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo= 2024-03-04T17:54:29.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo=', 'Content-Length': '0'} 2024-03-04T17:54:29.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084ae40078524ac4e-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.224 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:54:29.224 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.225 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.225 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.225 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.225 DEBUG:boto:Method: GET 2024-03-04T17:54:29.225 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.225 DEBUG:boto:Data: 2024-03-04T17:54:29.225 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.225 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.225 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.225 DEBUG:boto:Params: {} 2024-03-04T17:54:29.225 DEBUG:boto:Token: None 2024-03-04T17:54:29.226 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.226 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY= 2024-03-04T17:54:29.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY=', 'Content-Length': '0'} 2024-03-04T17:54:29.228 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000028c64a614164e3f-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.228 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.228 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.228 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.229 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.229 DEBUG:boto:Method: GET 2024-03-04T17:54:29.229 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.229 DEBUG:boto:Data: 2024-03-04T17:54:29.229 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.229 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.229 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.229 DEBUG:boto:Params: {} 2024-03-04T17:54:29.229 DEBUG:boto:Token: None 2024-03-04T17:54:29.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.230 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY= 2024-03-04T17:54:29.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY=', 'Content-Length': '0'} 2024-03-04T17:54:29.232 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001061bc523f9ef9cd-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.233 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.233 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.233 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.233 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.233 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.233 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.233 DEBUG:boto:Data: 2024-03-04T17:54:29.233 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.233 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.233 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.234 DEBUG:boto:Params: {} 2024-03-04T17:54:29.234 DEBUG:boto:Token: None 2024-03-04T17:54:29.234 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.234 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nbCFnOLzQJPHSpKKxxFzfmdvOOI= 2024-03-04T17:54:29.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nbCFnOLzQJPHSpKKxxFzfmdvOOI=', 'Content-Length': '0'} 2024-03-04T17:54:29.237 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f56fc68124fd819-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.237 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.237 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.237 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.237 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.237 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.237 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.237 DEBUG:boto:Data: 2024-03-04T17:54:29.237 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.238 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.238 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.238 DEBUG:boto:Params: {} 2024-03-04T17:54:29.238 DEBUG:boto:Token: None 2024-03-04T17:54:29.238 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.238 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nbCFnOLzQJPHSpKKxxFzfmdvOOI= 2024-03-04T17:54:29.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nbCFnOLzQJPHSpKKxxFzfmdvOOI=', 'Content-Length': '0'} 2024-03-04T17:54:29.241 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000059ad0c2f794336cf-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:54:29.241 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.241 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.241 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.241 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.241 DEBUG:boto:Method: GET 2024-03-04T17:54:29.241 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.242 DEBUG:boto:Data: 2024-03-04T17:54:29.242 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.242 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.242 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.242 DEBUG:boto:Params: {} 2024-03-04T17:54:29.242 DEBUG:boto:Token: None 2024-03-04T17:54:29.242 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.242 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY= 2024-03-04T17:54:29.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY=', 'Content-Length': '0'} 2024-03-04T17:54:29.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf98e02f9cec2f98-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.246 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.246 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.246 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.246 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.246 DEBUG:boto:Method: GET 2024-03-04T17:54:29.246 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.246 DEBUG:boto:Data: 2024-03-04T17:54:29.247 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.247 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.247 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.247 DEBUG:boto:Params: {} 2024-03-04T17:54:29.247 DEBUG:boto:Token: None 2024-03-04T17:54:29.247 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.247 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY= 2024-03-04T17:54:29.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY=', 'Content-Length': '0'} 2024-03-04T17:54:29.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000011b4a1f30d167d0d-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.250 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.250 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.250 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.251 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.251 DEBUG:boto:Method: GET 2024-03-04T17:54:29.251 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:54:29.251 DEBUG:boto:Data: 2024-03-04T17:54:29.251 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.251 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.251 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.251 DEBUG:boto:Params: {} 2024-03-04T17:54:29.251 DEBUG:boto:Token: None 2024-03-04T17:54:29.251 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.252 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q= 2024-03-04T17:54:29.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q=', 'Content-Length': '0'} 2024-03-04T17:54:29.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcde2f910a9465f4-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.300 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.300 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.300 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.300 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.301 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.301 DEBUG:boto:Method: GET 2024-03-04T17:54:29.301 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:54:29.301 DEBUG:boto:Data: 2024-03-04T17:54:29.301 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.301 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.301 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.301 DEBUG:boto:Params: {} 2024-03-04T17:54:29.301 DEBUG:boto:Token: None 2024-03-04T17:54:29.302 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.302 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q= 2024-03-04T17:54:29.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q=', 'Content-Length': '0'} 2024-03-04T17:54:29.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000667018dbf92174e9-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.306 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.306 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.306 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.306 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.307 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:54:29.307 DEBUG:boto:Data: 2024-03-04T17:54:29.307 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.307 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.307 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.307 DEBUG:boto:Params: {} 2024-03-04T17:54:29.307 DEBUG:boto:Token: None 2024-03-04T17:54:29.307 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj 2024-03-04T17:54:29.307 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yUsza5CORg793GXIEUVDTqVEBnE= 2024-03-04T17:54:29.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yUsza5CORg793GXIEUVDTqVEBnE=', 'Content-Length': '0'} 2024-03-04T17:54:29.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000000a394be64667b39-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.310 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.311 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.311 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.311 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:54:29.311 DEBUG:boto:Data: 2024-03-04T17:54:29.311 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.311 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.311 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.311 DEBUG:boto:Params: {} 2024-03-04T17:54:29.311 DEBUG:boto:Token: None 2024-03-04T17:54:29.311 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj 2024-03-04T17:54:29.312 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yUsza5CORg793GXIEUVDTqVEBnE= 2024-03-04T17:54:29.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yUsza5CORg793GXIEUVDTqVEBnE=', 'Content-Length': '0'} 2024-03-04T17:54:29.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fb20a582cc25b498-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:54:29.316 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.316 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.316 DEBUG:boto:Method: GET 2024-03-04T17:54:29.316 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:54:29.316 DEBUG:boto:Data: 2024-03-04T17:54:29.316 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.316 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.316 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.316 DEBUG:boto:Params: {} 2024-03-04T17:54:29.316 DEBUG:boto:Token: None 2024-03-04T17:54:29.317 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj 2024-03-04T17:54:29.317 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A4vOin1TNu3hEd2MmDL7dLG9pEg= 2024-03-04T17:54:29.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A4vOin1TNu3hEd2MmDL7dLG9pEg=', 'Content-Length': '0'} 2024-03-04T17:54:29.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000024aa8deddfee380e-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.320 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.320 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.320 DEBUG:boto:Method: GET 2024-03-04T17:54:29.320 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:54:29.320 DEBUG:boto:Data: 2024-03-04T17:54:29.321 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.321 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.321 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.321 DEBUG:boto:Params: {} 2024-03-04T17:54:29.321 DEBUG:boto:Token: None 2024-03-04T17:54:29.321 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj 2024-03-04T17:54:29.321 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A4vOin1TNu3hEd2MmDL7dLG9pEg= 2024-03-04T17:54:29.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A4vOin1TNu3hEd2MmDL7dLG9pEg=', 'Content-Length': '0'} 2024-03-04T17:54:29.325 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc6df6b8bd0ccaf2-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.325 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.325 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.325 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.325 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.325 DEBUG:boto:Method: GET 2024-03-04T17:54:29.325 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:54:29.326 DEBUG:boto:Data: 2024-03-04T17:54:29.326 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.326 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.326 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.326 DEBUG:boto:Params: {} 2024-03-04T17:54:29.326 DEBUG:boto:Token: None 2024-03-04T17:54:29.326 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.326 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q= 2024-03-04T17:54:29.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q=', 'Content-Length': '0'} 2024-03-04T17:54:29.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000430c395aa2e4fa9-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.331 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.331 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.332 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.332 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.332 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.332 DEBUG:boto:Method: GET 2024-03-04T17:54:29.332 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:54:29.332 DEBUG:boto:Data: 2024-03-04T17:54:29.332 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.332 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.332 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.332 DEBUG:boto:Params: {} 2024-03-04T17:54:29.333 DEBUG:boto:Token: None 2024-03-04T17:54:29.333 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.333 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q= 2024-03-04T17:54:29.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q=', 'Content-Length': '0'} 2024-03-04T17:54:29.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aa8e95ce43fd1f1-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.337 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.337 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-23 zones={test-zone2, test-zone1} 2024-03-04T17:54:29.338 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-23 zones={test-zone3, test-zone1} 2024-03-04T17:54:29.338 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.338 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.338 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.338 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:54:29.338 DEBUG:boto:Data: 2024-03-04T17:54:29.338 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.338 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.338 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.338 DEBUG:boto:Params: {} 2024-03-04T17:54:29.339 DEBUG:boto:Token: None 2024-03-04T17:54:29.339 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.339 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zlIqRhk9QNmhC49OX7iPnIXt0qs= 2024-03-04T17:54:29.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zlIqRhk9QNmhC49OX7iPnIXt0qs=', 'Content-Length': '0'} 2024-03-04T17:54:29.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006da6a813fa314f7-0065e60ad5-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.343 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.343 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.343 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.344 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:54:29.344 DEBUG:boto:Data: 2024-03-04T17:54:29.344 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.344 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.344 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.344 DEBUG:boto:Params: {} 2024-03-04T17:54:29.344 DEBUG:boto:Token: None 2024-03-04T17:54:29.344 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.344 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zlIqRhk9QNmhC49OX7iPnIXt0qs= 2024-03-04T17:54:29.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zlIqRhk9QNmhC49OX7iPnIXt0qs=', 'Content-Length': '0'} 2024-03-04T17:54:29.348 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef502b2ab3470164-0065e60ad5-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.348 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:54:29.349 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.349 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.349 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.349 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.349 DEBUG:boto:Method: GET 2024-03-04T17:54:29.349 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.349 DEBUG:boto:Data: 2024-03-04T17:54:29.349 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.349 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.350 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.350 DEBUG:boto:Params: {} 2024-03-04T17:54:29.350 DEBUG:boto:Token: None 2024-03-04T17:54:29.350 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/?versions 2024-03-04T17:54:29.350 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo= 2024-03-04T17:54:29.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo=', 'Content-Length': '0'} 2024-03-04T17:54:29.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045ba1d96e4f1df1e-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.355 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:54:29.355 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:54:29.356 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.356 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.356 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.356 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.356 DEBUG:boto:Method: GET 2024-03-04T17:54:29.356 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.356 DEBUG:boto:Data: 2024-03-04T17:54:29.356 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.356 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.356 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.357 DEBUG:boto:Params: {} 2024-03-04T17:54:29.357 DEBUG:boto:Token: None 2024-03-04T17:54:29.357 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/?versions 2024-03-04T17:54:29.357 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo= 2024-03-04T17:54:29.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo=', 'Content-Length': '0'} 2024-03-04T17:54:29.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000707ba3614859da4e-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.404 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:54:29.404 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.405 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.405 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.405 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.405 DEBUG:boto:Method: GET 2024-03-04T17:54:29.405 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.405 DEBUG:boto:Data: 2024-03-04T17:54:29.405 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.405 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.405 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.405 DEBUG:boto:Params: {} 2024-03-04T17:54:29.405 DEBUG:boto:Token: None 2024-03-04T17:54:29.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/?versions 2024-03-04T17:54:29.406 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo= 2024-03-04T17:54:29.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo=', 'Content-Length': '0'} 2024-03-04T17:54:29.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca936299fa12aecb-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.411 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.411 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.411 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.411 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.411 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.411 DEBUG:boto:Method: GET 2024-03-04T17:54:29.412 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.412 DEBUG:boto:Data: 2024-03-04T17:54:29.412 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.412 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.412 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.412 DEBUG:boto:Params: {} 2024-03-04T17:54:29.412 DEBUG:boto:Token: None 2024-03-04T17:54:29.412 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/?versions 2024-03-04T17:54:29.412 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo= 2024-03-04T17:54:29.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo=', 'Content-Length': '0'} 2024-03-04T17:54:29.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bd7d67f7f29aac0-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.416 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:54:29.417 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.417 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.417 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.417 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.417 DEBUG:boto:Method: GET 2024-03-04T17:54:29.417 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.417 DEBUG:boto:Data: 2024-03-04T17:54:29.417 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.417 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.418 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.418 DEBUG:boto:Params: {} 2024-03-04T17:54:29.418 DEBUG:boto:Token: None 2024-03-04T17:54:29.418 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.418 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY= 2024-03-04T17:54:29.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY=', 'Content-Length': '0'} 2024-03-04T17:54:29.421 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e8f9d8ef3246b64-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.421 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.421 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.421 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.422 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.422 DEBUG:boto:Method: GET 2024-03-04T17:54:29.422 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.422 DEBUG:boto:Data: 2024-03-04T17:54:29.422 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.422 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.422 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.422 DEBUG:boto:Params: {} 2024-03-04T17:54:29.422 DEBUG:boto:Token: None 2024-03-04T17:54:29.422 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.423 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY= 2024-03-04T17:54:29.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY=', 'Content-Length': '0'} 2024-03-04T17:54:29.425 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006573a2557f805d85-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.425 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.426 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.426 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.426 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.426 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.426 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.426 DEBUG:boto:Data: 2024-03-04T17:54:29.426 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.426 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.426 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.426 DEBUG:boto:Params: {} 2024-03-04T17:54:29.426 DEBUG:boto:Token: None 2024-03-04T17:54:29.427 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.427 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nbCFnOLzQJPHSpKKxxFzfmdvOOI= 2024-03-04T17:54:29.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nbCFnOLzQJPHSpKKxxFzfmdvOOI=', 'Content-Length': '0'} 2024-03-04T17:54:29.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000538f33e16ef2bffd-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.430 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.430 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.430 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.430 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.430 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.430 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.430 DEBUG:boto:Data: 2024-03-04T17:54:29.430 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.430 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.430 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.430 DEBUG:boto:Params: {} 2024-03-04T17:54:29.431 DEBUG:boto:Token: None 2024-03-04T17:54:29.431 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.431 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nbCFnOLzQJPHSpKKxxFzfmdvOOI= 2024-03-04T17:54:29.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nbCFnOLzQJPHSpKKxxFzfmdvOOI=', 'Content-Length': '0'} 2024-03-04T17:54:29.433 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e7247f827dd0627-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:54:29.434 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.434 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.434 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.434 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.434 DEBUG:boto:Method: GET 2024-03-04T17:54:29.434 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.434 DEBUG:boto:Data: 2024-03-04T17:54:29.435 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.435 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.435 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.435 DEBUG:boto:Params: {} 2024-03-04T17:54:29.435 DEBUG:boto:Token: None 2024-03-04T17:54:29.435 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.435 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY= 2024-03-04T17:54:29.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY=', 'Content-Length': '0'} 2024-03-04T17:54:29.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033883dec1d12625f-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.438 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.438 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.438 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.438 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.439 DEBUG:boto:Method: GET 2024-03-04T17:54:29.439 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.439 DEBUG:boto:Data: 2024-03-04T17:54:29.439 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.439 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.439 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.439 DEBUG:boto:Params: {} 2024-03-04T17:54:29.439 DEBUG:boto:Token: None 2024-03-04T17:54:29.439 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.439 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY= 2024-03-04T17:54:29.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY=', 'Content-Length': '0'} 2024-03-04T17:54:29.444 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000704bdbf0c77ba218-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.444 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.444 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.444 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.444 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.444 DEBUG:boto:Method: GET 2024-03-04T17:54:29.444 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:54:29.445 DEBUG:boto:Data: 2024-03-04T17:54:29.445 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.445 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.445 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.445 DEBUG:boto:Params: {} 2024-03-04T17:54:29.445 DEBUG:boto:Token: None 2024-03-04T17:54:29.445 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.445 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q= 2024-03-04T17:54:29.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q=', 'Content-Length': '0'} 2024-03-04T17:54:29.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042ac9d631a70729a-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.492 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.492 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.492 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.492 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.493 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.493 DEBUG:boto:Method: GET 2024-03-04T17:54:29.493 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:54:29.493 DEBUG:boto:Data: 2024-03-04T17:54:29.493 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.493 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.493 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.493 DEBUG:boto:Params: {} 2024-03-04T17:54:29.493 DEBUG:boto:Token: None 2024-03-04T17:54:29.493 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.494 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q= 2024-03-04T17:54:29.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q=', 'Content-Length': '0'} 2024-03-04T17:54:29.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d7eb170fb6dd638-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.501 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.501 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.501 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.501 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.501 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:54:29.502 DEBUG:boto:Data: 2024-03-04T17:54:29.502 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.502 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.502 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.502 DEBUG:boto:Params: {} 2024-03-04T17:54:29.502 DEBUG:boto:Token: None 2024-03-04T17:54:29.502 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj 2024-03-04T17:54:29.502 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yUsza5CORg793GXIEUVDTqVEBnE= 2024-03-04T17:54:29.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yUsza5CORg793GXIEUVDTqVEBnE=', 'Content-Length': '0'} 2024-03-04T17:54:29.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008215546e8464583-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.512 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.512 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.512 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.512 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:54:29.512 DEBUG:boto:Data: 2024-03-04T17:54:29.512 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.513 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.513 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.513 DEBUG:boto:Params: {} 2024-03-04T17:54:29.513 DEBUG:boto:Token: None 2024-03-04T17:54:29.513 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj 2024-03-04T17:54:29.513 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yUsza5CORg793GXIEUVDTqVEBnE= 2024-03-04T17:54:29.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yUsza5CORg793GXIEUVDTqVEBnE=', 'Content-Length': '0'} 2024-03-04T17:54:29.516 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fbe540b9206776a3-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:54:29.517 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.517 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.517 DEBUG:boto:Method: GET 2024-03-04T17:54:29.517 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:54:29.517 DEBUG:boto:Data: 2024-03-04T17:54:29.517 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.517 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.517 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.518 DEBUG:boto:Params: {} 2024-03-04T17:54:29.518 DEBUG:boto:Token: None 2024-03-04T17:54:29.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj 2024-03-04T17:54:29.518 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A4vOin1TNu3hEd2MmDL7dLG9pEg= 2024-03-04T17:54:29.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A4vOin1TNu3hEd2MmDL7dLG9pEg=', 'Content-Length': '0'} 2024-03-04T17:54:29.521 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c81b6767dddb03b-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.521 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.521 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.521 DEBUG:boto:Method: GET 2024-03-04T17:54:29.521 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:54:29.521 DEBUG:boto:Data: 2024-03-04T17:54:29.521 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.521 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.522 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.522 DEBUG:boto:Params: {} 2024-03-04T17:54:29.522 DEBUG:boto:Token: None 2024-03-04T17:54:29.522 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj 2024-03-04T17:54:29.522 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A4vOin1TNu3hEd2MmDL7dLG9pEg= 2024-03-04T17:54:29.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A4vOin1TNu3hEd2MmDL7dLG9pEg=', 'Content-Length': '0'} 2024-03-04T17:54:29.525 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056a4032c790a0fd4-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.525 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.525 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.525 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.525 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.526 DEBUG:boto:Method: GET 2024-03-04T17:54:29.526 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:54:29.526 DEBUG:boto:Data: 2024-03-04T17:54:29.526 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.526 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.526 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.526 DEBUG:boto:Params: {} 2024-03-04T17:54:29.526 DEBUG:boto:Token: None 2024-03-04T17:54:29.526 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.526 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q= 2024-03-04T17:54:29.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q=', 'Content-Length': '0'} 2024-03-04T17:54:29.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d759516df480412d-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.572 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.572 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.572 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.572 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.573 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.573 DEBUG:boto:Method: GET 2024-03-04T17:54:29.573 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:54:29.573 DEBUG:boto:Data: 2024-03-04T17:54:29.573 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.573 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.573 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.573 DEBUG:boto:Params: {} 2024-03-04T17:54:29.573 DEBUG:boto:Token: None 2024-03-04T17:54:29.573 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.573 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q= 2024-03-04T17:54:29.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q=', 'Content-Length': '0'} 2024-03-04T17:54:29.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7a73b9109d52412-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.578 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.578 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-23 zones={test-zone3, test-zone1} 2024-03-04T17:54:29.578 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-23 zones={test-zone3, test-zone2} 2024-03-04T17:54:29.578 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.578 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.579 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.579 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:54:29.579 DEBUG:boto:Data: 2024-03-04T17:54:29.579 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.579 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.579 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.579 DEBUG:boto:Params: {} 2024-03-04T17:54:29.579 DEBUG:boto:Token: None 2024-03-04T17:54:29.579 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.579 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zlIqRhk9QNmhC49OX7iPnIXt0qs= 2024-03-04T17:54:29.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zlIqRhk9QNmhC49OX7iPnIXt0qs=', 'Content-Length': '0'} 2024-03-04T17:54:29.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000184181678827b8f2-0065e60ad5-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.584 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.584 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.584 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.584 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:54:29.584 DEBUG:boto:Data: 2024-03-04T17:54:29.584 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.584 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.584 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.585 DEBUG:boto:Params: {} 2024-03-04T17:54:29.585 DEBUG:boto:Token: None 2024-03-04T17:54:29.585 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.585 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zlIqRhk9QNmhC49OX7iPnIXt0qs= 2024-03-04T17:54:29.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zlIqRhk9QNmhC49OX7iPnIXt0qs=', 'Content-Length': '0'} 2024-03-04T17:54:29.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019a1d479b8823535-0065e60ad5-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.589 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:54:29.589 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.589 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.589 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.590 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.590 DEBUG:boto:Method: GET 2024-03-04T17:54:29.590 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.590 DEBUG:boto:Data: 2024-03-04T17:54:29.590 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.590 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.590 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.590 DEBUG:boto:Params: {} 2024-03-04T17:54:29.590 DEBUG:boto:Token: None 2024-03-04T17:54:29.590 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/?versions 2024-03-04T17:54:29.591 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo= 2024-03-04T17:54:29.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo=', 'Content-Length': '0'} 2024-03-04T17:54:29.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000479b574c30947003-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.644 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:54:29.644 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:54:29.645 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.645 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.645 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.645 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.645 DEBUG:boto:Method: GET 2024-03-04T17:54:29.645 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.645 DEBUG:boto:Data: 2024-03-04T17:54:29.645 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.645 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.645 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.646 DEBUG:boto:Params: {} 2024-03-04T17:54:29.646 DEBUG:boto:Token: None 2024-03-04T17:54:29.646 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/?versions 2024-03-04T17:54:29.646 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo= 2024-03-04T17:54:29.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo=', 'Content-Length': '0'} 2024-03-04T17:54:29.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf875e8c42869f79-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.650 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:54:29.651 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.651 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.651 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.651 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.651 DEBUG:boto:Method: GET 2024-03-04T17:54:29.651 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.651 DEBUG:boto:Data: 2024-03-04T17:54:29.651 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.651 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.651 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.651 DEBUG:boto:Params: {} 2024-03-04T17:54:29.652 DEBUG:boto:Token: None 2024-03-04T17:54:29.652 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/?versions 2024-03-04T17:54:29.652 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo= 2024-03-04T17:54:29.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo=', 'Content-Length': '0'} 2024-03-04T17:54:29.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f710db3f0b146cc5-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.700 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.700 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.700 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.701 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.701 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.701 DEBUG:boto:Method: GET 2024-03-04T17:54:29.701 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:54:29.701 DEBUG:boto:Data: 2024-03-04T17:54:29.701 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.701 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.701 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.701 DEBUG:boto:Params: {} 2024-03-04T17:54:29.701 DEBUG:boto:Token: None 2024-03-04T17:54:29.702 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/?versions 2024-03-04T17:54:29.702 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo= 2024-03-04T17:54:29.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Tco7YD0wYAaIyEABnPDc4ZQ/bRo=', 'Content-Length': '0'} 2024-03-04T17:54:29.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000791a7b350eb580c8-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.706 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:54:29.706 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.706 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.706 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.706 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.707 DEBUG:boto:Method: GET 2024-03-04T17:54:29.707 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.707 DEBUG:boto:Data: 2024-03-04T17:54:29.707 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.707 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.707 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.707 DEBUG:boto:Params: {} 2024-03-04T17:54:29.707 DEBUG:boto:Token: None 2024-03-04T17:54:29.707 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.707 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY= 2024-03-04T17:54:29.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY=', 'Content-Length': '0'} 2024-03-04T17:54:29.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cb5e37b6ffc220fd-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.711 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.711 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.711 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.711 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.711 DEBUG:boto:Method: GET 2024-03-04T17:54:29.711 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.711 DEBUG:boto:Data: 2024-03-04T17:54:29.711 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.711 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.711 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.712 DEBUG:boto:Params: {} 2024-03-04T17:54:29.712 DEBUG:boto:Token: None 2024-03-04T17:54:29.712 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.712 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY= 2024-03-04T17:54:29.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY=', 'Content-Length': '0'} 2024-03-04T17:54:29.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000524a94d373b89aa4-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.715 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.715 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.715 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.715 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.715 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.715 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.715 DEBUG:boto:Data: 2024-03-04T17:54:29.716 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.716 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.716 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.716 DEBUG:boto:Params: {} 2024-03-04T17:54:29.716 DEBUG:boto:Token: None 2024-03-04T17:54:29.716 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.716 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nbCFnOLzQJPHSpKKxxFzfmdvOOI= 2024-03-04T17:54:29.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nbCFnOLzQJPHSpKKxxFzfmdvOOI=', 'Content-Length': '0'} 2024-03-04T17:54:29.719 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b70f5360421b445b-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.719 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.719 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.719 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.719 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.720 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.720 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.720 DEBUG:boto:Data: 2024-03-04T17:54:29.720 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.720 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.720 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.720 DEBUG:boto:Params: {} 2024-03-04T17:54:29.720 DEBUG:boto:Token: None 2024-03-04T17:54:29.720 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.721 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nbCFnOLzQJPHSpKKxxFzfmdvOOI= 2024-03-04T17:54:29.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nbCFnOLzQJPHSpKKxxFzfmdvOOI=', 'Content-Length': '0'} 2024-03-04T17:54:29.723 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000022f2334d8a44240b-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:54:29.724 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.724 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.724 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.724 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.724 DEBUG:boto:Method: GET 2024-03-04T17:54:29.724 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.724 DEBUG:boto:Data: 2024-03-04T17:54:29.724 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.724 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.724 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.725 DEBUG:boto:Params: {} 2024-03-04T17:54:29.725 DEBUG:boto:Token: None 2024-03-04T17:54:29.725 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.725 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY= 2024-03-04T17:54:29.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY=', 'Content-Length': '0'} 2024-03-04T17:54:29.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a8c413dd1eea756-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.728 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.728 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.728 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.728 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.728 DEBUG:boto:Method: GET 2024-03-04T17:54:29.729 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.729 DEBUG:boto:Data: 2024-03-04T17:54:29.729 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.729 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.729 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.729 DEBUG:boto:Params: {} 2024-03-04T17:54:29.729 DEBUG:boto:Token: None 2024-03-04T17:54:29.729 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:54:29.729 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY= 2024-03-04T17:54:29.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IYi+lq4P/kXLsmZ1EdcNCKL6iGY=', 'Content-Length': '0'} 2024-03-04T17:54:29.732 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004360304db286ea9b-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.732 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.733 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.733 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.733 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.733 DEBUG:boto:Method: GET 2024-03-04T17:54:29.733 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:54:29.733 DEBUG:boto:Data: 2024-03-04T17:54:29.733 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.733 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.733 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.733 DEBUG:boto:Params: {} 2024-03-04T17:54:29.733 DEBUG:boto:Token: None 2024-03-04T17:54:29.734 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.734 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q= 2024-03-04T17:54:29.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q=', 'Content-Length': '0'} 2024-03-04T17:54:29.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000399d12f18819ccb3-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.780 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.780 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.780 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.781 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.781 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.781 DEBUG:boto:Method: GET 2024-03-04T17:54:29.781 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:54:29.781 DEBUG:boto:Data: 2024-03-04T17:54:29.781 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.781 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.781 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.781 DEBUG:boto:Params: {} 2024-03-04T17:54:29.781 DEBUG:boto:Token: None 2024-03-04T17:54:29.782 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.782 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q= 2024-03-04T17:54:29.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q=', 'Content-Length': '0'} 2024-03-04T17:54:29.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2adcb55d54d3173-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.786 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.787 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.787 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.787 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.787 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:54:29.787 DEBUG:boto:Data: 2024-03-04T17:54:29.787 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.787 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.787 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.787 DEBUG:boto:Params: {} 2024-03-04T17:54:29.788 DEBUG:boto:Token: None 2024-03-04T17:54:29.788 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj 2024-03-04T17:54:29.788 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yUsza5CORg793GXIEUVDTqVEBnE= 2024-03-04T17:54:29.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yUsza5CORg793GXIEUVDTqVEBnE=', 'Content-Length': '0'} 2024-03-04T17:54:29.791 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002a68c9f161a1a31b-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.791 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.791 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.791 DEBUG:boto:Method: HEAD 2024-03-04T17:54:29.791 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:54:29.791 DEBUG:boto:Data: 2024-03-04T17:54:29.791 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.791 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.792 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.792 DEBUG:boto:Params: {} 2024-03-04T17:54:29.792 DEBUG:boto:Token: None 2024-03-04T17:54:29.792 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj 2024-03-04T17:54:29.792 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yUsza5CORg793GXIEUVDTqVEBnE= 2024-03-04T17:54:29.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yUsza5CORg793GXIEUVDTqVEBnE=', 'Content-Length': '0'} 2024-03-04T17:54:29.795 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a9cfdc90741a13c-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:54:29.795 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.795 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.795 DEBUG:boto:Method: GET 2024-03-04T17:54:29.796 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:54:29.796 DEBUG:boto:Data: 2024-03-04T17:54:29.796 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.796 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.796 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.796 DEBUG:boto:Params: {} 2024-03-04T17:54:29.796 DEBUG:boto:Token: None 2024-03-04T17:54:29.796 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj 2024-03-04T17:54:29.796 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A4vOin1TNu3hEd2MmDL7dLG9pEg= 2024-03-04T17:54:29.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A4vOin1TNu3hEd2MmDL7dLG9pEg=', 'Content-Length': '0'} 2024-03-04T17:54:29.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a43700cf9d65d87-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.799 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:29.799 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:29.799 DEBUG:boto:Method: GET 2024-03-04T17:54:29.800 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:54:29.800 DEBUG:boto:Data: 2024-03-04T17:54:29.800 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.800 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.800 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.800 DEBUG:boto:Params: {} 2024-03-04T17:54:29.800 DEBUG:boto:Token: None 2024-03-04T17:54:29.800 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/obj 2024-03-04T17:54:29.800 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:A4vOin1TNu3hEd2MmDL7dLG9pEg= 2024-03-04T17:54:29.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:A4vOin1TNu3hEd2MmDL7dLG9pEg=', 'Content-Length': '0'} 2024-03-04T17:54:29.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000081049c742766e569-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.804 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.804 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.804 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.804 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.804 DEBUG:boto:Method: GET 2024-03-04T17:54:29.804 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:54:29.804 DEBUG:boto:Data: 2024-03-04T17:54:29.804 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.804 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:54:29.804 DEBUG:boto:Port: 8001 2024-03-04T17:54:29.804 DEBUG:boto:Params: {} 2024-03-04T17:54:29.805 DEBUG:boto:Token: None 2024-03-04T17:54:29.805 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.805 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q= 2024-03-04T17:54:29.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q=', 'Content-Length': '0'} 2024-03-04T17:54:29.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006618c49aafc59520-0065e60ad5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.809 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.809 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.810 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.810 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.810 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:54:29.810 DEBUG:boto:Method: GET 2024-03-04T17:54:29.810 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:54:29.810 DEBUG:boto:Data: 2024-03-04T17:54:29.810 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.810 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:54:29.810 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.810 DEBUG:boto:Params: {} 2024-03-04T17:54:29.810 DEBUG:boto:Token: None 2024-03-04T17:54:29.811 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/ 2024-03-04T17:54:29.811 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q= 2024-03-04T17:54:29.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T3cxj20NlyUyj5OKta128fTRc8Q=', 'Content-Length': '0'} 2024-03-04T17:54:29.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000203a4f07274c116-0065e60ad5-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.860 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:54:29.860 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-23 zones={test-zone3, test-zone2} 2024-03-04T17:54:29.860 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:29.861 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:29.861 DEBUG:boto:path=/ppfahn-23/?versioning 2024-03-04T17:54:29.861 DEBUG:boto:auth_path=/ppfahn-23/?versioning 2024-03-04T17:54:29.861 DEBUG:boto:Method: PUT 2024-03-04T17:54:29.861 DEBUG:boto:Path: /ppfahn-23/?versioning 2024-03-04T17:54:29.861 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:54:29.861 DEBUG:boto:Headers: {} 2024-03-04T17:54:29.861 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:29.861 DEBUG:boto:Port: 8000 2024-03-04T17:54:29.861 DEBUG:boto:Params: {} 2024-03-04T17:54:29.862 DEBUG:boto:Token: None 2024-03-04T17:54:29.862 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:54:29 GMT /ppfahn-23/?versioning 2024-03-04T17:54:29.862 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1tOtdg/u8LDy9+4YNwD2mf6VSog= 2024-03-04T17:54:29.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1tOtdg/u8LDy9+4YNwD2mf6VSog=', 'Content-Length': '228'} 2024-03-04T17:54:29.874 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000901af8e6752c457c-0065e60ad5-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:54:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:29.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:29.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:29.874 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:29.915 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:29.915 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:29.969 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:54:29.991 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709574869.877175_201.1","last_update":"2024-03-04T17:54:29.877175Z"}] 2024-03-04T17:54:29.992 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574869.877175_201.1'} 2024-03-04T17:54:29.992 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:54:29.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:29.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:29.992 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:30.035 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:30.035 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:30.096+0000 7f975d3a0980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:30.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:30.096+0000 7f975d3a0980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:30.121 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574832.724592_190.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:52.724592Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:30Z"} 2024-03-04T17:54:30.121 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574832.724592_190.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:53:52.724592Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:30Z"} 2024-03-04T17:54:30.121 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574832.724592_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:53:52.724592Z', 'realm_epoch': 2}}] 2024-03-04T17:54:30.121 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574869.877175_201.1'} 2024-03-04T17:54:30.121 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574832.724592_190.1'} 2024-03-04T17:54:30.122 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709574869.877175_201.1 target=1_1709574832.724592_190.1 2024-03-04T17:54:35.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:35.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:35.123 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:35.165 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:35.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:35.224+0000 7f27f6e08980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:35.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:35.224+0000 7f27f6e08980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:35.245 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574869.877175_201.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:54:29.877175Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:35Z"} 2024-03-04T17:54:35.245 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574869.877175_201.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:54:29.877175Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:35Z"} 2024-03-04T17:54:35.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574869.877175_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:54:29.877175Z', 'realm_epoch': 2}}] 2024-03-04T17:54:35.245 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574869.877175_201.1'} 2024-03-04T17:54:35.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574869.877175_201.1'} 2024-03-04T17:54:35.245 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:54:35.245 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:54:35.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:35.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:35.246 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:35.289 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:35.289 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:35.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:35.346+0000 7f18b1953980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:35.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:35.346+0000 7f18b1953980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:35.368 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574869.877175_201.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:54:29.877175Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:35Z"} 2024-03-04T17:54:35.369 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574869.877175_201.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:54:29.877175Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:35Z"} 2024-03-04T17:54:35.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574869.877175_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:54:29.877175Z', 'realm_epoch': 2}}] 2024-03-04T17:54:35.369 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574869.877175_201.1'} 2024-03-04T17:54:35.369 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574869.877175_201.1'} 2024-03-04T17:54:35.369 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:54:35.369 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:54:35.369 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:54:35.369 DEBUG:boto:Method: HEAD 2024-03-04T17:54:35.370 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:54:35.370 DEBUG:boto:Data: 2024-03-04T17:54:35.370 DEBUG:boto:Headers: {} 2024-03-04T17:54:35.370 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:35.370 DEBUG:boto:Port: 8000 2024-03-04T17:54:35.370 DEBUG:boto:Params: {} 2024-03-04T17:54:35.370 DEBUG:boto:Token: None 2024-03-04T17:54:35.370 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:54:35 GMT /ppfahn-23/ 2024-03-04T17:54:35.370 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:i729Z9KkVVtnrVo9EvIcaHm8X0I= 2024-03-04T17:54:35.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:54:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:i729Z9KkVVtnrVo9EvIcaHm8X0I=', 'Content-Length': '0'} 2024-03-04T17:54:35.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9c431c1582fd64b-0065e60adb-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:35.373 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:54:35.373 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:54:35.373 DEBUG:boto:Method: PUT 2024-03-04T17:54:35.373 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:54:35.373 DEBUG:boto:Data: 2024-03-04T17:54:35.374 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T17:54:35.374 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:54:35.374 DEBUG:boto:Port: 8000 2024-03-04T17:54:35.374 DEBUG:boto:Params: {} 2024-03-04T17:54:35.374 DEBUG:boto:Token: None 2024-03-04T17:54:35.374 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 17:54:35 GMT /ppfahn-23/obj 2024-03-04T17:54:35.374 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:natyabL9+Ubyf6ZiAYDll717RG8= 2024-03-04T17:54:35.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:54:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:natyabL9+Ubyf6ZiAYDll717RG8='} 2024-03-04T17:54:35.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-amz-request-id', 'tx000008acb9be0cfa14a4d-0065e60adb-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:54:35.420 DEBUG:rgw_multi.tests:created new version id=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:54:35.421 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-23 2024-03-04T17:54:35.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:35.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:35.421 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:54:35.472 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:35.472 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:35.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:35.536+0000 7f852c49f980 1 waiting for incremental sync to catch up: 2024-03-04T17:54:35.535 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000001.27.6, , ] 2024-03-04T17:54:35.535 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000003.30.13, , ] 2024-03-04T17:54:40.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:40.539+0000 7f852c49f980 1 waiting for incremental sync to catch up: 2024-03-04T17:54:40.537 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000001.27.6, , ] 2024-03-04T17:54:40.538 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000003.30.13, , ] 2024-03-04T17:54:45.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:45.539+0000 7f852c49f980 1 waiting for incremental sync to catch up: 2024-03-04T17:54:45.540 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000001.27.6, , ] 2024-03-04T17:54:45.540 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000003.30.13, , ] 2024-03-04T17:54:50.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:50.543+0000 7f852c49f980 1 bucket sync caught up with source: 2024-03-04T17:54:50.542 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000003.30.13, , ] 2024-03-04T17:54:50.542 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000003.30.13, , ] 2024-03-04T17:54:50.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:54:50.543+0000 7f852c49f980 0 bucket checkpoint complete 2024-03-04T17:54:50.588 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-23 2024-03-04T17:54:50.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:50.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:50.588 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:54:50.639 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:50.639 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:50.705+0000 7f685a80e980 1 bucket sync caught up with source: 2024-03-04T17:54:50.706 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000003.30.13, , ] 2024-03-04T17:54:50.706 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.30.13, , ] 2024-03-04T17:54:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:50.705+0000 7f685a80e980 0 bucket checkpoint complete 2024-03-04T17:54:50.734 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-23 2024-03-04T17:54:50.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:50.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:54:50.734 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:54:50.783 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:50.783 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:50.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:50.845+0000 7f382b3e2980 1 waiting for incremental sync to catch up: 2024-03-04T17:54:50.845 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.38.6, , ] 2024-03-04T17:54:50.845 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.43.13, , ] 2024-03-04T17:54:55.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:54:55.849+0000 7f382b3e2980 1 waiting for incremental sync to catch up: 2024-03-04T17:54:55.848 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.38.6, , ] 2024-03-04T17:54:55.848 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.43.13, , ] 2024-03-04T17:55:00.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:00.849+0000 7f382b3e2980 1 waiting for incremental sync to catch up: 2024-03-04T17:55:00.851 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.38.6, , ] 2024-03-04T17:55:00.851 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.43.13, , ] 2024-03-04T17:55:05.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:05.853+0000 7f382b3e2980 1 bucket sync caught up with source: 2024-03-04T17:55:05.853 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000004.45.10, , ] 2024-03-04T17:55:05.853 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.43.13, , ] 2024-03-04T17:55:05.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:05.853+0000 7f382b3e2980 0 bucket checkpoint complete 2024-03-04T17:55:05.893 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-23 2024-03-04T17:55:05.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:55:05.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:55:05.893 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:55:05.942 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:05.943 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:06.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:06.005+0000 7f2ef4d74980 1 waiting for incremental sync to catch up: 2024-03-04T17:55:06.004 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.38.6, , ] 2024-03-04T17:55:06.004 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000004.45.10, , ] 2024-03-04T17:55:11.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:11.008+0000 7f2ef4d74980 1 bucket sync caught up with source: 2024-03-04T17:55:11.007 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000004.45.10, , ] 2024-03-04T17:55:11.007 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000004.45.10, , ] 2024-03-04T17:55:11.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:11.008+0000 7f2ef4d74980 0 bucket checkpoint complete 2024-03-04T17:55:11.045 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-23 2024-03-04T17:55:11.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:55:11.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:55:11.046 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:55:11.137 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:11.137 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:11.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:11.200+0000 7f907eca8980 1 waiting for incremental sync to catch up: 2024-03-04T17:55:11.198 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000003.49.13, , ] 2024-03-04T17:55:11.199 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000004.51.10, , ] 2024-03-04T17:55:16.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:16.204+0000 7f907eca8980 1 waiting for incremental sync to catch up: 2024-03-04T17:55:16.202 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000003.49.13, , ] 2024-03-04T17:55:16.202 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000004.51.10, , ] 2024-03-04T17:55:21.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:21.204+0000 7f907eca8980 1 waiting for incremental sync to catch up: 2024-03-04T17:55:21.204 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000003.49.13, , ] 2024-03-04T17:55:21.204 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000004.51.10, , ] 2024-03-04T17:55:26.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:26.208+0000 7f907eca8980 1 bucket sync caught up with source: 2024-03-04T17:55:26.205 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000004.51.10, , ] 2024-03-04T17:55:26.205 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000004.51.10, , ] 2024-03-04T17:55:26.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:26.208+0000 7f907eca8980 0 bucket checkpoint complete 2024-03-04T17:55:26.249 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-23 2024-03-04T17:55:26.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:55:26.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:55:26.249 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:55:26.301 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:26.301 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:26.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:55:26.370+0000 7f26ad77e980 1 bucket sync caught up with source: 2024-03-04T17:55:26.364 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000004.51.10, , ] 2024-03-04T17:55:26.364 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000004.51.10, , ] 2024-03-04T17:55:26.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:55:26.370+0000 7f26ad77e980 0 bucket checkpoint complete 2024-03-04T17:55:26.387 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-23 zones={test-zone2, test-zone1} 2024-03-04T17:55:26.387 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.387 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.388 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.388 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:55:26.388 DEBUG:boto:Data: 2024-03-04T17:55:26.388 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.388 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.388 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.388 DEBUG:boto:Params: {} 2024-03-04T17:55:26.388 DEBUG:boto:Token: None 2024-03-04T17:55:26.388 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.389 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DJfw+wNCfRgexOdEiYgrlaVQa/g= 2024-03-04T17:55:26.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DJfw+wNCfRgexOdEiYgrlaVQa/g=', 'Content-Length': '0'} 2024-03-04T17:55:26.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024575b63b504fdd5-0065e60b0e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.391 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.391 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.392 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.392 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:55:26.392 DEBUG:boto:Data: 2024-03-04T17:55:26.392 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.392 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.392 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.392 DEBUG:boto:Params: {} 2024-03-04T17:55:26.392 DEBUG:boto:Token: None 2024-03-04T17:55:26.392 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.392 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DJfw+wNCfRgexOdEiYgrlaVQa/g= 2024-03-04T17:55:26.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DJfw+wNCfRgexOdEiYgrlaVQa/g=', 'Content-Length': '0'} 2024-03-04T17:55:26.397 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006119617f0e3bd78d-0065e60b0e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.397 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:55:26.397 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.398 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.398 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.398 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.398 DEBUG:boto:Method: GET 2024-03-04T17:55:26.398 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.398 DEBUG:boto:Data: 2024-03-04T17:55:26.398 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.398 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.398 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.398 DEBUG:boto:Params: {} 2024-03-04T17:55:26.398 DEBUG:boto:Token: None 2024-03-04T17:55:26.399 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/?versions 2024-03-04T17:55:26.399 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw= 2024-03-04T17:55:26.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw=', 'Content-Length': '0'} 2024-03-04T17:55:26.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017da4b21998c364c-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.401 DEBUG:boto:b'ppfahn-23999falseobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCtrue2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:55:26.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:55:26.402 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:55:26.402 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.402 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.402 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.403 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.403 DEBUG:boto:Method: GET 2024-03-04T17:55:26.403 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.403 DEBUG:boto:Data: 2024-03-04T17:55:26.403 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.403 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.403 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.403 DEBUG:boto:Params: {} 2024-03-04T17:55:26.403 DEBUG:boto:Token: None 2024-03-04T17:55:26.403 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/?versions 2024-03-04T17:55:26.404 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw= 2024-03-04T17:55:26.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw=', 'Content-Length': '0'} 2024-03-04T17:55:26.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f395ea0420197d25-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.408 DEBUG:boto:b'ppfahn-23999falseobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCtrue2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:55:26.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:55:26.409 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.409 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.409 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.409 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.409 DEBUG:boto:Method: GET 2024-03-04T17:55:26.409 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.409 DEBUG:boto:Data: 2024-03-04T17:55:26.409 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.409 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.409 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.409 DEBUG:boto:Params: {} 2024-03-04T17:55:26.410 DEBUG:boto:Token: None 2024-03-04T17:55:26.410 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/?versions 2024-03-04T17:55:26.410 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw= 2024-03-04T17:55:26.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw=', 'Content-Length': '0'} 2024-03-04T17:55:26.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a354181cbea84275-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.412 DEBUG:boto:b'ppfahn-23999falseobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCtrue2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.413 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.413 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.413 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.413 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.413 DEBUG:boto:Method: GET 2024-03-04T17:55:26.413 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.413 DEBUG:boto:Data: 2024-03-04T17:55:26.413 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.414 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.414 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.414 DEBUG:boto:Params: {} 2024-03-04T17:55:26.414 DEBUG:boto:Token: None 2024-03-04T17:55:26.414 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/?versions 2024-03-04T17:55:26.414 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw= 2024-03-04T17:55:26.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw=', 'Content-Length': '0'} 2024-03-04T17:55:26.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb469ca8e49724d9-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.419 DEBUG:boto:b'ppfahn-23999falseobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCtrue2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:26.419 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.419 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.419 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.419 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.419 DEBUG:boto:Method: GET 2024-03-04T17:55:26.420 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.420 DEBUG:boto:Data: 2024-03-04T17:55:26.420 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.420 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.420 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.420 DEBUG:boto:Params: {} 2024-03-04T17:55:26.420 DEBUG:boto:Token: None 2024-03-04T17:55:26.420 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.420 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e22190110a701200-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.423 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.423 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.423 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.423 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.423 DEBUG:boto:Method: GET 2024-03-04T17:55:26.423 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.423 DEBUG:boto:Data: 2024-03-04T17:55:26.423 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.423 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.423 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.423 DEBUG:boto:Params: {} 2024-03-04T17:55:26.424 DEBUG:boto:Token: None 2024-03-04T17:55:26.424 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.424 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d6d6d39588621692-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.427 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.427 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.427 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.427 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.427 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.427 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.428 DEBUG:boto:Data: 2024-03-04T17:55:26.428 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.428 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.428 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.428 DEBUG:boto:Params: {} 2024-03-04T17:55:26.428 DEBUG:boto:Token: None 2024-03-04T17:55:26.428 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.428 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8bqXbgKud+qBHLLmvYC5adjw9Us= 2024-03-04T17:55:26.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8bqXbgKud+qBHLLmvYC5adjw9Us=', 'Content-Length': '0'} 2024-03-04T17:55:26.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ce367b19fcc14786-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.431 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.431 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.431 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.431 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.431 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.431 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.431 DEBUG:boto:Data: 2024-03-04T17:55:26.431 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.432 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.432 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.432 DEBUG:boto:Params: {} 2024-03-04T17:55:26.432 DEBUG:boto:Token: None 2024-03-04T17:55:26.432 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.432 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8bqXbgKud+qBHLLmvYC5adjw9Us= 2024-03-04T17:55:26.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8bqXbgKud+qBHLLmvYC5adjw9Us=', 'Content-Length': '0'} 2024-03-04T17:55:26.435 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006793eb19221172b6-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:26.435 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.435 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.435 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.435 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.435 DEBUG:boto:Method: GET 2024-03-04T17:55:26.435 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.436 DEBUG:boto:Data: 2024-03-04T17:55:26.436 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.436 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.436 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.436 DEBUG:boto:Params: {} 2024-03-04T17:55:26.436 DEBUG:boto:Token: None 2024-03-04T17:55:26.436 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.436 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dbe7037f4794b368-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.438 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.439 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.439 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.439 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.439 DEBUG:boto:Method: GET 2024-03-04T17:55:26.439 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.439 DEBUG:boto:Data: 2024-03-04T17:55:26.439 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.439 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.439 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.439 DEBUG:boto:Params: {} 2024-03-04T17:55:26.439 DEBUG:boto:Token: None 2024-03-04T17:55:26.440 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.440 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092010f5dbf2dec7a-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.443 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.443 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.444 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.444 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.444 DEBUG:boto:Method: GET 2024-03-04T17:55:26.444 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.444 DEBUG:boto:Data: 2024-03-04T17:55:26.444 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.444 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.444 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.444 DEBUG:boto:Params: {} 2024-03-04T17:55:26.444 DEBUG:boto:Token: None 2024-03-04T17:55:26.444 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.445 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb6fcba66393df5-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.488 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.488 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.489 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.489 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.489 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.489 DEBUG:boto:Method: GET 2024-03-04T17:55:26.489 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.489 DEBUG:boto:Data: 2024-03-04T17:55:26.489 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.489 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.489 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.489 DEBUG:boto:Params: {} 2024-03-04T17:55:26.490 DEBUG:boto:Token: None 2024-03-04T17:55:26.490 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.490 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062bdd3a9ae309905-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.494 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.494 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.494 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.495 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.495 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:55:26.495 DEBUG:boto:Data: 2024-03-04T17:55:26.495 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.495 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.495 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.495 DEBUG:boto:Params: {} 2024-03-04T17:55:26.496 DEBUG:boto:Token: None 2024-03-04T17:55:26.496 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj 2024-03-04T17:55:26.496 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4= 2024-03-04T17:55:26.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4=', 'Content-Length': '0'} 2024-03-04T17:55:26.500 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008a1557d8d623947-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.500 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.500 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.500 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.500 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:55:26.500 DEBUG:boto:Data: 2024-03-04T17:55:26.500 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.500 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.501 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.501 DEBUG:boto:Params: {} 2024-03-04T17:55:26.501 DEBUG:boto:Token: None 2024-03-04T17:55:26.501 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj 2024-03-04T17:55:26.501 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4= 2024-03-04T17:55:26.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4=', 'Content-Length': '0'} 2024-03-04T17:55:26.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af822747d5bbf407-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:26.506 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.506 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.506 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.506 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.506 DEBUG:boto:Method: GET 2024-03-04T17:55:26.506 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.506 DEBUG:boto:Data: 2024-03-04T17:55:26.506 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.507 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.507 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.507 DEBUG:boto:Params: {} 2024-03-04T17:55:26.507 DEBUG:boto:Token: None 2024-03-04T17:55:26.507 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.507 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000637eae4515cd5d3e-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.510 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.510 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.510 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.510 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.510 DEBUG:boto:Method: GET 2024-03-04T17:55:26.511 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.511 DEBUG:boto:Data: 2024-03-04T17:55:26.511 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.511 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.511 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.511 DEBUG:boto:Params: {} 2024-03-04T17:55:26.511 DEBUG:boto:Token: None 2024-03-04T17:55:26.511 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.511 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d03a12d5e8897b7-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.514 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.514 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.515 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.515 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.515 DEBUG:boto:Method: GET 2024-03-04T17:55:26.515 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.515 DEBUG:boto:Data: 2024-03-04T17:55:26.515 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.515 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.515 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.515 DEBUG:boto:Params: {} 2024-03-04T17:55:26.515 DEBUG:boto:Token: None 2024-03-04T17:55:26.516 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.516 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccb81bbee36e24b0-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.520 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.521 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.521 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.521 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.521 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.521 DEBUG:boto:Method: GET 2024-03-04T17:55:26.521 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.521 DEBUG:boto:Data: 2024-03-04T17:55:26.521 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.521 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.521 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.522 DEBUG:boto:Params: {} 2024-03-04T17:55:26.522 DEBUG:boto:Token: None 2024-03-04T17:55:26.522 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.522 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e02cc6023647faa8-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.526 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:26.527 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.527 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.527 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.527 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.527 DEBUG:boto:Method: GET 2024-03-04T17:55:26.527 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.527 DEBUG:boto:Data: 2024-03-04T17:55:26.528 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.528 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.528 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.528 DEBUG:boto:Params: {} 2024-03-04T17:55:26.528 DEBUG:boto:Token: None 2024-03-04T17:55:26.528 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.528 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo= 2024-03-04T17:55:26.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo=', 'Content-Length': '0'} 2024-03-04T17:55:26.531 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000010ddf4027406232c-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.531 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.531 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.532 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.532 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.532 DEBUG:boto:Method: GET 2024-03-04T17:55:26.532 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.532 DEBUG:boto:Data: 2024-03-04T17:55:26.532 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.532 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.532 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.532 DEBUG:boto:Params: {} 2024-03-04T17:55:26.532 DEBUG:boto:Token: None 2024-03-04T17:55:26.532 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.533 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo= 2024-03-04T17:55:26.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo=', 'Content-Length': '0'} 2024-03-04T17:55:26.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000217901f018f90501-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.536 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.539 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.539 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.539 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.540 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.540 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.540 DEBUG:boto:Data: 2024-03-04T17:55:26.540 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.540 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.540 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.540 DEBUG:boto:Params: {} 2024-03-04T17:55:26.540 DEBUG:boto:Token: None 2024-03-04T17:55:26.541 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.541 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:riUKHhInHTvsY1UZuab2K4gACL4= 2024-03-04T17:55:26.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:riUKHhInHTvsY1UZuab2K4gACL4=', 'Content-Length': '0'} 2024-03-04T17:55:26.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd521329f98b7ba5-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.544 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.544 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.544 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.544 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.544 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.545 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.545 DEBUG:boto:Data: 2024-03-04T17:55:26.545 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.545 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.545 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.545 DEBUG:boto:Params: {} 2024-03-04T17:55:26.545 DEBUG:boto:Token: None 2024-03-04T17:55:26.545 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.545 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:riUKHhInHTvsY1UZuab2K4gACL4= 2024-03-04T17:55:26.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:riUKHhInHTvsY1UZuab2K4gACL4=', 'Content-Length': '0'} 2024-03-04T17:55:26.548 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e35d5f2e80be3f0-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:26.549 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.549 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.549 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.549 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.549 DEBUG:boto:Method: GET 2024-03-04T17:55:26.549 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.549 DEBUG:boto:Data: 2024-03-04T17:55:26.549 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.549 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.550 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.550 DEBUG:boto:Params: {} 2024-03-04T17:55:26.550 DEBUG:boto:Token: None 2024-03-04T17:55:26.550 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.550 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo= 2024-03-04T17:55:26.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo=', 'Content-Length': '0'} 2024-03-04T17:55:26.553 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c7d69dcd96cd235f-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.553 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.553 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.554 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.554 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.554 DEBUG:boto:Method: GET 2024-03-04T17:55:26.554 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.554 DEBUG:boto:Data: 2024-03-04T17:55:26.554 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.554 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.554 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.554 DEBUG:boto:Params: {} 2024-03-04T17:55:26.554 DEBUG:boto:Token: None 2024-03-04T17:55:26.555 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.555 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo= 2024-03-04T17:55:26.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo=', 'Content-Length': '0'} 2024-03-04T17:55:26.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cf9b4687f7fd60c0-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.558 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.558 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.558 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.558 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.558 DEBUG:boto:Method: GET 2024-03-04T17:55:26.558 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.558 DEBUG:boto:Data: 2024-03-04T17:55:26.558 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.558 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.558 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.559 DEBUG:boto:Params: {} 2024-03-04T17:55:26.559 DEBUG:boto:Token: None 2024-03-04T17:55:26.559 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.559 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054851a148ad364b7-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.604 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.604 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.605 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.605 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.605 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.605 DEBUG:boto:Method: GET 2024-03-04T17:55:26.605 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.605 DEBUG:boto:Data: 2024-03-04T17:55:26.605 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.605 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.605 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.606 DEBUG:boto:Params: {} 2024-03-04T17:55:26.606 DEBUG:boto:Token: None 2024-03-04T17:55:26.606 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.606 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0a1787bef57ae55-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.612 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.612 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.612 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.612 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.612 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:55:26.613 DEBUG:boto:Data: 2024-03-04T17:55:26.613 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.613 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.613 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.613 DEBUG:boto:Params: {} 2024-03-04T17:55:26.613 DEBUG:boto:Token: None 2024-03-04T17:55:26.613 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj 2024-03-04T17:55:26.613 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4= 2024-03-04T17:55:26.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4=', 'Content-Length': '0'} 2024-03-04T17:55:26.617 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084ca6c7e780a6bd1-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.617 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.617 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.617 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.618 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:55:26.618 DEBUG:boto:Data: 2024-03-04T17:55:26.618 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.618 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.618 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.618 DEBUG:boto:Params: {} 2024-03-04T17:55:26.618 DEBUG:boto:Token: None 2024-03-04T17:55:26.618 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj 2024-03-04T17:55:26.618 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4= 2024-03-04T17:55:26.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4=', 'Content-Length': '0'} 2024-03-04T17:55:26.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000972b6c46eaf80820-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:26.622 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.622 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.623 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.623 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.623 DEBUG:boto:Method: GET 2024-03-04T17:55:26.623 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.623 DEBUG:boto:Data: 2024-03-04T17:55:26.623 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.623 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.623 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.623 DEBUG:boto:Params: {} 2024-03-04T17:55:26.623 DEBUG:boto:Token: None 2024-03-04T17:55:26.624 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.624 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005dd859f756950ee6-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.627 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.627 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.627 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.627 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.627 DEBUG:boto:Method: GET 2024-03-04T17:55:26.627 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.627 DEBUG:boto:Data: 2024-03-04T17:55:26.627 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.628 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.628 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.628 DEBUG:boto:Params: {} 2024-03-04T17:55:26.628 DEBUG:boto:Token: None 2024-03-04T17:55:26.628 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.628 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c94cb2266f589199-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.632 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.632 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.632 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.632 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.632 DEBUG:boto:Method: GET 2024-03-04T17:55:26.632 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.632 DEBUG:boto:Data: 2024-03-04T17:55:26.632 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.632 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:26.633 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.633 DEBUG:boto:Params: {} 2024-03-04T17:55:26.633 DEBUG:boto:Token: None 2024-03-04T17:55:26.633 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.633 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077bda85fa9fd8ed1-0065e60b0e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.638 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.638 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.638 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.638 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.638 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.638 DEBUG:boto:Method: GET 2024-03-04T17:55:26.638 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.639 DEBUG:boto:Data: 2024-03-04T17:55:26.639 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.639 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.639 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.639 DEBUG:boto:Params: {} 2024-03-04T17:55:26.639 DEBUG:boto:Token: None 2024-03-04T17:55:26.639 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.639 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ab7296eb3278d34-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.643 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.644 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-23 zones={test-zone2, test-zone1} 2024-03-04T17:55:26.644 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-23 zones={test-zone3, test-zone1} 2024-03-04T17:55:26.644 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.644 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.644 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.644 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:55:26.644 DEBUG:boto:Data: 2024-03-04T17:55:26.645 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.645 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.645 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.645 DEBUG:boto:Params: {} 2024-03-04T17:55:26.645 DEBUG:boto:Token: None 2024-03-04T17:55:26.645 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.645 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DJfw+wNCfRgexOdEiYgrlaVQa/g= 2024-03-04T17:55:26.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DJfw+wNCfRgexOdEiYgrlaVQa/g=', 'Content-Length': '0'} 2024-03-04T17:55:26.649 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fa9bb7d6203c9ae-0065e60b0e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.649 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.649 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.650 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.650 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:55:26.650 DEBUG:boto:Data: 2024-03-04T17:55:26.650 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.650 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.650 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.650 DEBUG:boto:Params: {} 2024-03-04T17:55:26.650 DEBUG:boto:Token: None 2024-03-04T17:55:26.650 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.650 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DJfw+wNCfRgexOdEiYgrlaVQa/g= 2024-03-04T17:55:26.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DJfw+wNCfRgexOdEiYgrlaVQa/g=', 'Content-Length': '0'} 2024-03-04T17:55:26.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccd23956c81f3915-0065e60b0e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.654 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:55:26.654 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.655 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.655 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.655 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.655 DEBUG:boto:Method: GET 2024-03-04T17:55:26.655 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.655 DEBUG:boto:Data: 2024-03-04T17:55:26.655 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.655 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.655 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.655 DEBUG:boto:Params: {} 2024-03-04T17:55:26.656 DEBUG:boto:Token: None 2024-03-04T17:55:26.656 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/?versions 2024-03-04T17:55:26.656 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw= 2024-03-04T17:55:26.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw=', 'Content-Length': '0'} 2024-03-04T17:55:26.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abaf2dfe452f02b6-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.660 DEBUG:boto:b'ppfahn-23999falseobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCtrue2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:55:26.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:55:26.661 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:55:26.661 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.661 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.661 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.661 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.662 DEBUG:boto:Method: GET 2024-03-04T17:55:26.662 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.662 DEBUG:boto:Data: 2024-03-04T17:55:26.662 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.662 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.662 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.662 DEBUG:boto:Params: {} 2024-03-04T17:55:26.662 DEBUG:boto:Token: None 2024-03-04T17:55:26.662 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/?versions 2024-03-04T17:55:26.663 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw= 2024-03-04T17:55:26.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw=', 'Content-Length': '0'} 2024-03-04T17:55:26.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008137418b20856343-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.708 DEBUG:boto:b'ppfahn-23999falseobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCtrue2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.708 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:55:26.708 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:55:26.709 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.709 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.709 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.709 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.709 DEBUG:boto:Method: GET 2024-03-04T17:55:26.709 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.709 DEBUG:boto:Data: 2024-03-04T17:55:26.709 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.709 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.710 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.710 DEBUG:boto:Params: {} 2024-03-04T17:55:26.710 DEBUG:boto:Token: None 2024-03-04T17:55:26.710 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/?versions 2024-03-04T17:55:26.710 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw= 2024-03-04T17:55:26.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw=', 'Content-Length': '0'} 2024-03-04T17:55:26.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e9c1edf5cef04ce-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.715 DEBUG:boto:b'ppfahn-23999falseobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCtrue2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.715 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.715 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.715 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.715 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.716 DEBUG:boto:Method: GET 2024-03-04T17:55:26.716 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:26.716 DEBUG:boto:Data: 2024-03-04T17:55:26.716 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.716 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.716 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.716 DEBUG:boto:Params: {} 2024-03-04T17:55:26.716 DEBUG:boto:Token: None 2024-03-04T17:55:26.716 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/?versions 2024-03-04T17:55:26.717 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw= 2024-03-04T17:55:26.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5YnMDnwnLusTr8CIRfuUouokEIw=', 'Content-Length': '0'} 2024-03-04T17:55:26.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000872d6b76d12109f3-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.721 DEBUG:boto:b'ppfahn-23999falseobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCtrue2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:26.721 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.721 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.722 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.722 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.722 DEBUG:boto:Method: GET 2024-03-04T17:55:26.722 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.722 DEBUG:boto:Data: 2024-03-04T17:55:26.722 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.722 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.722 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.722 DEBUG:boto:Params: {} 2024-03-04T17:55:26.722 DEBUG:boto:Token: None 2024-03-04T17:55:26.723 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.723 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008a168f6cf5ec95b1-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.726 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.726 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.726 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.726 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.726 DEBUG:boto:Method: GET 2024-03-04T17:55:26.726 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.726 DEBUG:boto:Data: 2024-03-04T17:55:26.727 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.727 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.727 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.727 DEBUG:boto:Params: {} 2024-03-04T17:55:26.727 DEBUG:boto:Token: None 2024-03-04T17:55:26.727 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.727 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.731 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009aa44764c407bc5c-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.731 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.731 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.731 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.731 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.731 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.731 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.731 DEBUG:boto:Data: 2024-03-04T17:55:26.731 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.732 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.732 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.732 DEBUG:boto:Params: {} 2024-03-04T17:55:26.732 DEBUG:boto:Token: None 2024-03-04T17:55:26.732 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.732 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8bqXbgKud+qBHLLmvYC5adjw9Us= 2024-03-04T17:55:26.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8bqXbgKud+qBHLLmvYC5adjw9Us=', 'Content-Length': '0'} 2024-03-04T17:55:26.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d09350ffad0d5a5-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.735 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.735 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.735 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.735 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.736 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.736 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.736 DEBUG:boto:Data: 2024-03-04T17:55:26.736 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.736 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.736 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.736 DEBUG:boto:Params: {} 2024-03-04T17:55:26.736 DEBUG:boto:Token: None 2024-03-04T17:55:26.736 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.736 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8bqXbgKud+qBHLLmvYC5adjw9Us= 2024-03-04T17:55:26.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8bqXbgKud+qBHLLmvYC5adjw9Us=', 'Content-Length': '0'} 2024-03-04T17:55:26.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba5f4589a0228440-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:26.739 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.740 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.740 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.740 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.740 DEBUG:boto:Method: GET 2024-03-04T17:55:26.740 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.740 DEBUG:boto:Data: 2024-03-04T17:55:26.740 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.740 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.740 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.740 DEBUG:boto:Params: {} 2024-03-04T17:55:26.740 DEBUG:boto:Token: None 2024-03-04T17:55:26.741 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.741 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cace07e30d9ad121-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.744 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.744 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.744 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.744 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.744 DEBUG:boto:Method: GET 2024-03-04T17:55:26.744 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.744 DEBUG:boto:Data: 2024-03-04T17:55:26.744 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.744 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.745 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.745 DEBUG:boto:Params: {} 2024-03-04T17:55:26.745 DEBUG:boto:Token: None 2024-03-04T17:55:26.745 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.745 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.748 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001586d95a3a347b64-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.749 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.749 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.749 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.749 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.749 DEBUG:boto:Method: GET 2024-03-04T17:55:26.749 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.749 DEBUG:boto:Data: 2024-03-04T17:55:26.749 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.750 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.750 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.750 DEBUG:boto:Params: {} 2024-03-04T17:55:26.750 DEBUG:boto:Token: None 2024-03-04T17:55:26.750 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.750 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6e1863f44e615a2-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.796 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.796 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.796 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.797 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.797 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.797 DEBUG:boto:Method: GET 2024-03-04T17:55:26.797 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.797 DEBUG:boto:Data: 2024-03-04T17:55:26.797 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.797 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.797 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.797 DEBUG:boto:Params: {} 2024-03-04T17:55:26.797 DEBUG:boto:Token: None 2024-03-04T17:55:26.798 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.798 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000225112a267f8ef52-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.802 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.802 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.802 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.802 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.803 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:55:26.803 DEBUG:boto:Data: 2024-03-04T17:55:26.803 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.803 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.803 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.803 DEBUG:boto:Params: {} 2024-03-04T17:55:26.803 DEBUG:boto:Token: None 2024-03-04T17:55:26.803 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj 2024-03-04T17:55:26.803 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4= 2024-03-04T17:55:26.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4=', 'Content-Length': '0'} 2024-03-04T17:55:26.807 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016e95dd80a7b0196-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.808 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.808 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.808 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.808 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:55:26.808 DEBUG:boto:Data: 2024-03-04T17:55:26.808 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.808 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.808 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.808 DEBUG:boto:Params: {} 2024-03-04T17:55:26.809 DEBUG:boto:Token: None 2024-03-04T17:55:26.809 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj 2024-03-04T17:55:26.809 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4= 2024-03-04T17:55:26.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4=', 'Content-Length': '0'} 2024-03-04T17:55:26.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bfa1b6812525d7b0-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:26.813 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.813 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.813 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.813 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.813 DEBUG:boto:Method: GET 2024-03-04T17:55:26.813 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.813 DEBUG:boto:Data: 2024-03-04T17:55:26.814 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.814 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.814 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.814 DEBUG:boto:Params: {} 2024-03-04T17:55:26.814 DEBUG:boto:Token: None 2024-03-04T17:55:26.814 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.814 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000225f97adda404734-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.817 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.818 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.818 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.818 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.818 DEBUG:boto:Method: GET 2024-03-04T17:55:26.818 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.818 DEBUG:boto:Data: 2024-03-04T17:55:26.818 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.818 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.818 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.818 DEBUG:boto:Params: {} 2024-03-04T17:55:26.819 DEBUG:boto:Token: None 2024-03-04T17:55:26.819 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.819 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004693ffb622a35b99-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.822 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.822 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.822 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.822 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.822 DEBUG:boto:Method: GET 2024-03-04T17:55:26.822 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.822 DEBUG:boto:Data: 2024-03-04T17:55:26.822 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.823 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.823 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.823 DEBUG:boto:Params: {} 2024-03-04T17:55:26.823 DEBUG:boto:Token: None 2024-03-04T17:55:26.823 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.823 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000605bda32db36952d-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.828 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.828 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.828 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.828 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.828 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.828 DEBUG:boto:Method: GET 2024-03-04T17:55:26.829 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.829 DEBUG:boto:Data: 2024-03-04T17:55:26.829 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.829 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.829 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.829 DEBUG:boto:Params: {} 2024-03-04T17:55:26.829 DEBUG:boto:Token: None 2024-03-04T17:55:26.829 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.830 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000610f2589ce97e20b-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.876 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:26.877 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.877 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.877 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.877 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.877 DEBUG:boto:Method: GET 2024-03-04T17:55:26.877 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.877 DEBUG:boto:Data: 2024-03-04T17:55:26.877 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.877 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.877 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.877 DEBUG:boto:Params: {} 2024-03-04T17:55:26.878 DEBUG:boto:Token: None 2024-03-04T17:55:26.878 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.878 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo= 2024-03-04T17:55:26.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo=', 'Content-Length': '0'} 2024-03-04T17:55:26.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c993a3d451863688-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.882 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.882 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.882 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.882 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.882 DEBUG:boto:Method: GET 2024-03-04T17:55:26.882 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.882 DEBUG:boto:Data: 2024-03-04T17:55:26.882 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.883 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.883 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.883 DEBUG:boto:Params: {} 2024-03-04T17:55:26.883 DEBUG:boto:Token: None 2024-03-04T17:55:26.883 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.883 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo= 2024-03-04T17:55:26.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo=', 'Content-Length': '0'} 2024-03-04T17:55:26.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006fca2a37c40c9719-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.886 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.886 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.886 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.886 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.886 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.887 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.887 DEBUG:boto:Data: 2024-03-04T17:55:26.887 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.887 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.887 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.887 DEBUG:boto:Params: {} 2024-03-04T17:55:26.887 DEBUG:boto:Token: None 2024-03-04T17:55:26.887 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.887 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:riUKHhInHTvsY1UZuab2K4gACL4= 2024-03-04T17:55:26.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:riUKHhInHTvsY1UZuab2K4gACL4=', 'Content-Length': '0'} 2024-03-04T17:55:26.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dc64e2f5d550efaa-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.891 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.891 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.891 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.891 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.891 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.891 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.891 DEBUG:boto:Data: 2024-03-04T17:55:26.891 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.891 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.891 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.892 DEBUG:boto:Params: {} 2024-03-04T17:55:26.892 DEBUG:boto:Token: None 2024-03-04T17:55:26.892 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.892 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:riUKHhInHTvsY1UZuab2K4gACL4= 2024-03-04T17:55:26.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:riUKHhInHTvsY1UZuab2K4gACL4=', 'Content-Length': '0'} 2024-03-04T17:55:26.895 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b737f36217ba4973-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:26.895 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.895 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.895 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.895 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.896 DEBUG:boto:Method: GET 2024-03-04T17:55:26.896 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.896 DEBUG:boto:Data: 2024-03-04T17:55:26.896 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.896 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.896 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.896 DEBUG:boto:Params: {} 2024-03-04T17:55:26.896 DEBUG:boto:Token: None 2024-03-04T17:55:26.896 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.897 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo= 2024-03-04T17:55:26.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo=', 'Content-Length': '0'} 2024-03-04T17:55:26.900 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002dc2d681ee1739d0-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.900 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.900 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.900 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.900 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.900 DEBUG:boto:Method: GET 2024-03-04T17:55:26.901 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.901 DEBUG:boto:Data: 2024-03-04T17:55:26.901 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.901 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.901 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.901 DEBUG:boto:Params: {} 2024-03-04T17:55:26.901 DEBUG:boto:Token: None 2024-03-04T17:55:26.901 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:26.901 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo= 2024-03-04T17:55:26.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:v1XR4QfDklVhrwn7VprQ5zV1gLo=', 'Content-Length': '0'} 2024-03-04T17:55:26.904 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a36508136e58c347-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.905 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.905 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.905 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.905 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.905 DEBUG:boto:Method: GET 2024-03-04T17:55:26.905 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.905 DEBUG:boto:Data: 2024-03-04T17:55:26.905 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.905 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.906 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.906 DEBUG:boto:Params: {} 2024-03-04T17:55:26.906 DEBUG:boto:Token: None 2024-03-04T17:55:26.906 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.906 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd7e28ce555d4d58-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.952 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.952 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.953 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.953 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.953 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.953 DEBUG:boto:Method: GET 2024-03-04T17:55:26.953 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.953 DEBUG:boto:Data: 2024-03-04T17:55:26.953 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.953 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.953 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.953 DEBUG:boto:Params: {} 2024-03-04T17:55:26.954 DEBUG:boto:Token: None 2024-03-04T17:55:26.954 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.954 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000138411f8395fb100-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.959 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.959 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.959 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.959 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.960 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:55:26.960 DEBUG:boto:Data: 2024-03-04T17:55:26.960 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.960 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.960 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.960 DEBUG:boto:Params: {} 2024-03-04T17:55:26.960 DEBUG:boto:Token: None 2024-03-04T17:55:26.960 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj 2024-03-04T17:55:26.960 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4= 2024-03-04T17:55:26.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4=', 'Content-Length': '0'} 2024-03-04T17:55:26.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003d14b5e0410f3658-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.965 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.965 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.965 DEBUG:boto:Method: HEAD 2024-03-04T17:55:26.965 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:55:26.965 DEBUG:boto:Data: 2024-03-04T17:55:26.965 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.966 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.966 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.966 DEBUG:boto:Params: {} 2024-03-04T17:55:26.966 DEBUG:boto:Token: None 2024-03-04T17:55:26.966 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj 2024-03-04T17:55:26.966 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4= 2024-03-04T17:55:26.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NFasbMokzQfRuV8aa5pp/2jTYk4=', 'Content-Length': '0'} 2024-03-04T17:55:26.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004699b4db0ffba751-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:26.970 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.970 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.970 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.971 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.971 DEBUG:boto:Method: GET 2024-03-04T17:55:26.971 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.971 DEBUG:boto:Data: 2024-03-04T17:55:26.971 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.971 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.971 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.971 DEBUG:boto:Params: {} 2024-03-04T17:55:26.971 DEBUG:boto:Token: None 2024-03-04T17:55:26.972 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.972 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e6a1c5b572fc844-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.975 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:26.975 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:26.975 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.975 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.975 DEBUG:boto:Method: GET 2024-03-04T17:55:26.976 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.976 DEBUG:boto:Data: 2024-03-04T17:55:26.976 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.976 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.976 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.976 DEBUG:boto:Params: {} 2024-03-04T17:55:26.976 DEBUG:boto:Token: None 2024-03-04T17:55:26.976 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:26.976 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk= 2024-03-04T17:55:26.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yHdOUS7Z9C/OJn6MqReGnVLhvpk=', 'Content-Length': '0'} 2024-03-04T17:55:26.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036671fbd7f0de060-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.979 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.980 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.980 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.980 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.980 DEBUG:boto:Method: GET 2024-03-04T17:55:26.980 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.980 DEBUG:boto:Data: 2024-03-04T17:55:26.980 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.980 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:26.980 DEBUG:boto:Port: 8001 2024-03-04T17:55:26.980 DEBUG:boto:Params: {} 2024-03-04T17:55:26.981 DEBUG:boto:Token: None 2024-03-04T17:55:26.981 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.981 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca3c41c64b16b529-0065e60b0e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:26.985 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:26.985 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:26.986 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:26.986 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.986 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:26.986 DEBUG:boto:Method: GET 2024-03-04T17:55:26.986 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:26.986 DEBUG:boto:Data: 2024-03-04T17:55:26.986 DEBUG:boto:Headers: {} 2024-03-04T17:55:26.986 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:26.986 DEBUG:boto:Port: 8000 2024-03-04T17:55:26.986 DEBUG:boto:Params: {} 2024-03-04T17:55:26.987 DEBUG:boto:Token: None 2024-03-04T17:55:26.987 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:26 GMT /ppfahn-23/ 2024-03-04T17:55:26.987 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM= 2024-03-04T17:55:26.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:26 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VE1swzZqZiGEKuTtMlKm5QPudM=', 'Content-Length': '0'} 2024-03-04T17:55:26.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e696587796c72fb-0065e60b0e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.036 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:27.036 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-23 zones={test-zone3, test-zone1} 2024-03-04T17:55:27.037 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-23 zones={test-zone3, test-zone2} 2024-03-04T17:55:27.037 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.037 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.037 DEBUG:boto:Method: HEAD 2024-03-04T17:55:27.037 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:55:27.037 DEBUG:boto:Data: 2024-03-04T17:55:27.037 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.037 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.037 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.037 DEBUG:boto:Params: {} 2024-03-04T17:55:27.038 DEBUG:boto:Token: None 2024-03-04T17:55:27.038 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/ 2024-03-04T17:55:27.038 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6f/EOLHDyE4p+UDlAAZQ94HgkGs= 2024-03-04T17:55:27.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6f/EOLHDyE4p+UDlAAZQ94HgkGs=', 'Content-Length': '0'} 2024-03-04T17:55:27.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006500f4b8af82d6ad-0065e60b0f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.042 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.043 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.043 DEBUG:boto:Method: HEAD 2024-03-04T17:55:27.043 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:55:27.043 DEBUG:boto:Data: 2024-03-04T17:55:27.043 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.043 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.043 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.043 DEBUG:boto:Params: {} 2024-03-04T17:55:27.043 DEBUG:boto:Token: None 2024-03-04T17:55:27.043 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/ 2024-03-04T17:55:27.044 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6f/EOLHDyE4p+UDlAAZQ94HgkGs= 2024-03-04T17:55:27.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6f/EOLHDyE4p+UDlAAZQ94HgkGs=', 'Content-Length': '0'} 2024-03-04T17:55:27.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ba55bab3bb5b07e-0065e60b0f-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.048 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:55:27.048 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.048 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.048 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:27.049 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:27.049 DEBUG:boto:Method: GET 2024-03-04T17:55:27.049 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:27.049 DEBUG:boto:Data: 2024-03-04T17:55:27.049 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.049 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.049 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.049 DEBUG:boto:Params: {} 2024-03-04T17:55:27.049 DEBUG:boto:Token: None 2024-03-04T17:55:27.049 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/?versions 2024-03-04T17:55:27.050 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Xili7M8f20dCZNyZIx5uD2qprNY= 2024-03-04T17:55:27.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Xili7M8f20dCZNyZIx5uD2qprNY=', 'Content-Length': '0'} 2024-03-04T17:55:27.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029b2b1b1fe962185-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.100 DEBUG:boto:b'ppfahn-23999falseobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCtrue2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:27.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:55:27.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:55:27.101 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:55:27.101 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.101 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.101 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:27.101 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:27.102 DEBUG:boto:Method: GET 2024-03-04T17:55:27.102 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:27.102 DEBUG:boto:Data: 2024-03-04T17:55:27.102 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.102 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.102 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.102 DEBUG:boto:Params: {} 2024-03-04T17:55:27.102 DEBUG:boto:Token: None 2024-03-04T17:55:27.102 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/?versions 2024-03-04T17:55:27.103 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Xili7M8f20dCZNyZIx5uD2qprNY= 2024-03-04T17:55:27.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Xili7M8f20dCZNyZIx5uD2qprNY=', 'Content-Length': '0'} 2024-03-04T17:55:27.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004946b02b5a963f2f-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.107 DEBUG:boto:b'ppfahn-23999falseobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCtrue2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:27.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:55:27.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:55:27.108 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.108 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.108 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:27.108 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:27.108 DEBUG:boto:Method: GET 2024-03-04T17:55:27.108 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:27.108 DEBUG:boto:Data: 2024-03-04T17:55:27.108 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.108 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.108 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.109 DEBUG:boto:Params: {} 2024-03-04T17:55:27.109 DEBUG:boto:Token: None 2024-03-04T17:55:27.109 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/?versions 2024-03-04T17:55:27.109 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Xili7M8f20dCZNyZIx5uD2qprNY= 2024-03-04T17:55:27.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Xili7M8f20dCZNyZIx5uD2qprNY=', 'Content-Length': '0'} 2024-03-04T17:55:27.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cadb156d3f4e5912-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.114 DEBUG:boto:b'ppfahn-23999falseobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCtrue2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:27.114 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.114 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.114 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:27.114 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:27.115 DEBUG:boto:Method: GET 2024-03-04T17:55:27.115 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:55:27.115 DEBUG:boto:Data: 2024-03-04T17:55:27.115 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.115 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.115 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.115 DEBUG:boto:Params: {} 2024-03-04T17:55:27.115 DEBUG:boto:Token: None 2024-03-04T17:55:27.115 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/?versions 2024-03-04T17:55:27.116 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Xili7M8f20dCZNyZIx5uD2qprNY= 2024-03-04T17:55:27.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Xili7M8f20dCZNyZIx5uD2qprNY=', 'Content-Length': '0'} 2024-03-04T17:55:27.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000115f64e85636782d-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.168 DEBUG:boto:b'ppfahn-23999falseobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCtrue2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T17:53:58.226Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:27.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:27.169 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.169 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.169 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.169 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.169 DEBUG:boto:Method: GET 2024-03-04T17:55:27.169 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.169 DEBUG:boto:Data: 2024-03-04T17:55:27.169 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.170 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.170 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.170 DEBUG:boto:Params: {} 2024-03-04T17:55:27.170 DEBUG:boto:Token: None 2024-03-04T17:55:27.170 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.170 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc= 2024-03-04T17:55:27.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc=', 'Content-Length': '0'} 2024-03-04T17:55:27.173 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054c4272de0c13ad7-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.173 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.173 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.174 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.174 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.174 DEBUG:boto:Method: GET 2024-03-04T17:55:27.174 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.174 DEBUG:boto:Data: 2024-03-04T17:55:27.174 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.174 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.174 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.174 DEBUG:boto:Params: {} 2024-03-04T17:55:27.174 DEBUG:boto:Token: None 2024-03-04T17:55:27.175 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.175 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc= 2024-03-04T17:55:27.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc=', 'Content-Length': '0'} 2024-03-04T17:55:27.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002753a074a948aacf-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.178 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.178 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.178 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.178 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.178 DEBUG:boto:Method: HEAD 2024-03-04T17:55:27.178 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.178 DEBUG:boto:Data: 2024-03-04T17:55:27.179 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.179 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.179 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.179 DEBUG:boto:Params: {} 2024-03-04T17:55:27.179 DEBUG:boto:Token: None 2024-03-04T17:55:27.179 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.179 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iTEt+Y539KbtwxhskVPake/1oR8= 2024-03-04T17:55:27.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iTEt+Y539KbtwxhskVPake/1oR8=', 'Content-Length': '0'} 2024-03-04T17:55:27.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002290dbec9e6a3e9d-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.182 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.182 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.182 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.182 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.183 DEBUG:boto:Method: HEAD 2024-03-04T17:55:27.183 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.183 DEBUG:boto:Data: 2024-03-04T17:55:27.183 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.183 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.183 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.183 DEBUG:boto:Params: {} 2024-03-04T17:55:27.183 DEBUG:boto:Token: None 2024-03-04T17:55:27.183 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.183 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iTEt+Y539KbtwxhskVPake/1oR8= 2024-03-04T17:55:27.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iTEt+Y539KbtwxhskVPake/1oR8=', 'Content-Length': '0'} 2024-03-04T17:55:27.186 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000061ac867104ac5c18-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:27.186 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.187 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.187 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.187 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.187 DEBUG:boto:Method: GET 2024-03-04T17:55:27.187 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.187 DEBUG:boto:Data: 2024-03-04T17:55:27.187 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.187 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.187 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.187 DEBUG:boto:Params: {} 2024-03-04T17:55:27.187 DEBUG:boto:Token: None 2024-03-04T17:55:27.188 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.188 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc= 2024-03-04T17:55:27.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc=', 'Content-Length': '0'} 2024-03-04T17:55:27.191 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009be67888004edac4-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.191 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.191 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.191 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.191 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.191 DEBUG:boto:Method: GET 2024-03-04T17:55:27.191 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.191 DEBUG:boto:Data: 2024-03-04T17:55:27.192 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.192 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.192 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.192 DEBUG:boto:Params: {} 2024-03-04T17:55:27.192 DEBUG:boto:Token: None 2024-03-04T17:55:27.192 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.192 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc= 2024-03-04T17:55:27.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc=', 'Content-Length': '0'} 2024-03-04T17:55:27.195 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bbc2b8faeaf4b756-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.195 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.195 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.195 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.196 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.196 DEBUG:boto:Method: GET 2024-03-04T17:55:27.196 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:27.196 DEBUG:boto:Data: 2024-03-04T17:55:27.196 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.196 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.196 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.196 DEBUG:boto:Params: {} 2024-03-04T17:55:27.196 DEBUG:boto:Token: None 2024-03-04T17:55:27.196 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/ 2024-03-04T17:55:27.197 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc= 2024-03-04T17:55:27.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc=', 'Content-Length': '0'} 2024-03-04T17:55:27.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e15fe33536ff85a1-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.201 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:27.201 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.201 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.202 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.202 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.202 DEBUG:boto:Method: GET 2024-03-04T17:55:27.202 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:27.202 DEBUG:boto:Data: 2024-03-04T17:55:27.202 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.202 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.202 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.202 DEBUG:boto:Params: {} 2024-03-04T17:55:27.202 DEBUG:boto:Token: None 2024-03-04T17:55:27.203 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/ 2024-03-04T17:55:27.203 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc= 2024-03-04T17:55:27.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc=', 'Content-Length': '0'} 2024-03-04T17:55:27.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd9bb3727bd94cde-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.207 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:27.207 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.207 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.207 DEBUG:boto:Method: HEAD 2024-03-04T17:55:27.207 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:55:27.207 DEBUG:boto:Data: 2024-03-04T17:55:27.208 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.208 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.208 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.208 DEBUG:boto:Params: {} 2024-03-04T17:55:27.208 DEBUG:boto:Token: None 2024-03-04T17:55:27.208 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj 2024-03-04T17:55:27.208 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G3H796KIfaZ+iT+3EjZHiDV7iLQ= 2024-03-04T17:55:27.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G3H796KIfaZ+iT+3EjZHiDV7iLQ=', 'Content-Length': '0'} 2024-03-04T17:55:27.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d5f410bcbd20f995-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.212 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.212 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.212 DEBUG:boto:Method: HEAD 2024-03-04T17:55:27.212 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:55:27.212 DEBUG:boto:Data: 2024-03-04T17:55:27.213 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.213 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.213 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.213 DEBUG:boto:Params: {} 2024-03-04T17:55:27.213 DEBUG:boto:Token: None 2024-03-04T17:55:27.213 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj 2024-03-04T17:55:27.213 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G3H796KIfaZ+iT+3EjZHiDV7iLQ= 2024-03-04T17:55:27.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G3H796KIfaZ+iT+3EjZHiDV7iLQ=', 'Content-Length': '0'} 2024-03-04T17:55:27.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f1f351817315491-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:27.218 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.218 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.218 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.218 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.219 DEBUG:boto:Method: GET 2024-03-04T17:55:27.219 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.219 DEBUG:boto:Data: 2024-03-04T17:55:27.219 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.219 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.219 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.219 DEBUG:boto:Params: {} 2024-03-04T17:55:27.219 DEBUG:boto:Token: None 2024-03-04T17:55:27.219 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.219 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc= 2024-03-04T17:55:27.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc=', 'Content-Length': '0'} 2024-03-04T17:55:27.222 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000271c5dd6d2388eea-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.222 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.223 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.223 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.223 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.223 DEBUG:boto:Method: GET 2024-03-04T17:55:27.223 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.223 DEBUG:boto:Data: 2024-03-04T17:55:27.223 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.223 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.223 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.223 DEBUG:boto:Params: {} 2024-03-04T17:55:27.223 DEBUG:boto:Token: None 2024-03-04T17:55:27.224 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.224 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc= 2024-03-04T17:55:27.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc=', 'Content-Length': '0'} 2024-03-04T17:55:27.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000425b78fbfd5b3781-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.227 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.227 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.227 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.227 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.227 DEBUG:boto:Method: GET 2024-03-04T17:55:27.227 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:27.227 DEBUG:boto:Data: 2024-03-04T17:55:27.227 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.228 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.228 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.228 DEBUG:boto:Params: {} 2024-03-04T17:55:27.228 DEBUG:boto:Token: None 2024-03-04T17:55:27.228 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/ 2024-03-04T17:55:27.228 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc= 2024-03-04T17:55:27.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc=', 'Content-Length': '0'} 2024-03-04T17:55:27.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056d6557a47b3c494-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.276 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:27.277 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.277 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.277 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.277 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.277 DEBUG:boto:Method: GET 2024-03-04T17:55:27.277 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:27.277 DEBUG:boto:Data: 2024-03-04T17:55:27.277 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.277 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.277 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.277 DEBUG:boto:Params: {} 2024-03-04T17:55:27.278 DEBUG:boto:Token: None 2024-03-04T17:55:27.278 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/ 2024-03-04T17:55:27.278 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc= 2024-03-04T17:55:27.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc=', 'Content-Length': '0'} 2024-03-04T17:55:27.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000515b87b95aecd10-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.282 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:27.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:27.283 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.283 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.283 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.283 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.283 DEBUG:boto:Method: GET 2024-03-04T17:55:27.283 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.283 DEBUG:boto:Data: 2024-03-04T17:55:27.284 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.284 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.284 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.284 DEBUG:boto:Params: {} 2024-03-04T17:55:27.284 DEBUG:boto:Token: None 2024-03-04T17:55:27.284 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.284 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XPByp0Hwq9ViIJJdsr2ZL2ij+II= 2024-03-04T17:55:27.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XPByp0Hwq9ViIJJdsr2ZL2ij+II=', 'Content-Length': '0'} 2024-03-04T17:55:27.287 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c4f47dbb8b06a536-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.288 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.288 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.288 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.288 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.288 DEBUG:boto:Method: GET 2024-03-04T17:55:27.288 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.288 DEBUG:boto:Data: 2024-03-04T17:55:27.288 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.288 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.288 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.289 DEBUG:boto:Params: {} 2024-03-04T17:55:27.289 DEBUG:boto:Token: None 2024-03-04T17:55:27.289 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.289 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XPByp0Hwq9ViIJJdsr2ZL2ij+II= 2024-03-04T17:55:27.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XPByp0Hwq9ViIJJdsr2ZL2ij+II=', 'Content-Length': '0'} 2024-03-04T17:55:27.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a490139f4bb6560-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.292 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.292 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.292 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.292 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.292 DEBUG:boto:Method: HEAD 2024-03-04T17:55:27.292 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.292 DEBUG:boto:Data: 2024-03-04T17:55:27.293 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.293 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.293 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.293 DEBUG:boto:Params: {} 2024-03-04T17:55:27.293 DEBUG:boto:Token: None 2024-03-04T17:55:27.293 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.293 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cV3O39B4eZyvE65T93lxZDa91UQ= 2024-03-04T17:55:27.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cV3O39B4eZyvE65T93lxZDa91UQ=', 'Content-Length': '0'} 2024-03-04T17:55:27.296 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000030ed498591374692-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.296 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.296 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.296 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.297 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.297 DEBUG:boto:Method: HEAD 2024-03-04T17:55:27.297 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.297 DEBUG:boto:Data: 2024-03-04T17:55:27.297 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.297 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.297 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.297 DEBUG:boto:Params: {} 2024-03-04T17:55:27.297 DEBUG:boto:Token: None 2024-03-04T17:55:27.297 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.298 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cV3O39B4eZyvE65T93lxZDa91UQ= 2024-03-04T17:55:27.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cV3O39B4eZyvE65T93lxZDa91UQ=', 'Content-Length': '0'} 2024-03-04T17:55:27.300 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000331c5f6acb080da7-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:27.301 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.301 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.301 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.301 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.301 DEBUG:boto:Method: GET 2024-03-04T17:55:27.301 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.301 DEBUG:boto:Data: 2024-03-04T17:55:27.301 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.301 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.301 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.302 DEBUG:boto:Params: {} 2024-03-04T17:55:27.302 DEBUG:boto:Token: None 2024-03-04T17:55:27.302 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.302 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XPByp0Hwq9ViIJJdsr2ZL2ij+II= 2024-03-04T17:55:27.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XPByp0Hwq9ViIJJdsr2ZL2ij+II=', 'Content-Length': '0'} 2024-03-04T17:55:27.305 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f06d99c99564597f-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.306 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.306 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.306 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.306 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.306 DEBUG:boto:Method: GET 2024-03-04T17:55:27.306 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.306 DEBUG:boto:Data: 2024-03-04T17:55:27.306 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.306 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.306 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.306 DEBUG:boto:Params: {} 2024-03-04T17:55:27.307 DEBUG:boto:Token: None 2024-03-04T17:55:27.307 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:55:27.307 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XPByp0Hwq9ViIJJdsr2ZL2ij+II= 2024-03-04T17:55:27.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XPByp0Hwq9ViIJJdsr2ZL2ij+II=', 'Content-Length': '0'} 2024-03-04T17:55:27.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:53:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fdcc79ea7f8213c0-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.310 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.310 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.310 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.310 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.310 DEBUG:boto:Method: GET 2024-03-04T17:55:27.310 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:27.310 DEBUG:boto:Data: 2024-03-04T17:55:27.310 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.311 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.311 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.311 DEBUG:boto:Params: {} 2024-03-04T17:55:27.311 DEBUG:boto:Token: None 2024-03-04T17:55:27.311 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/ 2024-03-04T17:55:27.311 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc= 2024-03-04T17:55:27.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc=', 'Content-Length': '0'} 2024-03-04T17:55:27.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7a67feb49994624-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.356 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:27.356 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.357 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.357 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.357 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.357 DEBUG:boto:Method: GET 2024-03-04T17:55:27.357 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:27.357 DEBUG:boto:Data: 2024-03-04T17:55:27.357 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.357 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.357 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.357 DEBUG:boto:Params: {} 2024-03-04T17:55:27.357 DEBUG:boto:Token: None 2024-03-04T17:55:27.358 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/ 2024-03-04T17:55:27.358 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc= 2024-03-04T17:55:27.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc=', 'Content-Length': '0'} 2024-03-04T17:55:27.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031af99b2386939b6-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.404 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:27.405 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.405 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.405 DEBUG:boto:Method: HEAD 2024-03-04T17:55:27.405 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:55:27.405 DEBUG:boto:Data: 2024-03-04T17:55:27.405 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.405 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.405 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.405 DEBUG:boto:Params: {} 2024-03-04T17:55:27.406 DEBUG:boto:Token: None 2024-03-04T17:55:27.406 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj 2024-03-04T17:55:27.406 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G3H796KIfaZ+iT+3EjZHiDV7iLQ= 2024-03-04T17:55:27.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G3H796KIfaZ+iT+3EjZHiDV7iLQ=', 'Content-Length': '0'} 2024-03-04T17:55:27.411 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003b576624e08935dc-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.411 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.412 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.412 DEBUG:boto:Method: HEAD 2024-03-04T17:55:27.412 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:55:27.412 DEBUG:boto:Data: 2024-03-04T17:55:27.412 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.412 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.412 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.412 DEBUG:boto:Params: {} 2024-03-04T17:55:27.412 DEBUG:boto:Token: None 2024-03-04T17:55:27.413 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj 2024-03-04T17:55:27.413 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G3H796KIfaZ+iT+3EjZHiDV7iLQ= 2024-03-04T17:55:27.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G3H796KIfaZ+iT+3EjZHiDV7iLQ=', 'Content-Length': '0'} 2024-03-04T17:55:27.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000003f7bbceeab73ae8-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:55:27.417 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.417 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.417 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.417 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.417 DEBUG:boto:Method: GET 2024-03-04T17:55:27.417 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.417 DEBUG:boto:Data: 2024-03-04T17:55:27.417 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.417 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.417 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.417 DEBUG:boto:Params: {} 2024-03-04T17:55:27.418 DEBUG:boto:Token: None 2024-03-04T17:55:27.418 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.418 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc= 2024-03-04T17:55:27.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc=', 'Content-Length': '0'} 2024-03-04T17:55:27.421 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006f01185825c7cb8a-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.421 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:27.421 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:27.421 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.421 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.422 DEBUG:boto:Method: GET 2024-03-04T17:55:27.422 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.422 DEBUG:boto:Data: 2024-03-04T17:55:27.422 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.422 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.422 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.422 DEBUG:boto:Params: {} 2024-03-04T17:55:27.422 DEBUG:boto:Token: None 2024-03-04T17:55:27.422 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:55:27.422 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc= 2024-03-04T17:55:27.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9cqPnDIYhtqkn9GcdIDmiiC9Ecc=', 'Content-Length': '0'} 2024-03-04T17:55:27.425 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008b93c057f424fc26-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.425 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.425 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.426 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.426 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.426 DEBUG:boto:Method: GET 2024-03-04T17:55:27.426 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:27.426 DEBUG:boto:Data: 2024-03-04T17:55:27.426 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.426 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:55:27.426 DEBUG:boto:Port: 8001 2024-03-04T17:55:27.426 DEBUG:boto:Params: {} 2024-03-04T17:55:27.426 DEBUG:boto:Token: None 2024-03-04T17:55:27.426 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/ 2024-03-04T17:55:27.427 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc= 2024-03-04T17:55:27.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc=', 'Content-Length': '0'} 2024-03-04T17:55:27.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5f25e0f1e33a527-0065e60b0f-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.431 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:27.431 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.432 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.432 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.432 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:55:27.432 DEBUG:boto:Method: GET 2024-03-04T17:55:27.432 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:55:27.432 DEBUG:boto:Data: 2024-03-04T17:55:27.432 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.432 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:55:27.432 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.432 DEBUG:boto:Params: {} 2024-03-04T17:55:27.432 DEBUG:boto:Token: None 2024-03-04T17:55:27.433 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/ 2024-03-04T17:55:27.433 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc= 2024-03-04T17:55:27.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vuCN9YPR7isecY1HOxbseqWcFJc=', 'Content-Length': '0'} 2024-03-04T17:55:27.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcf25652603c9e78-0065e60b0f-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.484 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:55:27.485 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-23 zones={test-zone3, test-zone2} 2024-03-04T17:55:27.485 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:27.485 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:27.485 DEBUG:boto:path=/ppfahn-23/?versioning 2024-03-04T17:55:27.485 DEBUG:boto:auth_path=/ppfahn-23/?versioning 2024-03-04T17:55:27.485 DEBUG:boto:Method: PUT 2024-03-04T17:55:27.485 DEBUG:boto:Path: /ppfahn-23/?versioning 2024-03-04T17:55:27.485 DEBUG:boto:Data: Suspended Disabled 2024-03-04T17:55:27.485 DEBUG:boto:Headers: {} 2024-03-04T17:55:27.486 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:27.486 DEBUG:boto:Port: 8000 2024-03-04T17:55:27.486 DEBUG:boto:Params: {} 2024-03-04T17:55:27.486 DEBUG:boto:Token: None 2024-03-04T17:55:27.486 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:55:27 GMT /ppfahn-23/?versioning 2024-03-04T17:55:27.486 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cvyr0P0BONkOL6w2zHuzIEYCX3E= 2024-03-04T17:55:27.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cvyr0P0BONkOL6w2zHuzIEYCX3E=', 'Content-Length': '230'} 2024-03-04T17:55:27.499 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dc8ff52cec1658bc-0065e60b0f-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:27.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:27.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:27.499 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:27.539 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:27.539 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:27.595 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:55:27.618 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709574927.501462_206.1","last_update":"2024-03-04T17:55:27.501462Z"}] 2024-03-04T17:55:27.618 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574927.501462_206.1'} 2024-03-04T17:55:27.619 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:55:27.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:27.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:27.619 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:27.663 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:27.663 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:27.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:55:27.722+0000 7f24be0ed980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:27.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:55:27.722+0000 7f24be0ed980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:27.745 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574927.494130_205.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:55:27.494130Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:27Z"} 2024-03-04T17:55:27.746 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574927.494130_205.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:55:27.494130Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:27Z"} 2024-03-04T17:55:27.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574927.494130_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:55:27.494130Z', 'realm_epoch': 2}}] 2024-03-04T17:55:27.746 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574927.501462_206.1'} 2024-03-04T17:55:27.746 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574927.494130_205.1'} 2024-03-04T17:55:27.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709574927.501462_206.1 target=1_1709574927.494130_205.1 2024-03-04T17:55:32.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:32.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:32.747 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:32.791 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:32.791 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:32.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:55:32.850+0000 7f59ff08a980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:32.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:55:32.850+0000 7f59ff08a980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:32.869 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574927.501462_206.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:55:27.501462Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:32Z"} 2024-03-04T17:55:32.870 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574927.501462_206.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:55:27.501462Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:32Z"} 2024-03-04T17:55:32.870 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574927.501462_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:55:27.501462Z', 'realm_epoch': 2}}] 2024-03-04T17:55:32.870 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574927.501462_206.1'} 2024-03-04T17:55:32.870 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574927.501462_206.1'} 2024-03-04T17:55:32.870 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:55:32.870 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:55:32.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:32.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:32.871 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:32.914 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:32.914 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:32.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:32.968+0000 7f72db3d1980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:32.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:32.972+0000 7f72db3d1980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:32.991 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574927.501462_206.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:55:27.501462Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:32Z"} 2024-03-04T17:55:32.991 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574927.501462_206.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:55:27.501462Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:32Z"} 2024-03-04T17:55:32.991 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574927.501462_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:55:27.501462Z', 'realm_epoch': 2}}] 2024-03-04T17:55:32.992 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574927.501462_206.1'} 2024-03-04T17:55:32.992 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574927.501462_206.1'} 2024-03-04T17:55:32.992 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:55:32.992 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:55:32.992 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:55:32.992 DEBUG:boto:Method: HEAD 2024-03-04T17:55:32.992 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:55:32.993 DEBUG:boto:Data: 2024-03-04T17:55:32.993 DEBUG:boto:Headers: {} 2024-03-04T17:55:32.993 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:32.993 DEBUG:boto:Port: 8000 2024-03-04T17:55:32.993 DEBUG:boto:Params: {} 2024-03-04T17:55:32.993 DEBUG:boto:Token: None 2024-03-04T17:55:32.993 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:55:32 GMT /ppfahn-23/ 2024-03-04T17:55:32.993 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+xMzReLNK1dO/N/o2ztm1uiTbps= 2024-03-04T17:55:32.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:55:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+xMzReLNK1dO/N/o2ztm1uiTbps=', 'Content-Length': '0'} 2024-03-04T17:55:32.996 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bb8fba6cc40a85d-0065e60b14-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:32.996 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:55:32.996 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:55:32.996 DEBUG:boto:Method: PUT 2024-03-04T17:55:32.997 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:55:32.997 DEBUG:boto:Data: 2024-03-04T17:55:32.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T17:55:32.997 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:55:32.997 DEBUG:boto:Port: 8000 2024-03-04T17:55:32.997 DEBUG:boto:Params: {} 2024-03-04T17:55:32.997 DEBUG:boto:Token: None 2024-03-04T17:55:32.998 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 17:55:32 GMT /ppfahn-23/obj 2024-03-04T17:55:32.998 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+HgguBbPFem04U7NCEN/f9rsjs4= 2024-03-04T17:55:32.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:55:32 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+HgguBbPFem04U7NCEN/f9rsjs4='} 2024-03-04T17:55:33.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031a78871153ff9e4-0065e60b15-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:55:33.011 DEBUG:rgw_multi.tests:created null version id=None 2024-03-04T17:55:33.011 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-23 2024-03-04T17:55:33.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:55:33.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:55:33.011 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:55:33.062 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:33.123 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:33.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:55:33.130+0000 7f7ca2e5c980 1 waiting for incremental sync to catch up: 2024-03-04T17:55:33.125 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000003.30.13, , ] 2024-03-04T17:55:33.126 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000005.34.18, , ] 2024-03-04T17:55:38.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:55:38.130+0000 7f7ca2e5c980 1 waiting for incremental sync to catch up: 2024-03-04T17:55:38.128 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000003.30.13, , ] 2024-03-04T17:55:38.128 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000005.34.18, , ] 2024-03-04T17:55:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:55:43.134+0000 7f7ca2e5c980 1 waiting for incremental sync to catch up: 2024-03-04T17:55:43.131 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000003.30.13, , ] 2024-03-04T17:55:43.131 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000005.34.18, , ] 2024-03-04T17:55:48.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:55:48.138+0000 7f7ca2e5c980 1 waiting for incremental sync to catch up: 2024-03-04T17:55:48.133 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000003.30.13, , ] 2024-03-04T17:55:48.133 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000005.34.18, , ] 2024-03-04T17:55:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:55:53.138+0000 7f7ca2e5c980 1 bucket sync caught up with source: 2024-03-04T17:55:53.135 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000005.34.18, , ] 2024-03-04T17:55:53.135 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000005.34.18, , ] 2024-03-04T17:55:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:55:53.138+0000 7f7ca2e5c980 0 bucket checkpoint complete 2024-03-04T17:55:53.172 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-23 2024-03-04T17:55:53.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:55:53.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:55:53.173 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:55:53.222 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:53.222 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:53.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:53.283+0000 7f2fc0cd1980 1 bucket sync caught up with source: 2024-03-04T17:55:53.282 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000005.34.18, , ] 2024-03-04T17:55:53.282 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000005.34.18, , ] 2024-03-04T17:55:53.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:53.283+0000 7f2fc0cd1980 0 bucket checkpoint complete 2024-03-04T17:55:53.304 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-23 2024-03-04T17:55:53.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:55:53.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:55:53.305 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:55:53.345 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:53.345 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:53.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:53.407+0000 7fd6acb53980 1 waiting for incremental sync to catch up: 2024-03-04T17:55:53.407 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000004.45.10, , ] 2024-03-04T17:55:53.407 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000006.50.18, , ] 2024-03-04T17:55:58.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:55:58.411+0000 7fd6acb53980 1 waiting for incremental sync to catch up: 2024-03-04T17:55:58.410 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000004.45.10, , ] 2024-03-04T17:55:58.410 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000006.50.18, , ] 2024-03-04T17:56:03.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:03.415+0000 7fd6acb53980 1 waiting for incremental sync to catch up: 2024-03-04T17:56:03.412 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000004.45.10, , ] 2024-03-04T17:56:03.413 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000006.50.18, , ] 2024-03-04T17:56:08.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:08.415+0000 7fd6acb53980 1 bucket sync caught up with source: 2024-03-04T17:56:08.416 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000006.50.18, , ] 2024-03-04T17:56:08.416 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000006.50.18, , ] 2024-03-04T17:56:08.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:08.415+0000 7fd6acb53980 0 bucket checkpoint complete 2024-03-04T17:56:08.453 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-23 2024-03-04T17:56:08.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:56:08.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:56:08.454 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:56:08.552 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:08.552 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:08.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:08.615+0000 7fd3ea7f9980 1 waiting for incremental sync to catch up: 2024-03-04T17:56:08.615 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000004.45.10, , ] 2024-03-04T17:56:08.615 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000006.50.18, , ] 2024-03-04T17:56:13.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:13.619+0000 7fd3ea7f9980 1 bucket sync caught up with source: 2024-03-04T17:56:13.617 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000006.50.18, , ] 2024-03-04T17:56:13.618 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000006.50.18, , ] 2024-03-04T17:56:13.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:13.619+0000 7fd3ea7f9980 0 bucket checkpoint complete 2024-03-04T17:56:13.661 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-23 2024-03-04T17:56:13.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:56:13.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:56:13.661 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:56:13.710 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:13.710 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:13.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:13.771+0000 7f34478c5980 1 waiting for incremental sync to catch up: 2024-03-04T17:56:13.769 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000004.51.10, , ] 2024-03-04T17:56:13.769 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000007.58.15, , ] 2024-03-04T17:56:18.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:18.770+0000 7f34478c5980 1 waiting for incremental sync to catch up: 2024-03-04T17:56:18.771 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000004.51.10, , ] 2024-03-04T17:56:18.771 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000007.58.15, , ] 2024-03-04T17:56:23.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:23.774+0000 7f34478c5980 1 waiting for incremental sync to catch up: 2024-03-04T17:56:23.774 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000004.51.10, , ] 2024-03-04T17:56:23.774 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000007.58.15, , ] 2024-03-04T17:56:28.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:28.774+0000 7f34478c5980 1 bucket sync caught up with source: 2024-03-04T17:56:28.776 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000007.58.15, , ] 2024-03-04T17:56:28.777 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000007.58.15, , ] 2024-03-04T17:56:28.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:28.774+0000 7f34478c5980 0 bucket checkpoint complete 2024-03-04T17:56:28.816 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-23 2024-03-04T17:56:28.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:56:28.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:56:28.817 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:56:28.869 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:28.869 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:28.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:56:28.933+0000 7f63979a1980 1 bucket sync caught up with source: 2024-03-04T17:56:28.932 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000007.58.15, , ] 2024-03-04T17:56:28.932 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000007.58.15, , ] 2024-03-04T17:56:28.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:56:28.933+0000 7f63979a1980 0 bucket checkpoint complete 2024-03-04T17:56:28.954 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-23 zones={test-zone2, test-zone1} 2024-03-04T17:56:28.954 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:28.955 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:28.955 DEBUG:boto:Method: HEAD 2024-03-04T17:56:28.955 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:56:28.955 DEBUG:boto:Data: 2024-03-04T17:56:28.955 DEBUG:boto:Headers: {} 2024-03-04T17:56:28.955 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:28.955 DEBUG:boto:Port: 8000 2024-03-04T17:56:28.955 DEBUG:boto:Params: {} 2024-03-04T17:56:28.956 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:56:28.956 DEBUG:boto:Token: None 2024-03-04T17:56:28.956 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:28 GMT /ppfahn-23/ 2024-03-04T17:56:28.956 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DBxLdlGAXxakM3QdS2TMO7vbbq4= 2024-03-04T17:56:28.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DBxLdlGAXxakM3QdS2TMO7vbbq4=', 'Content-Length': '0'} 2024-03-04T17:56:28.960 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a05d2f708a38f9b3-0065e60b4c-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:28.961 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:28.961 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:28.961 DEBUG:boto:Method: HEAD 2024-03-04T17:56:28.961 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:56:28.961 DEBUG:boto:Data: 2024-03-04T17:56:28.961 DEBUG:boto:Headers: {} 2024-03-04T17:56:28.961 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:28.961 DEBUG:boto:Port: 8000 2024-03-04T17:56:28.961 DEBUG:boto:Params: {} 2024-03-04T17:56:28.962 DEBUG:boto:Token: None 2024-03-04T17:56:28.962 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:28 GMT /ppfahn-23/ 2024-03-04T17:56:28.962 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DBxLdlGAXxakM3QdS2TMO7vbbq4= 2024-03-04T17:56:28.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DBxLdlGAXxakM3QdS2TMO7vbbq4=', 'Content-Length': '0'} 2024-03-04T17:56:28.966 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0cd8a55e9371377-0065e60b4c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:28.966 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:56:28.966 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:28.967 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:28.967 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:28.967 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:28.967 DEBUG:boto:Method: GET 2024-03-04T17:56:28.967 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:28.967 DEBUG:boto:Data: 2024-03-04T17:56:28.967 DEBUG:boto:Headers: {} 2024-03-04T17:56:28.967 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:28.967 DEBUG:boto:Port: 8000 2024-03-04T17:56:28.967 DEBUG:boto:Params: {} 2024-03-04T17:56:28.967 DEBUG:boto:Token: None 2024-03-04T17:56:28.968 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:28 GMT /ppfahn-23/?versions 2024-03-04T17:56:28.968 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:utUXh7UTa0a6gK4AoT7u/YLWfC4= 2024-03-04T17:56:28.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:utUXh7UTa0a6gK4AoT7u/YLWfC4=', 'Content-Length': '0'} 2024-03-04T17:56:28.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000375c32d02ea81dda-0065e60b4c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:28.970 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCfalse2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:28.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:56:28.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:56:28.971 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:56:28.972 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:28.972 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:28.972 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:28.972 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:28.972 DEBUG:boto:Method: GET 2024-03-04T17:56:28.972 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:28.972 DEBUG:boto:Data: 2024-03-04T17:56:28.972 DEBUG:boto:Headers: {} 2024-03-04T17:56:28.972 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:28.972 DEBUG:boto:Port: 8000 2024-03-04T17:56:28.972 DEBUG:boto:Params: {} 2024-03-04T17:56:28.973 DEBUG:boto:Token: None 2024-03-04T17:56:28.973 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:28 GMT /ppfahn-23/?versions 2024-03-04T17:56:28.973 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:utUXh7UTa0a6gK4AoT7u/YLWfC4= 2024-03-04T17:56:28.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:utUXh7UTa0a6gK4AoT7u/YLWfC4=', 'Content-Length': '0'} 2024-03-04T17:56:28.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a361bc1bbd852739-0065e60b4c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:28.977 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCfalse2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:28.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:56:28.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:56:28.978 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:28.978 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:28.978 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:28.978 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:28.978 DEBUG:boto:Method: GET 2024-03-04T17:56:28.979 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:28.979 DEBUG:boto:Data: 2024-03-04T17:56:28.979 DEBUG:boto:Headers: {} 2024-03-04T17:56:28.979 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:28.979 DEBUG:boto:Port: 8000 2024-03-04T17:56:28.979 DEBUG:boto:Params: {} 2024-03-04T17:56:28.979 DEBUG:boto:Token: None 2024-03-04T17:56:28.979 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:28 GMT /ppfahn-23/?versions 2024-03-04T17:56:28.979 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:utUXh7UTa0a6gK4AoT7u/YLWfC4= 2024-03-04T17:56:28.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:utUXh7UTa0a6gK4AoT7u/YLWfC4=', 'Content-Length': '0'} 2024-03-04T17:56:28.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d9d8c25ee42b77b-0065e60b4c-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:28.982 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCfalse2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:28.982 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:28.982 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:28.982 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:28.982 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:28.982 DEBUG:boto:Method: GET 2024-03-04T17:56:28.982 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:28.983 DEBUG:boto:Data: 2024-03-04T17:56:28.983 DEBUG:boto:Headers: {} 2024-03-04T17:56:28.983 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:28.983 DEBUG:boto:Port: 8000 2024-03-04T17:56:28.983 DEBUG:boto:Params: {} 2024-03-04T17:56:28.983 DEBUG:boto:Token: None 2024-03-04T17:56:28.983 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:28 GMT /ppfahn-23/?versions 2024-03-04T17:56:28.983 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:utUXh7UTa0a6gK4AoT7u/YLWfC4= 2024-03-04T17:56:28.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:28 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:utUXh7UTa0a6gK4AoT7u/YLWfC4=', 'Content-Length': '0'} 2024-03-04T17:56:28.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d695ee7ec0655a4f-0065e60b4c-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.028 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCfalse2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.029 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.029 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.029 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.029 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.029 DEBUG:boto:Method: GET 2024-03-04T17:56:29.029 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.029 DEBUG:boto:Data: 2024-03-04T17:56:29.029 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.029 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.030 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.030 DEBUG:boto:Params: {} 2024-03-04T17:56:29.030 DEBUG:boto:Token: None 2024-03-04T17:56:29.030 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.030 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE= 2024-03-04T17:56:29.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE=', 'Content-Length': '0'} 2024-03-04T17:56:29.033 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000000a2f9ee95018cbe-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.033 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.033 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.033 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.034 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.034 DEBUG:boto:Method: GET 2024-03-04T17:56:29.034 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.034 DEBUG:boto:Data: 2024-03-04T17:56:29.034 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.034 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.034 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.034 DEBUG:boto:Params: {} 2024-03-04T17:56:29.034 DEBUG:boto:Token: None 2024-03-04T17:56:29.034 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.035 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE= 2024-03-04T17:56:29.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE=', 'Content-Length': '0'} 2024-03-04T17:56:29.037 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b4d5605675fce32e-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.038 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.038 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.038 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.038 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.038 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.038 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.038 DEBUG:boto:Data: 2024-03-04T17:56:29.038 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.039 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.039 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.039 DEBUG:boto:Params: {} 2024-03-04T17:56:29.039 DEBUG:boto:Token: None 2024-03-04T17:56:29.039 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.039 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YvtdswZT/WJRHWItih+ayGZJ7ic= 2024-03-04T17:56:29.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YvtdswZT/WJRHWItih+ayGZJ7ic=', 'Content-Length': '0'} 2024-03-04T17:56:29.043 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b84dca3c94fc3539-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.043 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.043 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.044 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.044 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.044 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.044 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.044 DEBUG:boto:Data: 2024-03-04T17:56:29.044 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.044 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.044 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.045 DEBUG:boto:Params: {} 2024-03-04T17:56:29.045 DEBUG:boto:Token: None 2024-03-04T17:56:29.045 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.045 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YvtdswZT/WJRHWItih+ayGZJ7ic= 2024-03-04T17:56:29.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YvtdswZT/WJRHWItih+ayGZJ7ic=', 'Content-Length': '0'} 2024-03-04T17:56:29.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f957c32dfc42c731-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.049 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.049 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.049 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.049 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.049 DEBUG:boto:Method: GET 2024-03-04T17:56:29.049 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.049 DEBUG:boto:Data: 2024-03-04T17:56:29.050 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.050 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.050 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.050 DEBUG:boto:Params: {} 2024-03-04T17:56:29.050 DEBUG:boto:Token: None 2024-03-04T17:56:29.050 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.050 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE= 2024-03-04T17:56:29.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE=', 'Content-Length': '0'} 2024-03-04T17:56:29.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af7ad36c13b42267-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.054 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.054 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.054 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.055 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.055 DEBUG:boto:Method: GET 2024-03-04T17:56:29.055 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.055 DEBUG:boto:Data: 2024-03-04T17:56:29.055 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.055 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.055 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.055 DEBUG:boto:Params: {} 2024-03-04T17:56:29.055 DEBUG:boto:Token: None 2024-03-04T17:56:29.056 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.056 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE= 2024-03-04T17:56:29.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE=', 'Content-Length': '0'} 2024-03-04T17:56:29.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000602a85770e982448-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.059 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.059 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.060 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.060 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.060 DEBUG:boto:Method: GET 2024-03-04T17:56:29.060 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.060 DEBUG:boto:Data: 2024-03-04T17:56:29.060 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.060 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.060 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.060 DEBUG:boto:Params: {} 2024-03-04T17:56:29.061 DEBUG:boto:Token: None 2024-03-04T17:56:29.061 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.061 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c678948012dbc911-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.112 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.112 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.112 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.113 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.113 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.113 DEBUG:boto:Method: GET 2024-03-04T17:56:29.113 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.113 DEBUG:boto:Data: 2024-03-04T17:56:29.113 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.113 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.113 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.113 DEBUG:boto:Params: {} 2024-03-04T17:56:29.114 DEBUG:boto:Token: None 2024-03-04T17:56:29.114 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.114 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076b925018f7638f4-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.118 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.118 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.118 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.119 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.119 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.119 DEBUG:boto:Data: 2024-03-04T17:56:29.119 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.119 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.119 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.119 DEBUG:boto:Params: {} 2024-03-04T17:56:29.119 DEBUG:boto:Token: None 2024-03-04T17:56:29.120 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.120 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw= 2024-03-04T17:56:29.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw=', 'Content-Length': '0'} 2024-03-04T17:56:29.123 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ae3e6b584f04041-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.123 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.123 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.123 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.123 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.123 DEBUG:boto:Data: 2024-03-04T17:56:29.124 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.124 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.124 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.124 DEBUG:boto:Params: {} 2024-03-04T17:56:29.124 DEBUG:boto:Token: None 2024-03-04T17:56:29.124 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.124 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw= 2024-03-04T17:56:29.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw=', 'Content-Length': '0'} 2024-03-04T17:56:29.127 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d6e5ebda462a6994-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.127 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.127 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.128 DEBUG:boto:Method: GET 2024-03-04T17:56:29.128 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.128 DEBUG:boto:Data: 2024-03-04T17:56:29.128 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.128 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.128 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.128 DEBUG:boto:Params: {} 2024-03-04T17:56:29.128 DEBUG:boto:Token: None 2024-03-04T17:56:29.128 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.129 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0= 2024-03-04T17:56:29.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0=', 'Content-Length': '0'} 2024-03-04T17:56:29.131 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e68b87b70acb9e83-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.131 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.132 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.132 DEBUG:boto:Method: GET 2024-03-04T17:56:29.132 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.132 DEBUG:boto:Data: 2024-03-04T17:56:29.132 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.132 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.132 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.132 DEBUG:boto:Params: {} 2024-03-04T17:56:29.133 DEBUG:boto:Token: None 2024-03-04T17:56:29.133 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.133 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0= 2024-03-04T17:56:29.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0=', 'Content-Length': '0'} 2024-03-04T17:56:29.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d8e180dc1648b33-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.136 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.136 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.136 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.136 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.136 DEBUG:boto:Method: GET 2024-03-04T17:56:29.136 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.137 DEBUG:boto:Data: 2024-03-04T17:56:29.137 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.137 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.137 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.137 DEBUG:boto:Params: {} 2024-03-04T17:56:29.137 DEBUG:boto:Token: None 2024-03-04T17:56:29.137 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.137 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9828b410d01e2f9-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.141 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.142 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.142 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.142 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.142 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.142 DEBUG:boto:Method: GET 2024-03-04T17:56:29.142 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.142 DEBUG:boto:Data: 2024-03-04T17:56:29.142 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.142 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.143 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.143 DEBUG:boto:Params: {} 2024-03-04T17:56:29.143 DEBUG:boto:Token: None 2024-03-04T17:56:29.143 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.143 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e0e82657cecf880-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.147 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.148 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.148 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.148 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.148 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.148 DEBUG:boto:Method: GET 2024-03-04T17:56:29.148 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.149 DEBUG:boto:Data: 2024-03-04T17:56:29.149 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.149 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.149 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.149 DEBUG:boto:Params: {} 2024-03-04T17:56:29.149 DEBUG:boto:Token: None 2024-03-04T17:56:29.149 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.149 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY= 2024-03-04T17:56:29.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY=', 'Content-Length': '0'} 2024-03-04T17:56:29.152 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000099522303d9ab93f7-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.152 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.153 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.153 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.153 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.153 DEBUG:boto:Method: GET 2024-03-04T17:56:29.153 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.153 DEBUG:boto:Data: 2024-03-04T17:56:29.153 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.153 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.153 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.154 DEBUG:boto:Params: {} 2024-03-04T17:56:29.154 DEBUG:boto:Token: None 2024-03-04T17:56:29.154 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.154 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY= 2024-03-04T17:56:29.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY=', 'Content-Length': '0'} 2024-03-04T17:56:29.157 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cee09c243cd6f318-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.157 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.157 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.157 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.158 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.158 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.158 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.158 DEBUG:boto:Data: 2024-03-04T17:56:29.158 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.158 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.158 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.158 DEBUG:boto:Params: {} 2024-03-04T17:56:29.158 DEBUG:boto:Token: None 2024-03-04T17:56:29.159 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.159 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W0/I4J1/s4zhQ/E4Rrm9qG9E4L4= 2024-03-04T17:56:29.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W0/I4J1/s4zhQ/E4Rrm9qG9E4L4=', 'Content-Length': '0'} 2024-03-04T17:56:29.161 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f9e56d35a3380613-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.162 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.162 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.162 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.162 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.162 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.162 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.162 DEBUG:boto:Data: 2024-03-04T17:56:29.162 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.163 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.163 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.163 DEBUG:boto:Params: {} 2024-03-04T17:56:29.163 DEBUG:boto:Token: None 2024-03-04T17:56:29.163 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.163 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W0/I4J1/s4zhQ/E4Rrm9qG9E4L4= 2024-03-04T17:56:29.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W0/I4J1/s4zhQ/E4Rrm9qG9E4L4=', 'Content-Length': '0'} 2024-03-04T17:56:29.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008b730a052d63cf06-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.166 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.167 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.167 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.167 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.167 DEBUG:boto:Method: GET 2024-03-04T17:56:29.167 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.167 DEBUG:boto:Data: 2024-03-04T17:56:29.167 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.167 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.167 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.167 DEBUG:boto:Params: {} 2024-03-04T17:56:29.168 DEBUG:boto:Token: None 2024-03-04T17:56:29.168 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.168 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY= 2024-03-04T17:56:29.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY=', 'Content-Length': '0'} 2024-03-04T17:56:29.171 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002765620c447b272c-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.171 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.172 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.172 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.172 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.172 DEBUG:boto:Method: GET 2024-03-04T17:56:29.172 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.172 DEBUG:boto:Data: 2024-03-04T17:56:29.172 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.172 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.172 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.173 DEBUG:boto:Params: {} 2024-03-04T17:56:29.173 DEBUG:boto:Token: None 2024-03-04T17:56:29.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.173 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY= 2024-03-04T17:56:29.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY=', 'Content-Length': '0'} 2024-03-04T17:56:29.176 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073259c2587c811b1-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.176 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.176 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.177 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.177 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.177 DEBUG:boto:Method: GET 2024-03-04T17:56:29.177 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.177 DEBUG:boto:Data: 2024-03-04T17:56:29.177 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.177 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.177 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.177 DEBUG:boto:Params: {} 2024-03-04T17:56:29.178 DEBUG:boto:Token: None 2024-03-04T17:56:29.178 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.178 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002690eae6ce1884fb-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.228 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.229 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.229 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.229 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.229 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.229 DEBUG:boto:Method: GET 2024-03-04T17:56:29.229 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.229 DEBUG:boto:Data: 2024-03-04T17:56:29.229 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.229 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.229 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.229 DEBUG:boto:Params: {} 2024-03-04T17:56:29.230 DEBUG:boto:Token: None 2024-03-04T17:56:29.230 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.230 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068cb5a4a59fdab1f-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.234 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.235 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.235 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.235 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.235 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.235 DEBUG:boto:Data: 2024-03-04T17:56:29.235 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.235 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.235 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.235 DEBUG:boto:Params: {} 2024-03-04T17:56:29.236 DEBUG:boto:Token: None 2024-03-04T17:56:29.236 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.236 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw= 2024-03-04T17:56:29.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw=', 'Content-Length': '0'} 2024-03-04T17:56:29.239 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7e6469dfeb71416-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.239 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.239 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.239 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.239 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.240 DEBUG:boto:Data: 2024-03-04T17:56:29.240 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.240 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.240 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.240 DEBUG:boto:Params: {} 2024-03-04T17:56:29.240 DEBUG:boto:Token: None 2024-03-04T17:56:29.240 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.240 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw= 2024-03-04T17:56:29.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw=', 'Content-Length': '0'} 2024-03-04T17:56:29.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a0bae6a81a24d3e-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.243 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.243 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.244 DEBUG:boto:Method: GET 2024-03-04T17:56:29.244 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.244 DEBUG:boto:Data: 2024-03-04T17:56:29.244 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.244 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.244 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.244 DEBUG:boto:Params: {} 2024-03-04T17:56:29.246 DEBUG:boto:Token: None 2024-03-04T17:56:29.247 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.247 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0= 2024-03-04T17:56:29.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0=', 'Content-Length': '0'} 2024-03-04T17:56:29.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000835bffd78c858d13-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.250 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.250 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.250 DEBUG:boto:Method: GET 2024-03-04T17:56:29.250 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.251 DEBUG:boto:Data: 2024-03-04T17:56:29.251 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.251 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.251 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.251 DEBUG:boto:Params: {} 2024-03-04T17:56:29.251 DEBUG:boto:Token: None 2024-03-04T17:56:29.251 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.251 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0= 2024-03-04T17:56:29.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0=', 'Content-Length': '0'} 2024-03-04T17:56:29.254 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e42df7e57b3acf0-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.254 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.254 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.254 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.254 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.255 DEBUG:boto:Method: GET 2024-03-04T17:56:29.255 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.255 DEBUG:boto:Data: 2024-03-04T17:56:29.255 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.255 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.255 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.255 DEBUG:boto:Params: {} 2024-03-04T17:56:29.255 DEBUG:boto:Token: None 2024-03-04T17:56:29.255 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.255 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f4f1db04910aac1-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.260 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.260 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.261 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.261 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.261 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.261 DEBUG:boto:Method: GET 2024-03-04T17:56:29.261 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.261 DEBUG:boto:Data: 2024-03-04T17:56:29.261 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.261 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.261 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.261 DEBUG:boto:Params: {} 2024-03-04T17:56:29.261 DEBUG:boto:Token: None 2024-03-04T17:56:29.262 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.262 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9de1db3e36a7ddf-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.266 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.266 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-23 zones={test-zone2, test-zone1} 2024-03-04T17:56:29.267 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-23 zones={test-zone3, test-zone1} 2024-03-04T17:56:29.267 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.267 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.267 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.267 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:56:29.267 DEBUG:boto:Data: 2024-03-04T17:56:29.267 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.267 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.267 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.267 DEBUG:boto:Params: {} 2024-03-04T17:56:29.268 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T17:56:29.268 DEBUG:boto:Token: None 2024-03-04T17:56:29.268 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.268 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:abLOt54sfyubtqTGaY9tiwYIMX0= 2024-03-04T17:56:29.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:abLOt54sfyubtqTGaY9tiwYIMX0=', 'Content-Length': '0'} 2024-03-04T17:56:29.274 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c80a446f1401fff-0065e60b4d-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.274 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.274 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.274 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.274 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:56:29.274 DEBUG:boto:Data: 2024-03-04T17:56:29.274 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.274 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.274 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.275 DEBUG:boto:Params: {} 2024-03-04T17:56:29.275 DEBUG:boto:Token: None 2024-03-04T17:56:29.275 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.275 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:abLOt54sfyubtqTGaY9tiwYIMX0= 2024-03-04T17:56:29.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:abLOt54sfyubtqTGaY9tiwYIMX0=', 'Content-Length': '0'} 2024-03-04T17:56:29.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5ab9a87b3b0b8ca-0065e60b4d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.279 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:56:29.279 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.279 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.279 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.279 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.279 DEBUG:boto:Method: GET 2024-03-04T17:56:29.279 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.280 DEBUG:boto:Data: 2024-03-04T17:56:29.280 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.280 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.280 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.280 DEBUG:boto:Params: {} 2024-03-04T17:56:29.280 DEBUG:boto:Token: None 2024-03-04T17:56:29.280 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/?versions 2024-03-04T17:56:29.280 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk= 2024-03-04T17:56:29.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk=', 'Content-Length': '0'} 2024-03-04T17:56:29.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1353099d1414b6a-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.285 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCfalse2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:56:29.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:56:29.285 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:56:29.286 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.286 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.286 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.286 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.286 DEBUG:boto:Method: GET 2024-03-04T17:56:29.286 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.286 DEBUG:boto:Data: 2024-03-04T17:56:29.286 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.286 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.286 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.286 DEBUG:boto:Params: {} 2024-03-04T17:56:29.287 DEBUG:boto:Token: None 2024-03-04T17:56:29.287 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/?versions 2024-03-04T17:56:29.287 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk= 2024-03-04T17:56:29.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk=', 'Content-Length': '0'} 2024-03-04T17:56:29.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004358a67cd4d32129-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.336 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCfalse2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:56:29.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:56:29.337 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.337 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.337 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.337 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.337 DEBUG:boto:Method: GET 2024-03-04T17:56:29.337 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.337 DEBUG:boto:Data: 2024-03-04T17:56:29.337 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.337 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.337 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.338 DEBUG:boto:Params: {} 2024-03-04T17:56:29.338 DEBUG:boto:Token: None 2024-03-04T17:56:29.338 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/?versions 2024-03-04T17:56:29.338 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk= 2024-03-04T17:56:29.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk=', 'Content-Length': '0'} 2024-03-04T17:56:29.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000188b5d876925ba7e-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.343 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCfalse2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.343 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.343 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.343 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.343 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.343 DEBUG:boto:Method: GET 2024-03-04T17:56:29.344 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.344 DEBUG:boto:Data: 2024-03-04T17:56:29.344 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.344 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.344 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.344 DEBUG:boto:Params: {} 2024-03-04T17:56:29.344 DEBUG:boto:Token: None 2024-03-04T17:56:29.344 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/?versions 2024-03-04T17:56:29.344 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk= 2024-03-04T17:56:29.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk=', 'Content-Length': '0'} 2024-03-04T17:56:29.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c2645b3d8bcf935-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.350 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCfalse2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.350 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.350 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.350 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.351 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.351 DEBUG:boto:Method: GET 2024-03-04T17:56:29.351 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.351 DEBUG:boto:Data: 2024-03-04T17:56:29.351 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.351 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.351 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.351 DEBUG:boto:Params: {} 2024-03-04T17:56:29.351 DEBUG:boto:Token: None 2024-03-04T17:56:29.351 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.352 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE= 2024-03-04T17:56:29.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE=', 'Content-Length': '0'} 2024-03-04T17:56:29.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000076a9f3b8c4d9e070-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.355 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.355 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.355 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.355 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.355 DEBUG:boto:Method: GET 2024-03-04T17:56:29.355 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.355 DEBUG:boto:Data: 2024-03-04T17:56:29.355 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.355 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.355 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.356 DEBUG:boto:Params: {} 2024-03-04T17:56:29.356 DEBUG:boto:Token: None 2024-03-04T17:56:29.356 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.356 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE= 2024-03-04T17:56:29.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE=', 'Content-Length': '0'} 2024-03-04T17:56:29.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003020b9cca88a0f46-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.359 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.359 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.359 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.360 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.360 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.360 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.360 DEBUG:boto:Data: 2024-03-04T17:56:29.360 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.360 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.360 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.360 DEBUG:boto:Params: {} 2024-03-04T17:56:29.360 DEBUG:boto:Token: None 2024-03-04T17:56:29.360 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.361 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YvtdswZT/WJRHWItih+ayGZJ7ic= 2024-03-04T17:56:29.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YvtdswZT/WJRHWItih+ayGZJ7ic=', 'Content-Length': '0'} 2024-03-04T17:56:29.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000de5c89c1a1420ff-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.363 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.363 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.364 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.364 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.364 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.364 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.364 DEBUG:boto:Data: 2024-03-04T17:56:29.364 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.364 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.364 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.364 DEBUG:boto:Params: {} 2024-03-04T17:56:29.364 DEBUG:boto:Token: None 2024-03-04T17:56:29.365 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.365 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YvtdswZT/WJRHWItih+ayGZJ7ic= 2024-03-04T17:56:29.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YvtdswZT/WJRHWItih+ayGZJ7ic=', 'Content-Length': '0'} 2024-03-04T17:56:29.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000004c33a3d473c532a-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.368 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.368 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.368 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.368 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.368 DEBUG:boto:Method: GET 2024-03-04T17:56:29.368 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.369 DEBUG:boto:Data: 2024-03-04T17:56:29.369 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.369 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.369 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.369 DEBUG:boto:Params: {} 2024-03-04T17:56:29.369 DEBUG:boto:Token: None 2024-03-04T17:56:29.369 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.369 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE= 2024-03-04T17:56:29.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE=', 'Content-Length': '0'} 2024-03-04T17:56:29.372 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002af96d310efa528a-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.372 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.373 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.373 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.373 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.373 DEBUG:boto:Method: GET 2024-03-04T17:56:29.373 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.373 DEBUG:boto:Data: 2024-03-04T17:56:29.373 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.373 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.373 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.373 DEBUG:boto:Params: {} 2024-03-04T17:56:29.373 DEBUG:boto:Token: None 2024-03-04T17:56:29.374 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.374 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE= 2024-03-04T17:56:29.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE=', 'Content-Length': '0'} 2024-03-04T17:56:29.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002120530d1dd84e08-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.377 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.377 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.377 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.377 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.378 DEBUG:boto:Method: GET 2024-03-04T17:56:29.378 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.378 DEBUG:boto:Data: 2024-03-04T17:56:29.378 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.378 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.378 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.378 DEBUG:boto:Params: {} 2024-03-04T17:56:29.378 DEBUG:boto:Token: None 2024-03-04T17:56:29.378 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.378 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d99eed5a3b695b5f-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.428 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.428 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.429 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.429 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.429 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.429 DEBUG:boto:Method: GET 2024-03-04T17:56:29.429 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.429 DEBUG:boto:Data: 2024-03-04T17:56:29.429 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.429 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.429 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.429 DEBUG:boto:Params: {} 2024-03-04T17:56:29.430 DEBUG:boto:Token: None 2024-03-04T17:56:29.430 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.430 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015513c3b3b4e3337-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.434 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.435 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.435 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.435 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.435 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.435 DEBUG:boto:Data: 2024-03-04T17:56:29.435 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.435 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.436 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.436 DEBUG:boto:Params: {} 2024-03-04T17:56:29.436 DEBUG:boto:Token: None 2024-03-04T17:56:29.436 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.436 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw= 2024-03-04T17:56:29.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw=', 'Content-Length': '0'} 2024-03-04T17:56:29.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008976107d269313ac-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.439 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.439 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.440 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.440 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.440 DEBUG:boto:Data: 2024-03-04T17:56:29.440 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.440 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.440 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.440 DEBUG:boto:Params: {} 2024-03-04T17:56:29.440 DEBUG:boto:Token: None 2024-03-04T17:56:29.440 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.441 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw= 2024-03-04T17:56:29.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw=', 'Content-Length': '0'} 2024-03-04T17:56:29.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c802a3afaa897ce0-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.444 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.444 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.444 DEBUG:boto:Method: GET 2024-03-04T17:56:29.444 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.444 DEBUG:boto:Data: 2024-03-04T17:56:29.444 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.444 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.444 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.445 DEBUG:boto:Params: {} 2024-03-04T17:56:29.445 DEBUG:boto:Token: None 2024-03-04T17:56:29.445 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.445 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0= 2024-03-04T17:56:29.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0=', 'Content-Length': '0'} 2024-03-04T17:56:29.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a64a2acd0559fe04-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.448 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.448 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.448 DEBUG:boto:Method: GET 2024-03-04T17:56:29.449 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.449 DEBUG:boto:Data: 2024-03-04T17:56:29.449 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.449 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.449 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.449 DEBUG:boto:Params: {} 2024-03-04T17:56:29.449 DEBUG:boto:Token: None 2024-03-04T17:56:29.449 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.449 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0= 2024-03-04T17:56:29.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0=', 'Content-Length': '0'} 2024-03-04T17:56:29.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f50882c0af625c44-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.453 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.453 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.453 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.453 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.453 DEBUG:boto:Method: GET 2024-03-04T17:56:29.453 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.453 DEBUG:boto:Data: 2024-03-04T17:56:29.453 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.453 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.453 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.453 DEBUG:boto:Params: {} 2024-03-04T17:56:29.454 DEBUG:boto:Token: None 2024-03-04T17:56:29.454 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.454 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d13211511758db7b-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.458 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.458 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.458 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.458 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.459 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.459 DEBUG:boto:Method: GET 2024-03-04T17:56:29.459 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.459 DEBUG:boto:Data: 2024-03-04T17:56:29.459 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.459 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.459 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.459 DEBUG:boto:Params: {} 2024-03-04T17:56:29.459 DEBUG:boto:Token: None 2024-03-04T17:56:29.460 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.460 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e48660c55260847c-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.508 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.508 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.509 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.509 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.509 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.509 DEBUG:boto:Method: GET 2024-03-04T17:56:29.509 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.509 DEBUG:boto:Data: 2024-03-04T17:56:29.509 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.509 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.509 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.509 DEBUG:boto:Params: {} 2024-03-04T17:56:29.510 DEBUG:boto:Token: None 2024-03-04T17:56:29.510 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.510 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY= 2024-03-04T17:56:29.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY=', 'Content-Length': '0'} 2024-03-04T17:56:29.513 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000035953e595c0c8ca1-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.513 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.513 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.513 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.514 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.514 DEBUG:boto:Method: GET 2024-03-04T17:56:29.514 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.514 DEBUG:boto:Data: 2024-03-04T17:56:29.514 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.514 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.514 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.514 DEBUG:boto:Params: {} 2024-03-04T17:56:29.514 DEBUG:boto:Token: None 2024-03-04T17:56:29.514 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.515 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY= 2024-03-04T17:56:29.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY=', 'Content-Length': '0'} 2024-03-04T17:56:29.517 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b6e86439ca91073-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.518 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.518 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.518 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.518 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.518 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.518 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.518 DEBUG:boto:Data: 2024-03-04T17:56:29.518 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.519 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.519 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.519 DEBUG:boto:Params: {} 2024-03-04T17:56:29.519 DEBUG:boto:Token: None 2024-03-04T17:56:29.519 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.519 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W0/I4J1/s4zhQ/E4Rrm9qG9E4L4= 2024-03-04T17:56:29.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W0/I4J1/s4zhQ/E4Rrm9qG9E4L4=', 'Content-Length': '0'} 2024-03-04T17:56:29.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e579bceea0e265f5-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.522 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.523 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.523 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.523 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.523 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.523 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.523 DEBUG:boto:Data: 2024-03-04T17:56:29.523 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.523 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.523 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.523 DEBUG:boto:Params: {} 2024-03-04T17:56:29.524 DEBUG:boto:Token: None 2024-03-04T17:56:29.524 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.524 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W0/I4J1/s4zhQ/E4Rrm9qG9E4L4= 2024-03-04T17:56:29.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W0/I4J1/s4zhQ/E4Rrm9qG9E4L4=', 'Content-Length': '0'} 2024-03-04T17:56:29.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000022fd63954734af41-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.527 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.527 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.527 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.528 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.528 DEBUG:boto:Method: GET 2024-03-04T17:56:29.528 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.528 DEBUG:boto:Data: 2024-03-04T17:56:29.528 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.528 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.528 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.528 DEBUG:boto:Params: {} 2024-03-04T17:56:29.528 DEBUG:boto:Token: None 2024-03-04T17:56:29.529 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.529 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY= 2024-03-04T17:56:29.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY=', 'Content-Length': '0'} 2024-03-04T17:56:29.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007c9826ad5fc7beb2-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.532 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.532 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.532 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.532 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.533 DEBUG:boto:Method: GET 2024-03-04T17:56:29.533 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.533 DEBUG:boto:Data: 2024-03-04T17:56:29.533 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.533 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.533 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.533 DEBUG:boto:Params: {} 2024-03-04T17:56:29.533 DEBUG:boto:Token: None 2024-03-04T17:56:29.533 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.533 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY= 2024-03-04T17:56:29.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY=', 'Content-Length': '0'} 2024-03-04T17:56:29.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000940d1cac0178b11f-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.536 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.536 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.536 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.537 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.537 DEBUG:boto:Method: GET 2024-03-04T17:56:29.537 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.537 DEBUG:boto:Data: 2024-03-04T17:56:29.537 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.537 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.537 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.537 DEBUG:boto:Params: {} 2024-03-04T17:56:29.537 DEBUG:boto:Token: None 2024-03-04T17:56:29.538 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.538 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000203bd3e86ebb9d6b-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.588 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.588 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.589 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.589 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.589 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.589 DEBUG:boto:Method: GET 2024-03-04T17:56:29.589 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.589 DEBUG:boto:Data: 2024-03-04T17:56:29.589 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.589 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.589 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.589 DEBUG:boto:Params: {} 2024-03-04T17:56:29.590 DEBUG:boto:Token: None 2024-03-04T17:56:29.590 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.590 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2d24164e840acdf-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.594 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.595 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.595 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.595 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.595 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.595 DEBUG:boto:Data: 2024-03-04T17:56:29.595 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.595 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.596 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.596 DEBUG:boto:Params: {} 2024-03-04T17:56:29.596 DEBUG:boto:Token: None 2024-03-04T17:56:29.596 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.596 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw= 2024-03-04T17:56:29.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw=', 'Content-Length': '0'} 2024-03-04T17:56:29.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000377e211d15682599-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.599 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.599 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.599 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.599 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.599 DEBUG:boto:Data: 2024-03-04T17:56:29.599 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.600 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.600 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.600 DEBUG:boto:Params: {} 2024-03-04T17:56:29.600 DEBUG:boto:Token: None 2024-03-04T17:56:29.600 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.600 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw= 2024-03-04T17:56:29.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw=', 'Content-Length': '0'} 2024-03-04T17:56:29.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000067ba64115d17528d-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.604 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.604 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.604 DEBUG:boto:Method: GET 2024-03-04T17:56:29.604 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.604 DEBUG:boto:Data: 2024-03-04T17:56:29.604 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.604 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.604 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.605 DEBUG:boto:Params: {} 2024-03-04T17:56:29.605 DEBUG:boto:Token: None 2024-03-04T17:56:29.605 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.605 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0= 2024-03-04T17:56:29.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0=', 'Content-Length': '0'} 2024-03-04T17:56:29.608 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000235f566682c614b5-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.608 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.608 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.608 DEBUG:boto:Method: GET 2024-03-04T17:56:29.608 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.608 DEBUG:boto:Data: 2024-03-04T17:56:29.609 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.609 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.609 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.609 DEBUG:boto:Params: {} 2024-03-04T17:56:29.609 DEBUG:boto:Token: None 2024-03-04T17:56:29.609 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.609 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0= 2024-03-04T17:56:29.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0=', 'Content-Length': '0'} 2024-03-04T17:56:29.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007365807f4c1a21f7-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.613 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.613 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.613 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.613 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.613 DEBUG:boto:Method: GET 2024-03-04T17:56:29.613 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.613 DEBUG:boto:Data: 2024-03-04T17:56:29.614 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.614 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.614 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.614 DEBUG:boto:Params: {} 2024-03-04T17:56:29.614 DEBUG:boto:Token: None 2024-03-04T17:56:29.614 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.614 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d42a4de0fa181027-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.660 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.661 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.661 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.661 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.661 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.661 DEBUG:boto:Method: GET 2024-03-04T17:56:29.661 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.661 DEBUG:boto:Data: 2024-03-04T17:56:29.661 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.661 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:29.661 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.662 DEBUG:boto:Params: {} 2024-03-04T17:56:29.662 DEBUG:boto:Token: None 2024-03-04T17:56:29.662 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.662 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029d8812b62af474b-0065e60b4d-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.667 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.667 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-23 zones={test-zone3, test-zone1} 2024-03-04T17:56:29.667 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-23 zones={test-zone3, test-zone2} 2024-03-04T17:56:29.667 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.667 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.668 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.668 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:56:29.668 DEBUG:boto:Data: 2024-03-04T17:56:29.668 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.668 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.668 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.668 DEBUG:boto:Params: {} 2024-03-04T17:56:29.668 DEBUG:boto:Token: None 2024-03-04T17:56:29.668 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.669 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:abLOt54sfyubtqTGaY9tiwYIMX0= 2024-03-04T17:56:29.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:abLOt54sfyubtqTGaY9tiwYIMX0=', 'Content-Length': '0'} 2024-03-04T17:56:29.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d49533954c89d442-0065e60b4d-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.673 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.673 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.673 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.673 DEBUG:boto:Path: /ppfahn-23/ 2024-03-04T17:56:29.673 DEBUG:boto:Data: 2024-03-04T17:56:29.674 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.674 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.674 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.674 DEBUG:boto:Params: {} 2024-03-04T17:56:29.674 DEBUG:boto:Token: None 2024-03-04T17:56:29.674 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.674 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:abLOt54sfyubtqTGaY9tiwYIMX0= 2024-03-04T17:56:29.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:abLOt54sfyubtqTGaY9tiwYIMX0=', 'Content-Length': '0'} 2024-03-04T17:56:29.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a05a24687abaa47-0065e60b4d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.679 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:56:29.679 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.679 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.679 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.679 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.679 DEBUG:boto:Method: GET 2024-03-04T17:56:29.680 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.680 DEBUG:boto:Data: 2024-03-04T17:56:29.680 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.680 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.680 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.680 DEBUG:boto:Params: {} 2024-03-04T17:56:29.680 DEBUG:boto:Token: None 2024-03-04T17:56:29.680 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/?versions 2024-03-04T17:56:29.681 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk= 2024-03-04T17:56:29.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk=', 'Content-Length': '0'} 2024-03-04T17:56:29.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d11f897716790864-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.728 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCfalse2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:56:29.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:56:29.729 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:56:29.729 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.729 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.729 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.729 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.730 DEBUG:boto:Method: GET 2024-03-04T17:56:29.730 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.730 DEBUG:boto:Data: 2024-03-04T17:56:29.730 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.730 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.730 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.730 DEBUG:boto:Params: {} 2024-03-04T17:56:29.730 DEBUG:boto:Token: None 2024-03-04T17:56:29.730 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/?versions 2024-03-04T17:56:29.731 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk= 2024-03-04T17:56:29.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk=', 'Content-Length': '0'} 2024-03-04T17:56:29.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009b6f155ec611079-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.776 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCfalse2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:56:29.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:56:29.777 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.777 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.777 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.777 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.777 DEBUG:boto:Method: GET 2024-03-04T17:56:29.777 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.777 DEBUG:boto:Data: 2024-03-04T17:56:29.777 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.777 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.778 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.778 DEBUG:boto:Params: {} 2024-03-04T17:56:29.778 DEBUG:boto:Token: None 2024-03-04T17:56:29.778 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/?versions 2024-03-04T17:56:29.778 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk= 2024-03-04T17:56:29.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk=', 'Content-Length': '0'} 2024-03-04T17:56:29.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0b35cb2dc4c2dc6-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.783 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCfalse2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.783 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.783 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.783 DEBUG:boto:path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.783 DEBUG:boto:auth_path=/ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.783 DEBUG:boto:Method: GET 2024-03-04T17:56:29.784 DEBUG:boto:Path: /ppfahn-23/?versions&max-keys=999 2024-03-04T17:56:29.784 DEBUG:boto:Data: 2024-03-04T17:56:29.784 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.784 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.784 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.784 DEBUG:boto:Params: {} 2024-03-04T17:56:29.784 DEBUG:boto:Token: None 2024-03-04T17:56:29.784 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/?versions 2024-03-04T17:56:29.784 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk= 2024-03-04T17:56:29.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f5bgXLD8/no5uNe6Jusxj8AHSzk=', 'Content-Length': '0'} 2024-03-04T17:56:29.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001efd8654b107fd4-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.789 DEBUG:boto:b'ppfahn-23999falseobjnulltrue2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLRrfB3qcS.tGP7kptdPy.BOFCgTU7cCfalse2024-03-04T17:54:35.381Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.789 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.790 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.790 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.790 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.790 DEBUG:boto:Method: GET 2024-03-04T17:56:29.790 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.790 DEBUG:boto:Data: 2024-03-04T17:56:29.790 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.790 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.790 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.790 DEBUG:boto:Params: {} 2024-03-04T17:56:29.790 DEBUG:boto:Token: None 2024-03-04T17:56:29.791 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.791 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE= 2024-03-04T17:56:29.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE=', 'Content-Length': '0'} 2024-03-04T17:56:29.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba4c16f3c9c04ad0-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.794 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.794 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.794 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.794 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.795 DEBUG:boto:Method: GET 2024-03-04T17:56:29.795 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.795 DEBUG:boto:Data: 2024-03-04T17:56:29.795 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.795 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.795 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.795 DEBUG:boto:Params: {} 2024-03-04T17:56:29.795 DEBUG:boto:Token: None 2024-03-04T17:56:29.795 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.796 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE= 2024-03-04T17:56:29.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE=', 'Content-Length': '0'} 2024-03-04T17:56:29.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca9923cded9a23d9-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.799 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.799 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.799 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.799 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.799 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.799 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.799 DEBUG:boto:Data: 2024-03-04T17:56:29.799 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.800 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.800 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.800 DEBUG:boto:Params: {} 2024-03-04T17:56:29.800 DEBUG:boto:Token: None 2024-03-04T17:56:29.800 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.800 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YvtdswZT/WJRHWItih+ayGZJ7ic= 2024-03-04T17:56:29.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YvtdswZT/WJRHWItih+ayGZJ7ic=', 'Content-Length': '0'} 2024-03-04T17:56:29.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000065d7552a20d4ccd8-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.803 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.803 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.804 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.804 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.804 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.804 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.804 DEBUG:boto:Data: 2024-03-04T17:56:29.804 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.804 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.804 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.804 DEBUG:boto:Params: {} 2024-03-04T17:56:29.804 DEBUG:boto:Token: None 2024-03-04T17:56:29.805 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.805 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YvtdswZT/WJRHWItih+ayGZJ7ic= 2024-03-04T17:56:29.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YvtdswZT/WJRHWItih+ayGZJ7ic=', 'Content-Length': '0'} 2024-03-04T17:56:29.808 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000007779d0fddf29e49-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.808 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.808 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.808 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.808 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.808 DEBUG:boto:Method: GET 2024-03-04T17:56:29.808 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.809 DEBUG:boto:Data: 2024-03-04T17:56:29.809 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.809 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.809 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.809 DEBUG:boto:Params: {} 2024-03-04T17:56:29.809 DEBUG:boto:Token: None 2024-03-04T17:56:29.809 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.809 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE= 2024-03-04T17:56:29.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE=', 'Content-Length': '0'} 2024-03-04T17:56:29.813 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee14e04d80d4453f-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.813 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.813 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.813 DEBUG:boto:path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.813 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.813 DEBUG:boto:Method: GET 2024-03-04T17:56:29.813 DEBUG:boto:Path: /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.814 DEBUG:boto:Data: 2024-03-04T17:56:29.814 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.814 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.814 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.814 DEBUG:boto:Params: {} 2024-03-04T17:56:29.814 DEBUG:boto:Token: None 2024-03-04T17:56:29.814 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=null 2024-03-04T17:56:29.814 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE= 2024-03-04T17:56:29.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:veR6SQqa8MHwbkdXe5ViGX+DxFE=', 'Content-Length': '0'} 2024-03-04T17:56:29.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e9708824103587fd-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.817 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.817 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.818 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.818 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.818 DEBUG:boto:Method: GET 2024-03-04T17:56:29.818 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.818 DEBUG:boto:Data: 2024-03-04T17:56:29.818 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.818 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.819 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.819 DEBUG:boto:Params: {} 2024-03-04T17:56:29.819 DEBUG:boto:Token: None 2024-03-04T17:56:29.819 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.819 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fed3813464450e32-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.868 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.868 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.869 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.869 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.869 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.869 DEBUG:boto:Method: GET 2024-03-04T17:56:29.869 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.869 DEBUG:boto:Data: 2024-03-04T17:56:29.869 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.869 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.869 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.870 DEBUG:boto:Params: {} 2024-03-04T17:56:29.870 DEBUG:boto:Token: None 2024-03-04T17:56:29.870 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.870 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002737157505e7be57-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.875 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.875 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.875 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.875 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.875 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.875 DEBUG:boto:Data: 2024-03-04T17:56:29.876 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.876 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.876 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.876 DEBUG:boto:Params: {} 2024-03-04T17:56:29.876 DEBUG:boto:Token: None 2024-03-04T17:56:29.876 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.877 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw= 2024-03-04T17:56:29.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw=', 'Content-Length': '0'} 2024-03-04T17:56:29.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e88731db659b7c99-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.880 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.880 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.880 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.880 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.880 DEBUG:boto:Data: 2024-03-04T17:56:29.880 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.881 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.881 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.881 DEBUG:boto:Params: {} 2024-03-04T17:56:29.881 DEBUG:boto:Token: None 2024-03-04T17:56:29.881 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.881 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw= 2024-03-04T17:56:29.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lgbqbNLQ/8meoW3TC48Q0tnOmpw=', 'Content-Length': '0'} 2024-03-04T17:56:29.885 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc46dbe44f04df93-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.885 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.885 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.885 DEBUG:boto:Method: GET 2024-03-04T17:56:29.885 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.885 DEBUG:boto:Data: 2024-03-04T17:56:29.886 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.886 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.886 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.886 DEBUG:boto:Params: {} 2024-03-04T17:56:29.886 DEBUG:boto:Token: None 2024-03-04T17:56:29.886 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.886 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0= 2024-03-04T17:56:29.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0=', 'Content-Length': '0'} 2024-03-04T17:56:29.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000304d58a90abad0b1-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.889 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.890 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.890 DEBUG:boto:Method: GET 2024-03-04T17:56:29.890 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:29.890 DEBUG:boto:Data: 2024-03-04T17:56:29.890 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.890 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.890 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.890 DEBUG:boto:Params: {} 2024-03-04T17:56:29.890 DEBUG:boto:Token: None 2024-03-04T17:56:29.891 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj 2024-03-04T17:56:29.891 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0= 2024-03-04T17:56:29.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:avDjT3ZT0Q7abGx+87qIS0QbvU0=', 'Content-Length': '0'} 2024-03-04T17:56:29.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ccc3879e3eee7031-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.894 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.894 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.894 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.894 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.894 DEBUG:boto:Method: GET 2024-03-04T17:56:29.894 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.894 DEBUG:boto:Data: 2024-03-04T17:56:29.895 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.895 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.895 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.895 DEBUG:boto:Params: {} 2024-03-04T17:56:29.895 DEBUG:boto:Token: None 2024-03-04T17:56:29.895 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.895 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077d385aae08ce795-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.940 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.941 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:29.941 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:29.941 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.941 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:29.941 DEBUG:boto:Method: GET 2024-03-04T17:56:29.941 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:29.941 DEBUG:boto:Data: 2024-03-04T17:56:29.941 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.941 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.941 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.941 DEBUG:boto:Params: {} 2024-03-04T17:56:29.942 DEBUG:boto:Token: None 2024-03-04T17:56:29.942 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/ 2024-03-04T17:56:29.942 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag= 2024-03-04T17:56:29.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VscDogm8eUjnWx353NsUFGMgcag=', 'Content-Length': '0'} 2024-03-04T17:56:29.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5d4c0a06e159e30-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.988 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:29.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:29.988 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.989 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.989 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.989 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.989 DEBUG:boto:Method: GET 2024-03-04T17:56:29.989 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.989 DEBUG:boto:Data: 2024-03-04T17:56:29.989 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.989 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.989 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.989 DEBUG:boto:Params: {} 2024-03-04T17:56:29.989 DEBUG:boto:Token: None 2024-03-04T17:56:29.990 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.990 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY= 2024-03-04T17:56:29.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY=', 'Content-Length': '0'} 2024-03-04T17:56:29.993 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002a7eefda3d6e91e6-0065e60b4d-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.993 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.993 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.993 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.993 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.994 DEBUG:boto:Method: GET 2024-03-04T17:56:29.994 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.994 DEBUG:boto:Data: 2024-03-04T17:56:29.994 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.994 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:29.994 DEBUG:boto:Port: 8000 2024-03-04T17:56:29.994 DEBUG:boto:Params: {} 2024-03-04T17:56:29.994 DEBUG:boto:Token: None 2024-03-04T17:56:29.994 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.995 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY= 2024-03-04T17:56:29.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qDqa8LYbrdBuSG3KrYLAPrc2gdY=', 'Content-Length': '0'} 2024-03-04T17:56:29.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee8f31f8bbfa622a-0065e60b4d-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:29.998 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:29.998 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:29.998 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.998 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.998 DEBUG:boto:Method: HEAD 2024-03-04T17:56:29.998 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.998 DEBUG:boto:Data: 2024-03-04T17:56:29.999 DEBUG:boto:Headers: {} 2024-03-04T17:56:29.999 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:29.999 DEBUG:boto:Port: 8001 2024-03-04T17:56:29.999 DEBUG:boto:Params: {} 2024-03-04T17:56:29.999 DEBUG:boto:Token: None 2024-03-04T17:56:29.999 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:29 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:29.999 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:W0/I4J1/s4zhQ/E4Rrm9qG9E4L4= 2024-03-04T17:56:29.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:29 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:W0/I4J1/s4zhQ/E4Rrm9qG9E4L4=', 'Content-Length': '0'} 2024-03-04T17:56:30.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b4bd7bcfb52400b5-0065e60b4e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.002 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:30.002 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:30.002 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:30.002 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:30.002 DEBUG:boto:Method: HEAD 2024-03-04T17:56:30.003 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:30.003 DEBUG:boto:Data: 2024-03-04T17:56:30.003 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.003 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:30.003 DEBUG:boto:Port: 8000 2024-03-04T17:56:30.003 DEBUG:boto:Params: {} 2024-03-04T17:56:30.003 DEBUG:boto:Token: None 2024-03-04T17:56:30.003 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:30.003 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:G6Z/EQIDjmieqjrcpF1Dk/YewbA= 2024-03-04T17:56:30.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:G6Z/EQIDjmieqjrcpF1Dk/YewbA=', 'Content-Length': '0'} 2024-03-04T17:56:30.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e87bd70d12db915c-0065e60b4e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:30.007 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:30.007 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:30.007 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:30.007 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:30.007 DEBUG:boto:Method: GET 2024-03-04T17:56:30.007 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:30.007 DEBUG:boto:Data: 2024-03-04T17:56:30.007 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.007 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:30.007 DEBUG:boto:Port: 8001 2024-03-04T17:56:30.007 DEBUG:boto:Params: {} 2024-03-04T17:56:30.008 DEBUG:boto:Token: None 2024-03-04T17:56:30.008 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:30.008 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZfjLvx40tKhSS/TKLYIR/RgoIPs= 2024-03-04T17:56:30.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZfjLvx40tKhSS/TKLYIR/RgoIPs=', 'Content-Length': '0'} 2024-03-04T17:56:30.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e928c5b5430cc8d-0065e60b4e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.011 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:30.011 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:30.011 DEBUG:boto:path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:30.011 DEBUG:boto:auth_path=/ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:30.012 DEBUG:boto:Method: GET 2024-03-04T17:56:30.012 DEBUG:boto:Path: /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:30.012 DEBUG:boto:Data: 2024-03-04T17:56:30.012 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.012 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:30.012 DEBUG:boto:Port: 8000 2024-03-04T17:56:30.012 DEBUG:boto:Params: {} 2024-03-04T17:56:30.012 DEBUG:boto:Token: None 2024-03-04T17:56:30.012 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-23/obj?versionId=LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC 2024-03-04T17:56:30.012 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZfjLvx40tKhSS/TKLYIR/RgoIPs= 2024-03-04T17:56:30.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZfjLvx40tKhSS/TKLYIR/RgoIPs=', 'Content-Length': '0'} 2024-03-04T17:56:30.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:54:35 GMT'), ('x-amz-version-id', 'LRrfB3qcS.tGP7kptdPy.BOFCgTU7cC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045ab4d90cc36ab53-0065e60b4e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.016 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:30.016 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:30.016 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:30.016 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:30.016 DEBUG:boto:Method: GET 2024-03-04T17:56:30.016 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:30.016 DEBUG:boto:Data: 2024-03-04T17:56:30.017 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.017 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:30.017 DEBUG:boto:Port: 8001 2024-03-04T17:56:30.017 DEBUG:boto:Params: {} 2024-03-04T17:56:30.017 DEBUG:boto:Token: None 2024-03-04T17:56:30.017 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-23/ 2024-03-04T17:56:30.017 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E3WVk8rFO0prbGZd4P+Ef4e7g7A= 2024-03-04T17:56:30.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E3WVk8rFO0prbGZd4P+Ef4e7g7A=', 'Content-Length': '0'} 2024-03-04T17:56:30.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5804a12bd003beb-0065e60b4e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.064 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:30.065 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:30.065 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:30.065 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:30.065 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:30.065 DEBUG:boto:Method: GET 2024-03-04T17:56:30.065 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:30.065 DEBUG:boto:Data: 2024-03-04T17:56:30.065 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.065 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:30.065 DEBUG:boto:Port: 8000 2024-03-04T17:56:30.065 DEBUG:boto:Params: {} 2024-03-04T17:56:30.066 DEBUG:boto:Token: None 2024-03-04T17:56:30.066 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-23/ 2024-03-04T17:56:30.066 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E3WVk8rFO0prbGZd4P+Ef4e7g7A= 2024-03-04T17:56:30.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E3WVk8rFO0prbGZd4P+Ef4e7g7A=', 'Content-Length': '0'} 2024-03-04T17:56:30.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e4074079a03be1f-0065e60b4e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.071 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:30.071 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:30.071 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:30.071 DEBUG:boto:Method: HEAD 2024-03-04T17:56:30.072 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:30.072 DEBUG:boto:Data: 2024-03-04T17:56:30.072 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.072 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:30.072 DEBUG:boto:Port: 8001 2024-03-04T17:56:30.072 DEBUG:boto:Params: {} 2024-03-04T17:56:30.072 DEBUG:boto:Token: None 2024-03-04T17:56:30.072 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-23/obj 2024-03-04T17:56:30.072 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iEsQWO8XEoUa4vQn9snh/f1W0VI= 2024-03-04T17:56:30.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iEsQWO8XEoUa4vQn9snh/f1W0VI=', 'Content-Length': '0'} 2024-03-04T17:56:30.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a891c157d01463e3-0065e60b4e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.075 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:30.076 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:30.076 DEBUG:boto:Method: HEAD 2024-03-04T17:56:30.076 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:30.076 DEBUG:boto:Data: 2024-03-04T17:56:30.076 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.076 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:30.076 DEBUG:boto:Port: 8000 2024-03-04T17:56:30.076 DEBUG:boto:Params: {} 2024-03-04T17:56:30.076 DEBUG:boto:Token: None 2024-03-04T17:56:30.076 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-23/obj 2024-03-04T17:56:30.077 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:iEsQWO8XEoUa4vQn9snh/f1W0VI= 2024-03-04T17:56:30.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:iEsQWO8XEoUa4vQn9snh/f1W0VI=', 'Content-Length': '0'} 2024-03-04T17:56:30.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a32b06bbaee49941-0065e60b4e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:56:30.080 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:30.080 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:30.080 DEBUG:boto:Method: GET 2024-03-04T17:56:30.080 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:30.080 DEBUG:boto:Data: 2024-03-04T17:56:30.080 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.080 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:30.080 DEBUG:boto:Port: 8001 2024-03-04T17:56:30.080 DEBUG:boto:Params: {} 2024-03-04T17:56:30.081 DEBUG:boto:Token: None 2024-03-04T17:56:30.081 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-23/obj 2024-03-04T17:56:30.081 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gpSqEbbHhPuJuvmPU5rwoYjnryQ= 2024-03-04T17:56:30.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gpSqEbbHhPuJuvmPU5rwoYjnryQ=', 'Content-Length': '0'} 2024-03-04T17:56:30.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b6a3fa533481df2b-0065e60b4e-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.084 DEBUG:boto:path=/ppfahn-23/obj 2024-03-04T17:56:30.084 DEBUG:boto:auth_path=/ppfahn-23/obj 2024-03-04T17:56:30.084 DEBUG:boto:Method: GET 2024-03-04T17:56:30.084 DEBUG:boto:Path: /ppfahn-23/obj 2024-03-04T17:56:30.084 DEBUG:boto:Data: 2024-03-04T17:56:30.084 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.084 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:30.084 DEBUG:boto:Port: 8000 2024-03-04T17:56:30.084 DEBUG:boto:Params: {} 2024-03-04T17:56:30.085 DEBUG:boto:Token: None 2024-03-04T17:56:30.085 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-23/obj 2024-03-04T17:56:30.085 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gpSqEbbHhPuJuvmPU5rwoYjnryQ= 2024-03-04T17:56:30.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gpSqEbbHhPuJuvmPU5rwoYjnryQ=', 'Content-Length': '0'} 2024-03-04T17:56:30.088 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:55:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-23:d96636ac-054d-40e3-90f3-e3912da784b2.4349.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000be680f95ca1113db-0065e60b4e-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.088 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:30.088 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:30.088 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:30.089 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:30.089 DEBUG:boto:Method: GET 2024-03-04T17:56:30.089 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:30.089 DEBUG:boto:Data: 2024-03-04T17:56:30.089 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.089 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:30.089 DEBUG:boto:Port: 8001 2024-03-04T17:56:30.089 DEBUG:boto:Params: {} 2024-03-04T17:56:30.089 DEBUG:boto:Token: None 2024-03-04T17:56:30.089 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-23/ 2024-03-04T17:56:30.090 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E3WVk8rFO0prbGZd4P+Ef4e7g7A= 2024-03-04T17:56:30.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E3WVk8rFO0prbGZd4P+Ef4e7g7A=', 'Content-Length': '0'} 2024-03-04T17:56:30.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013d262236308debe-0065e60b4e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.094 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:30.094 DEBUG:boto:path=/ppfahn-23/ 2024-03-04T17:56:30.094 DEBUG:boto:auth_path=/ppfahn-23/ 2024-03-04T17:56:30.094 DEBUG:boto:path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:30.094 DEBUG:boto:auth_path=/ppfahn-23/?prefix=obj 2024-03-04T17:56:30.094 DEBUG:boto:Method: GET 2024-03-04T17:56:30.094 DEBUG:boto:Path: /ppfahn-23/?prefix=obj 2024-03-04T17:56:30.095 DEBUG:boto:Data: 2024-03-04T17:56:30.095 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.095 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:30.095 DEBUG:boto:Port: 8000 2024-03-04T17:56:30.095 DEBUG:boto:Params: {} 2024-03-04T17:56:30.095 DEBUG:boto:Token: None 2024-03-04T17:56:30.095 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-23/ 2024-03-04T17:56:30.095 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:E3WVk8rFO0prbGZd4P+Ef4e7g7A= 2024-03-04T17:56:30.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E3WVk8rFO0prbGZd4P+Ef4e7g7A=', 'Content-Length': '0'} 2024-03-04T17:56:30.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9c5155f6cebce07-0065e60b4e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.140 DEBUG:boto:b'ppfahn-23obj1000falseobj2024-03-04T17:55:33.004Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:56:30.140 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-23 zones={test-zone3, test-zone2} 2024-03-04T17:56:30.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2024-03-04T17:56:30.142 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-24 2024-03-04T17:56:30.143 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:56:30.143 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:56:30.143 DEBUG:boto:Method: PUT 2024-03-04T17:56:30.143 DEBUG:boto:Path: /ppfahn-24/ 2024-03-04T17:56:30.143 DEBUG:boto:Data: 2024-03-04T17:56:30.143 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.143 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:30.143 DEBUG:boto:Port: 8000 2024-03-04T17:56:30.143 DEBUG:boto:Params: {} 2024-03-04T17:56:30.143 DEBUG:boto:Token: None 2024-03-04T17:56:30.144 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-24/ 2024-03-04T17:56:30.144 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ijjthNMhiieiWypHVyCEVKNzG0Y= 2024-03-04T17:56:30.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ijjthNMhiieiWypHVyCEVKNzG0Y=', 'Content-Length': '0'} 2024-03-04T17:56:30.177 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000108617f6e506a4ff-0065e60b4e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.177 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-25 2024-03-04T17:56:30.177 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:56:30.177 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:56:30.177 DEBUG:boto:Method: PUT 2024-03-04T17:56:30.177 DEBUG:boto:Path: /ppfahn-25/ 2024-03-04T17:56:30.178 DEBUG:boto:Data: 2024-03-04T17:56:30.178 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.178 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:30.178 DEBUG:boto:Port: 8000 2024-03-04T17:56:30.178 DEBUG:boto:Params: {} 2024-03-04T17:56:30.178 DEBUG:boto:Token: None 2024-03-04T17:56:30.178 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-25/ 2024-03-04T17:56:30.178 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:X8qXVd/JQYI0hmEJBi2E3PaLTs8= 2024-03-04T17:56:30.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:X8qXVd/JQYI0hmEJBi2E3PaLTs8=', 'Content-Length': '0'} 2024-03-04T17:56:30.239 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006006c3ecb838529a-0065e60b4e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.239 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-26 2024-03-04T17:56:30.240 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:56:30.240 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:56:30.240 DEBUG:boto:Method: PUT 2024-03-04T17:56:30.240 DEBUG:boto:Path: /ppfahn-26/ 2024-03-04T17:56:30.240 DEBUG:boto:Data: 2024-03-04T17:56:30.240 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.240 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:30.240 DEBUG:boto:Port: 8001 2024-03-04T17:56:30.240 DEBUG:boto:Params: {} 2024-03-04T17:56:30.240 DEBUG:boto:Token: None 2024-03-04T17:56:30.240 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-26/ 2024-03-04T17:56:30.241 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Nhzy68hzXd2DvFZ9YmMSRjIAZCY= 2024-03-04T17:56:30.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Nhzy68hzXd2DvFZ9YmMSRjIAZCY=', 'Content-Length': '0'} 2024-03-04T17:56:30.297 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000039cc68de4ec0ea66-0065e60b4e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.297 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:56:30.297 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:56:30.297 DEBUG:boto:path=/ppfahn-24/?versioning 2024-03-04T17:56:30.297 DEBUG:boto:auth_path=/ppfahn-24/?versioning 2024-03-04T17:56:30.297 DEBUG:boto:Method: PUT 2024-03-04T17:56:30.297 DEBUG:boto:Path: /ppfahn-24/?versioning 2024-03-04T17:56:30.297 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:56:30.297 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.298 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:30.298 DEBUG:boto:Port: 8000 2024-03-04T17:56:30.298 DEBUG:boto:Params: {} 2024-03-04T17:56:30.298 DEBUG:boto:Token: None 2024-03-04T17:56:30.298 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-24/?versioning 2024-03-04T17:56:30.298 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:t4AegX7W+mfhZ6Sbm+Fs6+VnpPM= 2024-03-04T17:56:30.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:t4AegX7W+mfhZ6Sbm+Fs6+VnpPM=', 'Content-Length': '228'} 2024-03-04T17:56:30.309 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000055787008d6265bde-0065e60b4e-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.309 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:56:30.309 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:56:30.309 DEBUG:boto:path=/ppfahn-25/?versioning 2024-03-04T17:56:30.310 DEBUG:boto:auth_path=/ppfahn-25/?versioning 2024-03-04T17:56:30.310 DEBUG:boto:Method: PUT 2024-03-04T17:56:30.310 DEBUG:boto:Path: /ppfahn-25/?versioning 2024-03-04T17:56:30.310 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:56:30.310 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.310 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:30.310 DEBUG:boto:Port: 8000 2024-03-04T17:56:30.310 DEBUG:boto:Params: {} 2024-03-04T17:56:30.310 DEBUG:boto:Token: None 2024-03-04T17:56:30.310 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-25/?versioning 2024-03-04T17:56:30.310 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3QpMRpzXEpBcfCtRvz5kzNyicIk= 2024-03-04T17:56:30.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3QpMRpzXEpBcfCtRvz5kzNyicIk=', 'Content-Length': '228'} 2024-03-04T17:56:30.329 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007c55af6749887232-0065e60b4e-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.329 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:56:30.329 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:56:30.329 DEBUG:boto:path=/ppfahn-26/?versioning 2024-03-04T17:56:30.330 DEBUG:boto:auth_path=/ppfahn-26/?versioning 2024-03-04T17:56:30.330 DEBUG:boto:Method: PUT 2024-03-04T17:56:30.330 DEBUG:boto:Path: /ppfahn-26/?versioning 2024-03-04T17:56:30.330 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:56:30.330 DEBUG:boto:Headers: {} 2024-03-04T17:56:30.330 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:30.330 DEBUG:boto:Port: 8001 2024-03-04T17:56:30.330 DEBUG:boto:Params: {} 2024-03-04T17:56:30.330 DEBUG:boto:Token: None 2024-03-04T17:56:30.330 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:56:30 GMT /ppfahn-26/?versioning 2024-03-04T17:56:30.330 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ANYMbZ0G3T9AoDsWzUQhrrb+4Kk= 2024-03-04T17:56:30.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:30 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ANYMbZ0G3T9AoDsWzUQhrrb+4Kk=', 'Content-Length': '228'} 2024-03-04T17:56:30.360 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b11c7f584769dbf1-0065e60b4e-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:30.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:30.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:30.361 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:30.403 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:30.404 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:30.457 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:56:30.480 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709574990.354481_227.1","last_update":"2024-03-04T17:56:30.354481Z"}] 2024-03-04T17:56:30.480 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574990.354481_227.1'} 2024-03-04T17:56:30.480 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:56:30.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:30.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:30.481 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:30.523 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:30.523 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:30.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:56:30.589+0000 7f3c77f66980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:30.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:56:30.589+0000 7f3c77f66980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:30.610 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574990.161090_210.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:56:30.161090Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:30Z"} 2024-03-04T17:56:30.611 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574990.161090_210.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:56:30.161090Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:30Z"} 2024-03-04T17:56:30.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574990.161090_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:56:30.161090Z', 'realm_epoch': 2}}] 2024-03-04T17:56:30.611 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574990.354481_227.1'} 2024-03-04T17:56:30.611 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574990.161090_210.1'} 2024-03-04T17:56:30.611 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709574990.354481_227.1 target=1_1709574990.161090_210.1 2024-03-04T17:56:35.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:35.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:35.612 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:35.654 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:35.654 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:35.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:56:35.713+0000 7f5cd4113980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:35.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:56:35.713+0000 7f5cd4113980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:35.733 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574990.354481_227.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:56:30.354481Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:35Z"} 2024-03-04T17:56:35.733 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574990.354481_227.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:56:30.354481Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:35Z"} 2024-03-04T17:56:35.734 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574990.354481_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:56:30.354481Z', 'realm_epoch': 2}}] 2024-03-04T17:56:35.734 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574990.354481_227.1'} 2024-03-04T17:56:35.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574990.354481_227.1'} 2024-03-04T17:56:35.734 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:56:35.734 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:56:35.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:35.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:35.734 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:35.777 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:35.777 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:35.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:35.830+0000 7f43047dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:35.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:35.834+0000 7f43047dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:35.854 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574990.354481_227.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:56:30.354481Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:35Z"} 2024-03-04T17:56:35.855 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574990.354481_227.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:56:30.354481Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:35Z"} 2024-03-04T17:56:35.855 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574990.354481_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:56:30.354481Z', 'realm_epoch': 2}}] 2024-03-04T17:56:35.855 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574990.354481_227.1'} 2024-03-04T17:56:35.855 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574990.354481_227.1'} 2024-03-04T17:56:35.855 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:56:35.855 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:56:35.855 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:56:35.855 DEBUG:boto:Method: HEAD 2024-03-04T17:56:35.856 DEBUG:boto:Path: /ppfahn-24/ 2024-03-04T17:56:35.856 DEBUG:boto:Data: 2024-03-04T17:56:35.856 DEBUG:boto:Headers: {} 2024-03-04T17:56:35.856 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:35.856 DEBUG:boto:Port: 8000 2024-03-04T17:56:35.856 DEBUG:boto:Params: {} 2024-03-04T17:56:35.856 DEBUG:boto:Token: None 2024-03-04T17:56:35.856 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:35 GMT /ppfahn-24/ 2024-03-04T17:56:35.857 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lzUWR3Cqxq7iCMl2v8ei/onLNAY= 2024-03-04T17:56:35.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lzUWR3Cqxq7iCMl2v8ei/onLNAY=', 'Content-Length': '0'} 2024-03-04T17:56:35.859 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a2aae5de791e441-0065e60b53-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:35.860 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:56:35.860 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:56:35.860 DEBUG:boto:Method: PUT 2024-03-04T17:56:35.860 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:56:35.860 DEBUG:boto:Data: 2024-03-04T17:56:35.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T17:56:35.860 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:35.860 DEBUG:boto:Port: 8000 2024-03-04T17:56:35.860 DEBUG:boto:Params: {} 2024-03-04T17:56:35.860 DEBUG:boto:Token: None 2024-03-04T17:56:35.861 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 17:56:35 GMT /ppfahn-24/obj 2024-03-04T17:56:35.861 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ansHO0o8b6lT7nzMRapVxNV3ilw= 2024-03-04T17:56:35.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:56:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ansHO0o8b6lT7nzMRapVxNV3ilw='} 2024-03-04T17:56:35.873 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-amz-request-id', 'tx000004fca03fbf0707348-0065e60b53-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:35.873 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:56:35.873 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:56:35.873 DEBUG:boto:Method: HEAD 2024-03-04T17:56:35.873 DEBUG:boto:Path: /ppfahn-24/ 2024-03-04T17:56:35.874 DEBUG:boto:Data: 2024-03-04T17:56:35.874 DEBUG:boto:Headers: {} 2024-03-04T17:56:35.874 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:35.874 DEBUG:boto:Port: 8000 2024-03-04T17:56:35.874 DEBUG:boto:Params: {} 2024-03-04T17:56:35.874 DEBUG:boto:Token: None 2024-03-04T17:56:35.874 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:35 GMT /ppfahn-24/ 2024-03-04T17:56:35.874 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lzUWR3Cqxq7iCMl2v8ei/onLNAY= 2024-03-04T17:56:35.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lzUWR3Cqxq7iCMl2v8ei/onLNAY=', 'Content-Length': '0'} 2024-03-04T17:56:35.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000114f860532919fd9-0065e60b53-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:35.877 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:56:35.877 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:56:35.877 DEBUG:boto:Method: DELETE 2024-03-04T17:56:35.877 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:56:35.877 DEBUG:boto:Data: 2024-03-04T17:56:35.877 DEBUG:boto:Headers: {} 2024-03-04T17:56:35.877 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:56:35.877 DEBUG:boto:Port: 8000 2024-03-04T17:56:35.877 DEBUG:boto:Params: {} 2024-03-04T17:56:35.878 DEBUG:boto:Token: None 2024-03-04T17:56:35.878 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:56:35 GMT /ppfahn-24/obj 2024-03-04T17:56:35.878 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f9bsym9+RvdsYQhk209yz2LaskA= 2024-03-04T17:56:35.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f9bsym9+RvdsYQhk209yz2LaskA=', 'Content-Length': '0'} 2024-03-04T17:56:35.885 DEBUG:boto:Response headers: [('x-amz-version-id', 'ZGveJeVHJWiuK2dXm0p0CTQ1hzKAPBf'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000096cc727f35b2bd94-0065e60b53-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:35.885 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:56:35.885 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:56:35.885 DEBUG:boto:Method: HEAD 2024-03-04T17:56:35.885 DEBUG:boto:Path: /ppfahn-25/ 2024-03-04T17:56:35.885 DEBUG:boto:Data: 2024-03-04T17:56:35.886 DEBUG:boto:Headers: {} 2024-03-04T17:56:35.886 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:35.886 DEBUG:boto:Port: 8000 2024-03-04T17:56:35.886 DEBUG:boto:Params: {} 2024-03-04T17:56:35.886 DEBUG:boto:Token: None 2024-03-04T17:56:35.886 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:35 GMT /ppfahn-25/ 2024-03-04T17:56:35.886 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mwxOmoUStm3FPDGD9NozGm+8jkw= 2024-03-04T17:56:35.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mwxOmoUStm3FPDGD9NozGm+8jkw=', 'Content-Length': '0'} 2024-03-04T17:56:35.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f34681dae6d865fa-0065e60b53-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:35.891 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:56:35.892 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:56:35.892 DEBUG:boto:Method: PUT 2024-03-04T17:56:35.892 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:56:35.892 DEBUG:boto:Data: 2024-03-04T17:56:35.892 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T17:56:35.892 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:35.892 DEBUG:boto:Port: 8000 2024-03-04T17:56:35.892 DEBUG:boto:Params: {} 2024-03-04T17:56:35.892 DEBUG:boto:Token: None 2024-03-04T17:56:35.893 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 17:56:35 GMT /ppfahn-25/obj 2024-03-04T17:56:35.893 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lL2Vcs1zdnSEIne36yFulA0h3lY= 2024-03-04T17:56:35.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:56:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lL2Vcs1zdnSEIne36yFulA0h3lY='} 2024-03-04T17:56:35.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-amz-request-id', 'tx0000082152faa7186161c-0065e60b53-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:35.920 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:56:35.920 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:56:35.920 DEBUG:boto:Method: HEAD 2024-03-04T17:56:35.920 DEBUG:boto:Path: /ppfahn-25/ 2024-03-04T17:56:35.920 DEBUG:boto:Data: 2024-03-04T17:56:35.920 DEBUG:boto:Headers: {} 2024-03-04T17:56:35.920 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:35.921 DEBUG:boto:Port: 8000 2024-03-04T17:56:35.921 DEBUG:boto:Params: {} 2024-03-04T17:56:35.921 DEBUG:boto:Token: None 2024-03-04T17:56:35.921 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:35 GMT /ppfahn-25/ 2024-03-04T17:56:35.921 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mwxOmoUStm3FPDGD9NozGm+8jkw= 2024-03-04T17:56:35.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mwxOmoUStm3FPDGD9NozGm+8jkw=', 'Content-Length': '0'} 2024-03-04T17:56:35.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bde24a07af4e4e46-0065e60b53-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:35.926 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:56:35.926 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:56:35.926 DEBUG:boto:Method: DELETE 2024-03-04T17:56:35.926 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:56:35.926 DEBUG:boto:Data: 2024-03-04T17:56:35.926 DEBUG:boto:Headers: {} 2024-03-04T17:56:35.926 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:56:35.926 DEBUG:boto:Port: 8000 2024-03-04T17:56:35.926 DEBUG:boto:Params: {} 2024-03-04T17:56:35.926 DEBUG:boto:Token: None 2024-03-04T17:56:35.927 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:56:35 GMT /ppfahn-25/obj 2024-03-04T17:56:35.927 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9/yyU0AxQ2xD+t2NqdOkNi/HbDY= 2024-03-04T17:56:35.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9/yyU0AxQ2xD+t2NqdOkNi/HbDY=', 'Content-Length': '0'} 2024-03-04T17:56:35.943 DEBUG:boto:Response headers: [('x-amz-version-id', 'y476gJad3qtgVOCqSNyCSIsViYsdZnM'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000b5c3dfc0c621386c-0065e60b53-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:35.944 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:56:35.944 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:56:35.944 DEBUG:boto:Method: HEAD 2024-03-04T17:56:35.944 DEBUG:boto:Path: /ppfahn-26/ 2024-03-04T17:56:35.944 DEBUG:boto:Data: 2024-03-04T17:56:35.944 DEBUG:boto:Headers: {} 2024-03-04T17:56:35.944 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:35.944 DEBUG:boto:Port: 8001 2024-03-04T17:56:35.944 DEBUG:boto:Params: {} 2024-03-04T17:56:35.945 DEBUG:boto:Token: None 2024-03-04T17:56:35.945 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:35 GMT /ppfahn-26/ 2024-03-04T17:56:35.945 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:awPC/K7fh2mKI6DuIMPWLUHfKtg= 2024-03-04T17:56:35.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:awPC/K7fh2mKI6DuIMPWLUHfKtg=', 'Content-Length': '0'} 2024-03-04T17:56:35.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000196f1e47b69df48d-0065e60b53-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:35.949 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:56:35.949 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:56:35.950 DEBUG:boto:Method: PUT 2024-03-04T17:56:35.950 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:56:35.950 DEBUG:boto:Data: 2024-03-04T17:56:35.950 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T17:56:35.950 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:35.950 DEBUG:boto:Port: 8001 2024-03-04T17:56:35.950 DEBUG:boto:Params: {} 2024-03-04T17:56:35.950 DEBUG:boto:Token: None 2024-03-04T17:56:35.950 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 17:56:35 GMT /ppfahn-26/obj 2024-03-04T17:56:35.950 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9X/4iMy9PsM80Yyixotp46Cq77c= 2024-03-04T17:56:35.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:56:35 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9X/4iMy9PsM80Yyixotp46Cq77c='} 2024-03-04T17:56:36.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-amz-request-id', 'tx00000928fbfd0ecb7c74a-0065e60b53-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:36.000 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:56:36.000 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:56:36.000 DEBUG:boto:Method: HEAD 2024-03-04T17:56:36.000 DEBUG:boto:Path: /ppfahn-26/ 2024-03-04T17:56:36.001 DEBUG:boto:Data: 2024-03-04T17:56:36.001 DEBUG:boto:Headers: {} 2024-03-04T17:56:36.001 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:36.001 DEBUG:boto:Port: 8001 2024-03-04T17:56:36.001 DEBUG:boto:Params: {} 2024-03-04T17:56:36.001 DEBUG:boto:Token: None 2024-03-04T17:56:36.001 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:56:36 GMT /ppfahn-26/ 2024-03-04T17:56:36.001 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CPsSJrVp1jWcjxXOZk1IOvz6gzw= 2024-03-04T17:56:36.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CPsSJrVp1jWcjxXOZk1IOvz6gzw=', 'Content-Length': '0'} 2024-03-04T17:56:36.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003141f4f9fd6d7416-0065e60b54-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:56:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:36.006 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:56:36.006 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:56:36.006 DEBUG:boto:Method: DELETE 2024-03-04T17:56:36.006 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:56:36.006 DEBUG:boto:Data: 2024-03-04T17:56:36.006 DEBUG:boto:Headers: {} 2024-03-04T17:56:36.006 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:56:36.006 DEBUG:boto:Port: 8001 2024-03-04T17:56:36.006 DEBUG:boto:Params: {} 2024-03-04T17:56:36.006 DEBUG:boto:Token: None 2024-03-04T17:56:36.006 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:56:36 GMT /ppfahn-26/obj 2024-03-04T17:56:36.007 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lgYNTUV/gkSf39hArefw0Zgk/GA= 2024-03-04T17:56:36.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:56:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lgYNTUV/gkSf39hArefw0Zgk/GA=', 'Content-Length': '0'} 2024-03-04T17:56:36.023 DEBUG:boto:Response headers: [('x-amz-version-id', '4qnz-Ulcde6cJDXxKloMY8QwW.xzlq2'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000049cfa6a7184ef394-0065e60b54-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:56:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:56:36.023 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-24 2024-03-04T17:56:36.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:56:36.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:56:36.023 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:56:36.073 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:36.073 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:36.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:56:36.137+0000 7f8e563b3980 1 waiting to reach incremental sync.. 2024-03-04T17:56:41.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:56:41.137+0000 7f8e563b3980 1 waiting to reach incremental sync.. 2024-03-04T17:56:46.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:56:46.141+0000 7f8e563b3980 1 waiting to reach incremental sync.. 2024-03-04T17:56:51.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:56:51.141+0000 7f8e563b3980 1 waiting to reach incremental sync.. 2024-03-04T17:56:56.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:56:56.145+0000 7f8e563b3980 1 bucket sync caught up with source: 2024-03-04T17:56:56.141 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000003.124.12, , ] 2024-03-04T17:56:56.141 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000003.124.12, , ] 2024-03-04T17:56:56.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:56:56.145+0000 7f8e563b3980 0 bucket checkpoint complete 2024-03-04T17:56:56.184 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-24 2024-03-04T17:56:56.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:56:56.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:56:56.184 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:56:56.234 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:56.234 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:56.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:56.293+0000 7f9d31c3b980 1 bucket sync caught up with source: 2024-03-04T17:56:56.295 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000003.124.12, , ] 2024-03-04T17:56:56.295 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.124.12, , ] 2024-03-04T17:56:56.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:56.293+0000 7f9d31c3b980 0 bucket checkpoint complete 2024-03-04T17:56:56.317 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-24 2024-03-04T17:56:56.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:56:56.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:56:56.317 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:56:56.360 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:56.360 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:56.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:56:56.417+0000 7fc5805d0980 1 waiting to reach incremental sync.. 2024-03-04T17:57:01.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:01.417+0000 7fc5805d0980 1 waiting to reach incremental sync.. 2024-03-04T17:57:06.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:06.421+0000 7fc5805d0980 1 bucket sync caught up with source: 2024-03-04T17:57:06.423 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000002.133.6, , ] 2024-03-04T17:57:06.423 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000002.133.6, , ] 2024-03-04T17:57:06.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:06.421+0000 7fc5805d0980 0 bucket checkpoint complete 2024-03-04T17:57:06.464 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-24 2024-03-04T17:57:06.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:06.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:06.465 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:06.555 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:06.555 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:06.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:06.621+0000 7f34fd9c8980 1 waiting to reach incremental sync.. 2024-03-04T17:57:11.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:11.621+0000 7f34fd9c8980 1 bucket sync caught up with source: 2024-03-04T17:57:11.623 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000002.133.6, , ] 2024-03-04T17:57:11.623 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000002.133.6, , ] 2024-03-04T17:57:11.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:11.621+0000 7f34fd9c8980 0 bucket checkpoint complete 2024-03-04T17:57:11.656 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-24 2024-03-04T17:57:11.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:11.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:11.656 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:11.704 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:11.704 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:11.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:11.765+0000 7fbb8b7fe980 1 bucket sync caught up with source: 2024-03-04T17:57:11.765 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000002.137.6, , ] 2024-03-04T17:57:11.765 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000002.137.6, , ] 2024-03-04T17:57:11.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:11.765+0000 7fbb8b7fe980 0 bucket checkpoint complete 2024-03-04T17:57:11.788 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-24 2024-03-04T17:57:11.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:11.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:11.788 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:11.839 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:11.840 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:11.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:11.908+0000 7f1871b56980 1 bucket sync caught up with source: 2024-03-04T17:57:11.905 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000002.137.6, , ] 2024-03-04T17:57:11.905 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000002.137.6, , ] 2024-03-04T17:57:11.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:11.908+0000 7f1871b56980 0 bucket checkpoint complete 2024-03-04T17:57:11.927 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-24 zones={test-zone2, test-zone1} 2024-03-04T17:57:11.928 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:11.928 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:11.928 DEBUG:boto:Method: HEAD 2024-03-04T17:57:11.928 DEBUG:boto:Path: /ppfahn-24/ 2024-03-04T17:57:11.928 DEBUG:boto:Data: 2024-03-04T17:57:11.928 DEBUG:boto:Headers: {} 2024-03-04T17:57:11.928 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:11.928 DEBUG:boto:Port: 8000 2024-03-04T17:57:11.928 DEBUG:boto:Params: {} 2024-03-04T17:57:11.929 DEBUG:boto:Token: None 2024-03-04T17:57:11.929 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:11 GMT /ppfahn-24/ 2024-03-04T17:57:11.929 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wcRtqnG11SzJiEjU3smNFk1rbnI= 2024-03-04T17:57:11.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wcRtqnG11SzJiEjU3smNFk1rbnI=', 'Content-Length': '0'} 2024-03-04T17:57:11.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba1d069a26c7c441-0065e60b77-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:11.932 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:11.932 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:11.932 DEBUG:boto:Method: HEAD 2024-03-04T17:57:11.932 DEBUG:boto:Path: /ppfahn-24/ 2024-03-04T17:57:11.932 DEBUG:boto:Data: 2024-03-04T17:57:11.932 DEBUG:boto:Headers: {} 2024-03-04T17:57:11.932 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:11.932 DEBUG:boto:Port: 8000 2024-03-04T17:57:11.933 DEBUG:boto:Params: {} 2024-03-04T17:57:11.933 DEBUG:boto:Token: None 2024-03-04T17:57:11.933 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:11 GMT /ppfahn-24/ 2024-03-04T17:57:11.933 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wcRtqnG11SzJiEjU3smNFk1rbnI= 2024-03-04T17:57:11.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wcRtqnG11SzJiEjU3smNFk1rbnI=', 'Content-Length': '0'} 2024-03-04T17:57:11.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000026e7c2b3d7a2992-0065e60b77-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:11.938 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:11.938 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:11.938 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:11.938 DEBUG:boto:path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:11.938 DEBUG:boto:auth_path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:11.939 DEBUG:boto:Method: GET 2024-03-04T17:57:11.939 DEBUG:boto:Path: /ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:11.939 DEBUG:boto:Data: 2024-03-04T17:57:11.939 DEBUG:boto:Headers: {} 2024-03-04T17:57:11.939 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:11.939 DEBUG:boto:Port: 8000 2024-03-04T17:57:11.939 DEBUG:boto:Params: {} 2024-03-04T17:57:11.939 DEBUG:boto:Token: None 2024-03-04T17:57:11.939 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:11 GMT /ppfahn-24/?versions 2024-03-04T17:57:11.939 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+hzk8sKXSSTu6RkwaU+U4swDEPA= 2024-03-04T17:57:11.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+hzk8sKXSSTu6RkwaU+U4swDEPA=', 'Content-Length': '0'} 2024-03-04T17:57:11.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091f281a74874922a-0065e60b77-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:11.942 DEBUG:boto:b'ppfahn-24999falseobjZGveJeVHJWiuK2dXm0p0CTQ1hzKAPBftrue2024-03-04T17:56:35.885Zrgw-multisite-test-userMultisite Test UserNormalobjkgcxJqgjx7BxWx0pCRaDc4HQTylZmxYfalse2024-03-04T17:56:35.867Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:11.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:11.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:11.943 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:11.943 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:11.943 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:11.943 DEBUG:boto:path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:11.943 DEBUG:boto:auth_path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:11.943 DEBUG:boto:Method: GET 2024-03-04T17:57:11.943 DEBUG:boto:Path: /ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:11.943 DEBUG:boto:Data: 2024-03-04T17:57:11.944 DEBUG:boto:Headers: {} 2024-03-04T17:57:11.944 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:11.944 DEBUG:boto:Port: 8000 2024-03-04T17:57:11.944 DEBUG:boto:Params: {} 2024-03-04T17:57:11.944 DEBUG:boto:Token: None 2024-03-04T17:57:11.944 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:11 GMT /ppfahn-24/?versions 2024-03-04T17:57:11.944 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+hzk8sKXSSTu6RkwaU+U4swDEPA= 2024-03-04T17:57:11.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+hzk8sKXSSTu6RkwaU+U4swDEPA=', 'Content-Length': '0'} 2024-03-04T17:57:11.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c84726a74d5bff31-0065e60b77-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:11.992 DEBUG:boto:b'ppfahn-24999falseobjZGveJeVHJWiuK2dXm0p0CTQ1hzKAPBftrue2024-03-04T17:56:35.885Zrgw-multisite-test-userMultisite Test UserNormalobjkgcxJqgjx7BxWx0pCRaDc4HQTylZmxYfalse2024-03-04T17:56:35.867Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:11.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:11.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:11.993 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:11.993 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:11.993 DEBUG:boto:path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:11.993 DEBUG:boto:auth_path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:11.993 DEBUG:boto:Method: GET 2024-03-04T17:57:11.993 DEBUG:boto:Path: /ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:11.993 DEBUG:boto:Data: 2024-03-04T17:57:11.993 DEBUG:boto:Headers: {} 2024-03-04T17:57:11.993 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:11.993 DEBUG:boto:Port: 8000 2024-03-04T17:57:11.993 DEBUG:boto:Params: {} 2024-03-04T17:57:11.993 DEBUG:boto:Token: None 2024-03-04T17:57:11.994 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:11 GMT /ppfahn-24/?versions 2024-03-04T17:57:11.994 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+hzk8sKXSSTu6RkwaU+U4swDEPA= 2024-03-04T17:57:11.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+hzk8sKXSSTu6RkwaU+U4swDEPA=', 'Content-Length': '0'} 2024-03-04T17:57:11.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000064d2b3f4f700552-0065e60b77-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:11.998 DEBUG:boto:b'ppfahn-24999falseobjZGveJeVHJWiuK2dXm0p0CTQ1hzKAPBftrue2024-03-04T17:56:35.885Zrgw-multisite-test-userMultisite Test UserNormalobjkgcxJqgjx7BxWx0pCRaDc4HQTylZmxYfalse2024-03-04T17:56:35.867Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:11.998 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:11.999 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:11.999 DEBUG:boto:path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:11.999 DEBUG:boto:auth_path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:11.999 DEBUG:boto:Method: GET 2024-03-04T17:57:11.999 DEBUG:boto:Path: /ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:11.999 DEBUG:boto:Data: 2024-03-04T17:57:11.999 DEBUG:boto:Headers: {} 2024-03-04T17:57:11.999 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:11.999 DEBUG:boto:Port: 8000 2024-03-04T17:57:11.999 DEBUG:boto:Params: {} 2024-03-04T17:57:11.999 DEBUG:boto:Token: None 2024-03-04T17:57:12.000 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:11 GMT /ppfahn-24/?versions 2024-03-04T17:57:12.000 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+hzk8sKXSSTu6RkwaU+U4swDEPA= 2024-03-04T17:57:12.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:11 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+hzk8sKXSSTu6RkwaU+U4swDEPA=', 'Content-Length': '0'} 2024-03-04T17:57:12.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008769bf4241726584-0065e60b78-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.004 DEBUG:boto:b'ppfahn-24999falseobjZGveJeVHJWiuK2dXm0p0CTQ1hzKAPBftrue2024-03-04T17:56:35.885Zrgw-multisite-test-userMultisite Test UserNormalobjkgcxJqgjx7BxWx0pCRaDc4HQTylZmxYfalse2024-03-04T17:56:35.867Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:12.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:12.004 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.004 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.005 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.005 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:57:12.005 DEBUG:boto:Data: 2024-03-04T17:57:12.005 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.005 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.005 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.005 DEBUG:boto:Params: {} 2024-03-04T17:57:12.005 DEBUG:boto:Token: None 2024-03-04T17:57:12.005 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj 2024-03-04T17:57:12.005 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk= 2024-03-04T17:57:12.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk=', 'Content-Length': '0'} 2024-03-04T17:57:12.008 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a2d2bc4567446f25-0065e60b78-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.008 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.008 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.008 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.008 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:57:12.008 DEBUG:boto:Data: 2024-03-04T17:57:12.008 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.009 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.009 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.009 DEBUG:boto:Params: {} 2024-03-04T17:57:12.009 DEBUG:boto:Token: None 2024-03-04T17:57:12.009 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj 2024-03-04T17:57:12.009 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk= 2024-03-04T17:57:12.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk=', 'Content-Length': '0'} 2024-03-04T17:57:12.012 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000515d4799c09bda92-0065e60b78-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:12.013 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.013 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.013 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.013 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.013 DEBUG:boto:Method: GET 2024-03-04T17:57:12.013 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.013 DEBUG:boto:Data: 2024-03-04T17:57:12.013 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.013 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.013 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.013 DEBUG:boto:Params: {} 2024-03-04T17:57:12.014 DEBUG:boto:Token: None 2024-03-04T17:57:12.014 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.014 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw= 2024-03-04T17:57:12.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw=', 'Content-Length': '0'} 2024-03-04T17:57:12.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-24:d96636ac-054d-40e3-90f3-e3912da784b2.4349.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bff013d4d58e8ca3-0065e60b78-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.016 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.016 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.017 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.017 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.017 DEBUG:boto:Method: GET 2024-03-04T17:57:12.017 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.017 DEBUG:boto:Data: 2024-03-04T17:57:12.017 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.017 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.017 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.017 DEBUG:boto:Params: {} 2024-03-04T17:57:12.017 DEBUG:boto:Token: None 2024-03-04T17:57:12.017 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.018 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw= 2024-03-04T17:57:12.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw=', 'Content-Length': '0'} 2024-03-04T17:57:12.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000850fd98692e70868-0065e60b78-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.021 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.021 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.021 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.021 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.021 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.021 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.021 DEBUG:boto:Data: 2024-03-04T17:57:12.021 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.021 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.021 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.022 DEBUG:boto:Params: {} 2024-03-04T17:57:12.022 DEBUG:boto:Token: None 2024-03-04T17:57:12.022 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.022 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nAgfA5XLFwHb1ZnAz1VuOqSUgFE= 2024-03-04T17:57:12.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nAgfA5XLFwHb1ZnAz1VuOqSUgFE=', 'Content-Length': '0'} 2024-03-04T17:57:12.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-24:d96636ac-054d-40e3-90f3-e3912da784b2.4349.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000523f3b1c2187c50d-0065e60b78-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.025 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.025 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.025 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.025 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.025 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.025 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.025 DEBUG:boto:Data: 2024-03-04T17:57:12.025 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.025 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.026 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.026 DEBUG:boto:Params: {} 2024-03-04T17:57:12.026 DEBUG:boto:Token: None 2024-03-04T17:57:12.026 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.026 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nAgfA5XLFwHb1ZnAz1VuOqSUgFE= 2024-03-04T17:57:12.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nAgfA5XLFwHb1ZnAz1VuOqSUgFE=', 'Content-Length': '0'} 2024-03-04T17:57:12.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006899cff7f1e3b939-0065e60b78-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:12.029 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.029 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.030 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.030 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.030 DEBUG:boto:Method: GET 2024-03-04T17:57:12.030 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.030 DEBUG:boto:Data: 2024-03-04T17:57:12.030 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.030 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.030 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.030 DEBUG:boto:Params: {} 2024-03-04T17:57:12.030 DEBUG:boto:Token: None 2024-03-04T17:57:12.031 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.031 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw= 2024-03-04T17:57:12.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw=', 'Content-Length': '0'} 2024-03-04T17:57:12.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-24:d96636ac-054d-40e3-90f3-e3912da784b2.4349.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c0de61ef8c69340d-0065e60b78-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.034 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.034 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.034 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.034 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.034 DEBUG:boto:Method: GET 2024-03-04T17:57:12.034 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.034 DEBUG:boto:Data: 2024-03-04T17:57:12.035 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.035 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.035 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.035 DEBUG:boto:Params: {} 2024-03-04T17:57:12.035 DEBUG:boto:Token: None 2024-03-04T17:57:12.035 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.035 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw= 2024-03-04T17:57:12.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw=', 'Content-Length': '0'} 2024-03-04T17:57:12.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b68750f3818cc94d-0065e60b78-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.038 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.038 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.038 DEBUG:boto:path=/ppfahn-24/?prefix=obj 2024-03-04T17:57:12.039 DEBUG:boto:auth_path=/ppfahn-24/?prefix=obj 2024-03-04T17:57:12.039 DEBUG:boto:Method: GET 2024-03-04T17:57:12.039 DEBUG:boto:Path: /ppfahn-24/?prefix=obj 2024-03-04T17:57:12.039 DEBUG:boto:Data: 2024-03-04T17:57:12.039 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.039 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.039 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.039 DEBUG:boto:Params: {} 2024-03-04T17:57:12.039 DEBUG:boto:Token: None 2024-03-04T17:57:12.039 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/ 2024-03-04T17:57:12.039 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D+hmsOwm3Pd31ARI2TIsv0nereA= 2024-03-04T17:57:12.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D+hmsOwm3Pd31ARI2TIsv0nereA=', 'Content-Length': '0'} 2024-03-04T17:57:12.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000158104513c814bc4-0065e60b78-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.084 DEBUG:boto:b'ppfahn-24obj1000false' 2024-03-04T17:57:12.084 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.084 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.085 DEBUG:boto:path=/ppfahn-24/?prefix=obj 2024-03-04T17:57:12.085 DEBUG:boto:auth_path=/ppfahn-24/?prefix=obj 2024-03-04T17:57:12.085 DEBUG:boto:Method: GET 2024-03-04T17:57:12.085 DEBUG:boto:Path: /ppfahn-24/?prefix=obj 2024-03-04T17:57:12.085 DEBUG:boto:Data: 2024-03-04T17:57:12.085 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.085 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.085 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.085 DEBUG:boto:Params: {} 2024-03-04T17:57:12.085 DEBUG:boto:Token: None 2024-03-04T17:57:12.085 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/ 2024-03-04T17:57:12.086 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D+hmsOwm3Pd31ARI2TIsv0nereA= 2024-03-04T17:57:12.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D+hmsOwm3Pd31ARI2TIsv0nereA=', 'Content-Length': '0'} 2024-03-04T17:57:12.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dd0715a08439b4f-0065e60b78-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.091 DEBUG:boto:b'ppfahn-24obj1000false' 2024-03-04T17:57:12.091 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.091 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.091 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.091 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:57:12.091 DEBUG:boto:Data: 2024-03-04T17:57:12.091 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.091 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.091 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.092 DEBUG:boto:Params: {} 2024-03-04T17:57:12.092 DEBUG:boto:Token: None 2024-03-04T17:57:12.092 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj 2024-03-04T17:57:12.092 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk= 2024-03-04T17:57:12.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk=', 'Content-Length': '0'} 2024-03-04T17:57:12.100 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000084c73379d0ee615b-0065e60b78-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.100 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.100 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.101 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.101 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:57:12.101 DEBUG:boto:Data: 2024-03-04T17:57:12.101 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.101 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.101 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.101 DEBUG:boto:Params: {} 2024-03-04T17:57:12.101 DEBUG:boto:Token: None 2024-03-04T17:57:12.101 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj 2024-03-04T17:57:12.101 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk= 2024-03-04T17:57:12.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk=', 'Content-Length': '0'} 2024-03-04T17:57:12.105 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006d4a890abbadb4e1-0065e60b78-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.105 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-24 zones={test-zone2, test-zone1} 2024-03-04T17:57:12.105 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-24 zones={test-zone3, test-zone1} 2024-03-04T17:57:12.106 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.106 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.106 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.106 DEBUG:boto:Path: /ppfahn-24/ 2024-03-04T17:57:12.106 DEBUG:boto:Data: 2024-03-04T17:57:12.106 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.106 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.106 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.106 DEBUG:boto:Params: {} 2024-03-04T17:57:12.106 DEBUG:boto:Token: None 2024-03-04T17:57:12.106 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/ 2024-03-04T17:57:12.107 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0R/zCgfbf5zvx9Mnx7ECnxbnETU= 2024-03-04T17:57:12.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0R/zCgfbf5zvx9Mnx7ECnxbnETU=', 'Content-Length': '0'} 2024-03-04T17:57:12.111 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8d59cdc9d06e79b-0065e60b78-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.111 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.111 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.111 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.111 DEBUG:boto:Path: /ppfahn-24/ 2024-03-04T17:57:12.111 DEBUG:boto:Data: 2024-03-04T17:57:12.111 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.111 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.111 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.112 DEBUG:boto:Params: {} 2024-03-04T17:57:12.112 DEBUG:boto:Token: None 2024-03-04T17:57:12.112 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/ 2024-03-04T17:57:12.112 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0R/zCgfbf5zvx9Mnx7ECnxbnETU= 2024-03-04T17:57:12.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0R/zCgfbf5zvx9Mnx7ECnxbnETU=', 'Content-Length': '0'} 2024-03-04T17:57:12.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f0138da3673b71e-0065e60b78-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.117 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:12.117 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.117 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.117 DEBUG:boto:path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.117 DEBUG:boto:auth_path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.117 DEBUG:boto:Method: GET 2024-03-04T17:57:12.118 DEBUG:boto:Path: /ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.118 DEBUG:boto:Data: 2024-03-04T17:57:12.118 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.118 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.118 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.118 DEBUG:boto:Params: {} 2024-03-04T17:57:12.118 DEBUG:boto:Token: None 2024-03-04T17:57:12.118 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/?versions 2024-03-04T17:57:12.118 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k= 2024-03-04T17:57:12.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k=', 'Content-Length': '0'} 2024-03-04T17:57:12.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c856aa934259e579-0065e60b78-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.164 DEBUG:boto:b'ppfahn-24999falseobjZGveJeVHJWiuK2dXm0p0CTQ1hzKAPBftrue2024-03-04T17:56:35.885Zrgw-multisite-test-userMultisite Test UserNormalobjkgcxJqgjx7BxWx0pCRaDc4HQTylZmxYfalse2024-03-04T17:56:35.867Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:12.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:12.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:12.165 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:12.165 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.165 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.165 DEBUG:boto:path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.165 DEBUG:boto:auth_path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.165 DEBUG:boto:Method: GET 2024-03-04T17:57:12.165 DEBUG:boto:Path: /ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.165 DEBUG:boto:Data: 2024-03-04T17:57:12.165 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.165 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.166 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.166 DEBUG:boto:Params: {} 2024-03-04T17:57:12.166 DEBUG:boto:Token: None 2024-03-04T17:57:12.166 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/?versions 2024-03-04T17:57:12.166 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k= 2024-03-04T17:57:12.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k=', 'Content-Length': '0'} 2024-03-04T17:57:12.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013d0c7291043e2d3-0065e60b78-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.171 DEBUG:boto:b'ppfahn-24999falseobjZGveJeVHJWiuK2dXm0p0CTQ1hzKAPBftrue2024-03-04T17:56:35.885Zrgw-multisite-test-userMultisite Test UserNormalobjkgcxJqgjx7BxWx0pCRaDc4HQTylZmxYfalse2024-03-04T17:56:35.867Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:12.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:12.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:12.172 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.172 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.172 DEBUG:boto:path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.172 DEBUG:boto:auth_path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.172 DEBUG:boto:Method: GET 2024-03-04T17:57:12.172 DEBUG:boto:Path: /ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.172 DEBUG:boto:Data: 2024-03-04T17:57:12.172 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.172 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.173 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.173 DEBUG:boto:Params: {} 2024-03-04T17:57:12.173 DEBUG:boto:Token: None 2024-03-04T17:57:12.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/?versions 2024-03-04T17:57:12.173 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k= 2024-03-04T17:57:12.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k=', 'Content-Length': '0'} 2024-03-04T17:57:12.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ca7640bf0956e82-0065e60b78-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.177 DEBUG:boto:b'ppfahn-24999falseobjZGveJeVHJWiuK2dXm0p0CTQ1hzKAPBftrue2024-03-04T17:56:35.885Zrgw-multisite-test-userMultisite Test UserNormalobjkgcxJqgjx7BxWx0pCRaDc4HQTylZmxYfalse2024-03-04T17:56:35.867Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:12.177 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.178 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.178 DEBUG:boto:path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.178 DEBUG:boto:auth_path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.178 DEBUG:boto:Method: GET 2024-03-04T17:57:12.178 DEBUG:boto:Path: /ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.178 DEBUG:boto:Data: 2024-03-04T17:57:12.178 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.178 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.178 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.178 DEBUG:boto:Params: {} 2024-03-04T17:57:12.178 DEBUG:boto:Token: None 2024-03-04T17:57:12.179 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/?versions 2024-03-04T17:57:12.179 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k= 2024-03-04T17:57:12.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k=', 'Content-Length': '0'} 2024-03-04T17:57:12.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cbcb25a3dd0f152-0065e60b78-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.224 DEBUG:boto:b'ppfahn-24999falseobjZGveJeVHJWiuK2dXm0p0CTQ1hzKAPBftrue2024-03-04T17:56:35.885Zrgw-multisite-test-userMultisite Test UserNormalobjkgcxJqgjx7BxWx0pCRaDc4HQTylZmxYfalse2024-03-04T17:56:35.867Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:12.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:12.225 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.225 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.225 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.225 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:57:12.225 DEBUG:boto:Data: 2024-03-04T17:57:12.225 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.225 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.225 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.225 DEBUG:boto:Params: {} 2024-03-04T17:57:12.225 DEBUG:boto:Token: None 2024-03-04T17:57:12.225 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj 2024-03-04T17:57:12.226 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk= 2024-03-04T17:57:12.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk=', 'Content-Length': '0'} 2024-03-04T17:57:12.229 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000069b6a31f04490e1-0065e60b78-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.229 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.229 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.229 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.230 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:57:12.230 DEBUG:boto:Data: 2024-03-04T17:57:12.230 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.230 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.230 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.230 DEBUG:boto:Params: {} 2024-03-04T17:57:12.230 DEBUG:boto:Token: None 2024-03-04T17:57:12.230 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj 2024-03-04T17:57:12.230 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk= 2024-03-04T17:57:12.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk=', 'Content-Length': '0'} 2024-03-04T17:57:12.234 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000020d7cd246c5b5d63-0065e60b78-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:12.234 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.234 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.235 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.235 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.235 DEBUG:boto:Method: GET 2024-03-04T17:57:12.235 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.235 DEBUG:boto:Data: 2024-03-04T17:57:12.235 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.235 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.235 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.235 DEBUG:boto:Params: {} 2024-03-04T17:57:12.235 DEBUG:boto:Token: None 2024-03-04T17:57:12.235 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.236 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw= 2024-03-04T17:57:12.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw=', 'Content-Length': '0'} 2024-03-04T17:57:12.239 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-24:d96636ac-054d-40e3-90f3-e3912da784b2.4349.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000046a95a9a16524d97-0065e60b78-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.239 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.239 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.239 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.239 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.239 DEBUG:boto:Method: GET 2024-03-04T17:57:12.239 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.239 DEBUG:boto:Data: 2024-03-04T17:57:12.239 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.240 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.240 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.240 DEBUG:boto:Params: {} 2024-03-04T17:57:12.240 DEBUG:boto:Token: None 2024-03-04T17:57:12.240 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.240 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw= 2024-03-04T17:57:12.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw=', 'Content-Length': '0'} 2024-03-04T17:57:12.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fc7d0db7fa2ef44f-0065e60b78-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.243 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.243 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.243 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.243 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.244 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.244 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.244 DEBUG:boto:Data: 2024-03-04T17:57:12.244 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.244 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.244 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.244 DEBUG:boto:Params: {} 2024-03-04T17:57:12.244 DEBUG:boto:Token: None 2024-03-04T17:57:12.244 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.244 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nAgfA5XLFwHb1ZnAz1VuOqSUgFE= 2024-03-04T17:57:12.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nAgfA5XLFwHb1ZnAz1VuOqSUgFE=', 'Content-Length': '0'} 2024-03-04T17:57:12.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-24:d96636ac-054d-40e3-90f3-e3912da784b2.4349.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d26fb96c0031458f-0065e60b78-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.247 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.247 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.247 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.248 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.248 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.248 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.248 DEBUG:boto:Data: 2024-03-04T17:57:12.248 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.248 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.248 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.248 DEBUG:boto:Params: {} 2024-03-04T17:57:12.248 DEBUG:boto:Token: None 2024-03-04T17:57:12.248 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.249 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nAgfA5XLFwHb1ZnAz1VuOqSUgFE= 2024-03-04T17:57:12.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nAgfA5XLFwHb1ZnAz1VuOqSUgFE=', 'Content-Length': '0'} 2024-03-04T17:57:12.251 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cdcc15256bd3b0ec-0065e60b78-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:12.252 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.252 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.252 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.252 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.252 DEBUG:boto:Method: GET 2024-03-04T17:57:12.252 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.252 DEBUG:boto:Data: 2024-03-04T17:57:12.252 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.252 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.252 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.252 DEBUG:boto:Params: {} 2024-03-04T17:57:12.253 DEBUG:boto:Token: None 2024-03-04T17:57:12.253 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.253 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw= 2024-03-04T17:57:12.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw=', 'Content-Length': '0'} 2024-03-04T17:57:12.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-24:d96636ac-054d-40e3-90f3-e3912da784b2.4349.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e6186d4982fff1e6-0065e60b78-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.256 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.256 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.256 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.256 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.256 DEBUG:boto:Method: GET 2024-03-04T17:57:12.256 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.256 DEBUG:boto:Data: 2024-03-04T17:57:12.256 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.256 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.256 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.256 DEBUG:boto:Params: {} 2024-03-04T17:57:12.257 DEBUG:boto:Token: None 2024-03-04T17:57:12.257 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.257 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw= 2024-03-04T17:57:12.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw=', 'Content-Length': '0'} 2024-03-04T17:57:12.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001d0a045c8520f7ac-0065e60b78-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.260 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.260 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.260 DEBUG:boto:path=/ppfahn-24/?prefix=obj 2024-03-04T17:57:12.260 DEBUG:boto:auth_path=/ppfahn-24/?prefix=obj 2024-03-04T17:57:12.260 DEBUG:boto:Method: GET 2024-03-04T17:57:12.260 DEBUG:boto:Path: /ppfahn-24/?prefix=obj 2024-03-04T17:57:12.260 DEBUG:boto:Data: 2024-03-04T17:57:12.260 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.260 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.260 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.260 DEBUG:boto:Params: {} 2024-03-04T17:57:12.260 DEBUG:boto:Token: None 2024-03-04T17:57:12.261 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/ 2024-03-04T17:57:12.261 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D+hmsOwm3Pd31ARI2TIsv0nereA= 2024-03-04T17:57:12.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D+hmsOwm3Pd31ARI2TIsv0nereA=', 'Content-Length': '0'} 2024-03-04T17:57:12.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026d17a40ea1d44e7-0065e60b78-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.265 DEBUG:boto:b'ppfahn-24obj1000false' 2024-03-04T17:57:12.265 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.265 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.265 DEBUG:boto:path=/ppfahn-24/?prefix=obj 2024-03-04T17:57:12.265 DEBUG:boto:auth_path=/ppfahn-24/?prefix=obj 2024-03-04T17:57:12.266 DEBUG:boto:Method: GET 2024-03-04T17:57:12.266 DEBUG:boto:Path: /ppfahn-24/?prefix=obj 2024-03-04T17:57:12.266 DEBUG:boto:Data: 2024-03-04T17:57:12.266 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.266 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.266 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.266 DEBUG:boto:Params: {} 2024-03-04T17:57:12.266 DEBUG:boto:Token: None 2024-03-04T17:57:12.266 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/ 2024-03-04T17:57:12.266 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D+hmsOwm3Pd31ARI2TIsv0nereA= 2024-03-04T17:57:12.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D+hmsOwm3Pd31ARI2TIsv0nereA=', 'Content-Length': '0'} 2024-03-04T17:57:12.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000642cf14974780abe-0065e60b78-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.270 DEBUG:boto:b'ppfahn-24obj1000false' 2024-03-04T17:57:12.271 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.271 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.271 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.271 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:57:12.271 DEBUG:boto:Data: 2024-03-04T17:57:12.271 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.271 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.271 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.271 DEBUG:boto:Params: {} 2024-03-04T17:57:12.271 DEBUG:boto:Token: None 2024-03-04T17:57:12.272 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj 2024-03-04T17:57:12.272 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk= 2024-03-04T17:57:12.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk=', 'Content-Length': '0'} 2024-03-04T17:57:12.275 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004c73433494b69fb0-0065e60b78-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.276 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.276 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.276 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.276 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:57:12.276 DEBUG:boto:Data: 2024-03-04T17:57:12.276 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.276 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:12.276 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.276 DEBUG:boto:Params: {} 2024-03-04T17:57:12.277 DEBUG:boto:Token: None 2024-03-04T17:57:12.277 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj 2024-03-04T17:57:12.277 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk= 2024-03-04T17:57:12.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk=', 'Content-Length': '0'} 2024-03-04T17:57:12.281 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d080127ace7daee9-0065e60b78-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.281 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-24 zones={test-zone3, test-zone1} 2024-03-04T17:57:12.281 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-24 zones={test-zone3, test-zone2} 2024-03-04T17:57:12.281 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.281 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.281 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.281 DEBUG:boto:Path: /ppfahn-24/ 2024-03-04T17:57:12.282 DEBUG:boto:Data: 2024-03-04T17:57:12.282 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.282 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.282 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.282 DEBUG:boto:Params: {} 2024-03-04T17:57:12.282 DEBUG:boto:Token: None 2024-03-04T17:57:12.282 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/ 2024-03-04T17:57:12.282 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0R/zCgfbf5zvx9Mnx7ECnxbnETU= 2024-03-04T17:57:12.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0R/zCgfbf5zvx9Mnx7ECnxbnETU=', 'Content-Length': '0'} 2024-03-04T17:57:12.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4af80bdc99fdb92-0065e60b78-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.288 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.288 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.288 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.288 DEBUG:boto:Path: /ppfahn-24/ 2024-03-04T17:57:12.288 DEBUG:boto:Data: 2024-03-04T17:57:12.288 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.288 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.288 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.289 DEBUG:boto:Params: {} 2024-03-04T17:57:12.289 DEBUG:boto:Token: None 2024-03-04T17:57:12.289 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/ 2024-03-04T17:57:12.289 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0R/zCgfbf5zvx9Mnx7ECnxbnETU= 2024-03-04T17:57:12.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0R/zCgfbf5zvx9Mnx7ECnxbnETU=', 'Content-Length': '0'} 2024-03-04T17:57:12.293 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a317b46e46312646-0065e60b78-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.293 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:12.293 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.294 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.294 DEBUG:boto:path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.294 DEBUG:boto:auth_path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.294 DEBUG:boto:Method: GET 2024-03-04T17:57:12.294 DEBUG:boto:Path: /ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.294 DEBUG:boto:Data: 2024-03-04T17:57:12.294 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.294 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.294 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.294 DEBUG:boto:Params: {} 2024-03-04T17:57:12.294 DEBUG:boto:Token: None 2024-03-04T17:57:12.295 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/?versions 2024-03-04T17:57:12.295 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k= 2024-03-04T17:57:12.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k=', 'Content-Length': '0'} 2024-03-04T17:57:12.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000486ccecc9d47bd7f-0065e60b78-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.340 DEBUG:boto:b'ppfahn-24999falseobjZGveJeVHJWiuK2dXm0p0CTQ1hzKAPBftrue2024-03-04T17:56:35.885Zrgw-multisite-test-userMultisite Test UserNormalobjkgcxJqgjx7BxWx0pCRaDc4HQTylZmxYfalse2024-03-04T17:56:35.867Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:12.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:12.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:12.340 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:12.341 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.341 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.341 DEBUG:boto:path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.341 DEBUG:boto:auth_path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.341 DEBUG:boto:Method: GET 2024-03-04T17:57:12.341 DEBUG:boto:Path: /ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.341 DEBUG:boto:Data: 2024-03-04T17:57:12.341 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.341 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.341 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.341 DEBUG:boto:Params: {} 2024-03-04T17:57:12.341 DEBUG:boto:Token: None 2024-03-04T17:57:12.342 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/?versions 2024-03-04T17:57:12.342 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k= 2024-03-04T17:57:12.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k=', 'Content-Length': '0'} 2024-03-04T17:57:12.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040a11ffaa1f1b6cc-0065e60b78-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.346 DEBUG:boto:b'ppfahn-24999falseobjZGveJeVHJWiuK2dXm0p0CTQ1hzKAPBftrue2024-03-04T17:56:35.885Zrgw-multisite-test-userMultisite Test UserNormalobjkgcxJqgjx7BxWx0pCRaDc4HQTylZmxYfalse2024-03-04T17:56:35.867Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:12.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:12.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:12.347 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.347 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.347 DEBUG:boto:path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.347 DEBUG:boto:auth_path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.347 DEBUG:boto:Method: GET 2024-03-04T17:57:12.347 DEBUG:boto:Path: /ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.347 DEBUG:boto:Data: 2024-03-04T17:57:12.347 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.347 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.347 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.348 DEBUG:boto:Params: {} 2024-03-04T17:57:12.348 DEBUG:boto:Token: None 2024-03-04T17:57:12.348 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/?versions 2024-03-04T17:57:12.348 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k= 2024-03-04T17:57:12.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k=', 'Content-Length': '0'} 2024-03-04T17:57:12.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc43b8bdb31fc149-0065e60b78-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.353 DEBUG:boto:b'ppfahn-24999falseobjZGveJeVHJWiuK2dXm0p0CTQ1hzKAPBftrue2024-03-04T17:56:35.885Zrgw-multisite-test-userMultisite Test UserNormalobjkgcxJqgjx7BxWx0pCRaDc4HQTylZmxYfalse2024-03-04T17:56:35.867Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:12.353 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.354 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.354 DEBUG:boto:path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.354 DEBUG:boto:auth_path=/ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.354 DEBUG:boto:Method: GET 2024-03-04T17:57:12.354 DEBUG:boto:Path: /ppfahn-24/?versions&max-keys=999 2024-03-04T17:57:12.354 DEBUG:boto:Data: 2024-03-04T17:57:12.354 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.354 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.354 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.354 DEBUG:boto:Params: {} 2024-03-04T17:57:12.354 DEBUG:boto:Token: None 2024-03-04T17:57:12.355 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/?versions 2024-03-04T17:57:12.355 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k= 2024-03-04T17:57:12.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VC1xPPiJR5JZllGo2ds8yweKA1k=', 'Content-Length': '0'} 2024-03-04T17:57:12.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049f45a98eaffcdae-0065e60b78-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.360 DEBUG:boto:b'ppfahn-24999falseobjZGveJeVHJWiuK2dXm0p0CTQ1hzKAPBftrue2024-03-04T17:56:35.885Zrgw-multisite-test-userMultisite Test UserNormalobjkgcxJqgjx7BxWx0pCRaDc4HQTylZmxYfalse2024-03-04T17:56:35.867Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:12.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:12.360 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.360 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.361 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.361 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:57:12.361 DEBUG:boto:Data: 2024-03-04T17:57:12.361 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.361 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.361 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.361 DEBUG:boto:Params: {} 2024-03-04T17:57:12.361 DEBUG:boto:Token: None 2024-03-04T17:57:12.361 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj 2024-03-04T17:57:12.361 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk= 2024-03-04T17:57:12.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk=', 'Content-Length': '0'} 2024-03-04T17:57:12.364 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000009e6e0a969b3cc2b3-0065e60b78-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.365 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.365 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.365 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.365 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:57:12.365 DEBUG:boto:Data: 2024-03-04T17:57:12.365 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.365 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.365 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.365 DEBUG:boto:Params: {} 2024-03-04T17:57:12.366 DEBUG:boto:Token: None 2024-03-04T17:57:12.366 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj 2024-03-04T17:57:12.366 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk= 2024-03-04T17:57:12.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk=', 'Content-Length': '0'} 2024-03-04T17:57:12.369 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006e75102c551f2be8-0065e60b78-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:12.370 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.370 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.370 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.370 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.370 DEBUG:boto:Method: GET 2024-03-04T17:57:12.370 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.370 DEBUG:boto:Data: 2024-03-04T17:57:12.370 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.370 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.370 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.370 DEBUG:boto:Params: {} 2024-03-04T17:57:12.371 DEBUG:boto:Token: None 2024-03-04T17:57:12.371 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.371 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw= 2024-03-04T17:57:12.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw=', 'Content-Length': '0'} 2024-03-04T17:57:12.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-24:d96636ac-054d-40e3-90f3-e3912da784b2.4349.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b009ca9ce585aa24-0065e60b78-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.374 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.374 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.374 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.374 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.374 DEBUG:boto:Method: GET 2024-03-04T17:57:12.375 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.375 DEBUG:boto:Data: 2024-03-04T17:57:12.375 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.375 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.375 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.375 DEBUG:boto:Params: {} 2024-03-04T17:57:12.375 DEBUG:boto:Token: None 2024-03-04T17:57:12.375 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.375 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw= 2024-03-04T17:57:12.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw=', 'Content-Length': '0'} 2024-03-04T17:57:12.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-24:d96636ac-054d-40e3-90f3-e3912da784b2.4349.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b339d49c66ac021-0065e60b78-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.379 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.379 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.379 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.379 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.379 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.379 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.379 DEBUG:boto:Data: 2024-03-04T17:57:12.379 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.379 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.379 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.379 DEBUG:boto:Params: {} 2024-03-04T17:57:12.380 DEBUG:boto:Token: None 2024-03-04T17:57:12.380 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.380 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nAgfA5XLFwHb1ZnAz1VuOqSUgFE= 2024-03-04T17:57:12.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nAgfA5XLFwHb1ZnAz1VuOqSUgFE=', 'Content-Length': '0'} 2024-03-04T17:57:12.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-24:d96636ac-054d-40e3-90f3-e3912da784b2.4349.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b5c77e27a13b7590-0065e60b78-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.383 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.383 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.383 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.383 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.383 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.383 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.384 DEBUG:boto:Data: 2024-03-04T17:57:12.384 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.384 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.384 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.384 DEBUG:boto:Params: {} 2024-03-04T17:57:12.384 DEBUG:boto:Token: None 2024-03-04T17:57:12.384 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.384 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:nAgfA5XLFwHb1ZnAz1VuOqSUgFE= 2024-03-04T17:57:12.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nAgfA5XLFwHb1ZnAz1VuOqSUgFE=', 'Content-Length': '0'} 2024-03-04T17:57:12.387 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-24:d96636ac-054d-40e3-90f3-e3912da784b2.4349.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000035eb5d8786e61a45-0065e60b78-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:12.387 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.387 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.387 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.388 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.388 DEBUG:boto:Method: GET 2024-03-04T17:57:12.388 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.388 DEBUG:boto:Data: 2024-03-04T17:57:12.388 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.388 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.388 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.388 DEBUG:boto:Params: {} 2024-03-04T17:57:12.388 DEBUG:boto:Token: None 2024-03-04T17:57:12.388 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.389 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw= 2024-03-04T17:57:12.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw=', 'Content-Length': '0'} 2024-03-04T17:57:12.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-24:d96636ac-054d-40e3-90f3-e3912da784b2.4349.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc6850be577ba103-0065e60b78-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.392 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.392 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.392 DEBUG:boto:path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.392 DEBUG:boto:auth_path=/ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.392 DEBUG:boto:Method: GET 2024-03-04T17:57:12.392 DEBUG:boto:Path: /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.392 DEBUG:boto:Data: 2024-03-04T17:57:12.392 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.393 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.393 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.393 DEBUG:boto:Params: {} 2024-03-04T17:57:12.393 DEBUG:boto:Token: None 2024-03-04T17:57:12.393 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj?versionId=kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY 2024-03-04T17:57:12.393 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw= 2024-03-04T17:57:12.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hsqJVrM3QhbhCEUrZ3OYM5PMRdw=', 'Content-Length': '0'} 2024-03-04T17:57:12.396 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'kgcxJqgjx7BxWx0pCRaDc4HQTylZmxY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd96636ac-054d-40e3-90f3-e3912da784b2:ppfahn-24:d96636ac-054d-40e3-90f3-e3912da784b2.4349.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000261907e352461afd-0065e60b78-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.396 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.396 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.396 DEBUG:boto:path=/ppfahn-24/?prefix=obj 2024-03-04T17:57:12.396 DEBUG:boto:auth_path=/ppfahn-24/?prefix=obj 2024-03-04T17:57:12.397 DEBUG:boto:Method: GET 2024-03-04T17:57:12.397 DEBUG:boto:Path: /ppfahn-24/?prefix=obj 2024-03-04T17:57:12.397 DEBUG:boto:Data: 2024-03-04T17:57:12.397 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.397 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.397 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.397 DEBUG:boto:Params: {} 2024-03-04T17:57:12.397 DEBUG:boto:Token: None 2024-03-04T17:57:12.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/ 2024-03-04T17:57:12.397 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D+hmsOwm3Pd31ARI2TIsv0nereA= 2024-03-04T17:57:12.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D+hmsOwm3Pd31ARI2TIsv0nereA=', 'Content-Length': '0'} 2024-03-04T17:57:12.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000702031aa62ba560a-0065e60b78-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.444 DEBUG:boto:b'ppfahn-24obj1000false' 2024-03-04T17:57:12.444 DEBUG:boto:path=/ppfahn-24/ 2024-03-04T17:57:12.444 DEBUG:boto:auth_path=/ppfahn-24/ 2024-03-04T17:57:12.444 DEBUG:boto:path=/ppfahn-24/?prefix=obj 2024-03-04T17:57:12.445 DEBUG:boto:auth_path=/ppfahn-24/?prefix=obj 2024-03-04T17:57:12.445 DEBUG:boto:Method: GET 2024-03-04T17:57:12.445 DEBUG:boto:Path: /ppfahn-24/?prefix=obj 2024-03-04T17:57:12.445 DEBUG:boto:Data: 2024-03-04T17:57:12.445 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.445 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.445 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.445 DEBUG:boto:Params: {} 2024-03-04T17:57:12.445 DEBUG:boto:Token: None 2024-03-04T17:57:12.445 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/ 2024-03-04T17:57:12.445 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:D+hmsOwm3Pd31ARI2TIsv0nereA= 2024-03-04T17:57:12.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D+hmsOwm3Pd31ARI2TIsv0nereA=', 'Content-Length': '0'} 2024-03-04T17:57:12.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1675d998cc94d9e-0065e60b78-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.450 DEBUG:boto:b'ppfahn-24obj1000false' 2024-03-04T17:57:12.450 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.450 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.450 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.450 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:57:12.450 DEBUG:boto:Data: 2024-03-04T17:57:12.450 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.450 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:12.450 DEBUG:boto:Port: 8001 2024-03-04T17:57:12.451 DEBUG:boto:Params: {} 2024-03-04T17:57:12.451 DEBUG:boto:Token: None 2024-03-04T17:57:12.451 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj 2024-03-04T17:57:12.451 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk= 2024-03-04T17:57:12.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk=', 'Content-Length': '0'} 2024-03-04T17:57:12.455 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000f507e1aadcc589e7-0065e60b78-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.455 DEBUG:boto:path=/ppfahn-24/obj 2024-03-04T17:57:12.455 DEBUG:boto:auth_path=/ppfahn-24/obj 2024-03-04T17:57:12.455 DEBUG:boto:Method: HEAD 2024-03-04T17:57:12.455 DEBUG:boto:Path: /ppfahn-24/obj 2024-03-04T17:57:12.455 DEBUG:boto:Data: 2024-03-04T17:57:12.455 DEBUG:boto:Headers: {} 2024-03-04T17:57:12.455 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:12.456 DEBUG:boto:Port: 8000 2024-03-04T17:57:12.456 DEBUG:boto:Params: {} 2024-03-04T17:57:12.456 DEBUG:boto:Token: None 2024-03-04T17:57:12.456 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:12 GMT /ppfahn-24/obj 2024-03-04T17:57:12.456 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk= 2024-03-04T17:57:12.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:12 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hQ+wKcGPyVMo0t99aJaM+IeiCIk=', 'Content-Length': '0'} 2024-03-04T17:57:12.460 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000cd6c0d052be8b5a4-0065e60b78-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:12.460 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-24 zones={test-zone3, test-zone2} 2024-03-04T17:57:12.460 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-25 2024-03-04T17:57:12.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:12.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:12.461 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:12.510 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:12.510 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:12.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:12.576+0000 7f7f81075980 1 bucket sync caught up with source: 2024-03-04T17:57:12.574 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000003.36.12, , ] 2024-03-04T17:57:12.574 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000003.36.12, , ] 2024-03-04T17:57:12.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:12.576+0000 7f7f81075980 0 bucket checkpoint complete 2024-03-04T17:57:12.597 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-25 2024-03-04T17:57:12.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:12.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:12.597 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:12.647 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:12.648 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:12.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:12.709+0000 7f93afd23980 1 bucket sync caught up with source: 2024-03-04T17:57:12.708 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000003.36.12, , ] 2024-03-04T17:57:12.708 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.36.12, , ] 2024-03-04T17:57:12.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:12.709+0000 7f93afd23980 0 bucket checkpoint complete 2024-03-04T17:57:12.730 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-25 2024-03-04T17:57:12.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:12.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:12.731 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:12.771 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:12.771 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:12.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:12.833+0000 7fbf7056e980 1 bucket sync caught up with source: 2024-03-04T17:57:12.832 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000003.39.12, , ] 2024-03-04T17:57:12.832 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.39.12, , ] 2024-03-04T17:57:12.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:12.833+0000 7fbf7056e980 0 bucket checkpoint complete 2024-03-04T17:57:12.854 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-25 2024-03-04T17:57:12.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:12.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:12.855 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:12.942 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:12.942 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:13.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:13.001+0000 7f5fda889980 1 bucket sync caught up with source: 2024-03-04T17:57:13.003 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000003.39.12, , ] 2024-03-04T17:57:13.003 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.39.12, , ] 2024-03-04T17:57:13.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:13.001+0000 7f5fda889980 0 bucket checkpoint complete 2024-03-04T17:57:13.025 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-25 2024-03-04T17:57:13.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:13.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:13.025 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:13.065 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:13.066 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:13.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:13.121+0000 7fc5c7546980 1 bucket sync caught up with source: 2024-03-04T17:57:13.122 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000004.54.12, , ] 2024-03-04T17:57:13.122 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000004.54.12, , ] 2024-03-04T17:57:13.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:13.121+0000 7fc5c7546980 0 bucket checkpoint complete 2024-03-04T17:57:13.143 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-25 2024-03-04T17:57:13.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:13.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:13.144 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:13.194 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:13.194 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:13.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:13.260+0000 7fad08006980 1 bucket sync caught up with source: 2024-03-04T17:57:13.258 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000004.54.12, , ] 2024-03-04T17:57:13.258 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000004.54.12, , ] 2024-03-04T17:57:13.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:13.260+0000 7fad08006980 0 bucket checkpoint complete 2024-03-04T17:57:13.281 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-25 zones={test-zone2, test-zone1} 2024-03-04T17:57:13.281 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.282 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.282 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.282 DEBUG:boto:Path: /ppfahn-25/ 2024-03-04T17:57:13.282 DEBUG:boto:Data: 2024-03-04T17:57:13.282 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.282 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.282 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.282 DEBUG:boto:Params: {} 2024-03-04T17:57:13.282 DEBUG:boto:Token: None 2024-03-04T17:57:13.282 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/ 2024-03-04T17:57:13.283 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1qyZtLhN100tl470Y0Et9S9C9rs= 2024-03-04T17:57:13.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1qyZtLhN100tl470Y0Et9S9C9rs=', 'Content-Length': '0'} 2024-03-04T17:57:13.285 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031313cfb6a7108f4-0065e60b79-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.285 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.285 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.286 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.286 DEBUG:boto:Path: /ppfahn-25/ 2024-03-04T17:57:13.286 DEBUG:boto:Data: 2024-03-04T17:57:13.286 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.286 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.286 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.286 DEBUG:boto:Params: {} 2024-03-04T17:57:13.286 DEBUG:boto:Token: None 2024-03-04T17:57:13.286 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/ 2024-03-04T17:57:13.286 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1qyZtLhN100tl470Y0Et9S9C9rs= 2024-03-04T17:57:13.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1qyZtLhN100tl470Y0Et9S9C9rs=', 'Content-Length': '0'} 2024-03-04T17:57:13.290 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000804b967cc3dd84a0-0065e60b79-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.290 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:13.291 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.291 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.291 DEBUG:boto:path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.291 DEBUG:boto:auth_path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.291 DEBUG:boto:Method: GET 2024-03-04T17:57:13.291 DEBUG:boto:Path: /ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.291 DEBUG:boto:Data: 2024-03-04T17:57:13.291 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.291 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.291 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.291 DEBUG:boto:Params: {} 2024-03-04T17:57:13.292 DEBUG:boto:Token: None 2024-03-04T17:57:13.292 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/?versions 2024-03-04T17:57:13.292 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8= 2024-03-04T17:57:13.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8=', 'Content-Length': '0'} 2024-03-04T17:57:13.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a1cf5401de0aba5-0065e60b79-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.294 DEBUG:boto:b'ppfahn-25999falseobjy476gJad3qtgVOCqSNyCSIsViYsdZnMtrue2024-03-04T17:56:35.937Zrgw-multisite-test-userMultisite Test UserNormalobjAytj.t3jFObNFOXGB1.vARQ1kbJTYQTfalse2024-03-04T17:56:35.902Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:13.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:13.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:13.295 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:13.295 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.295 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.295 DEBUG:boto:path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.295 DEBUG:boto:auth_path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.295 DEBUG:boto:Method: GET 2024-03-04T17:57:13.296 DEBUG:boto:Path: /ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.296 DEBUG:boto:Data: 2024-03-04T17:57:13.296 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.296 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.296 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.296 DEBUG:boto:Params: {} 2024-03-04T17:57:13.296 DEBUG:boto:Token: None 2024-03-04T17:57:13.296 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/?versions 2024-03-04T17:57:13.296 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8= 2024-03-04T17:57:13.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8=', 'Content-Length': '0'} 2024-03-04T17:57:13.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d93240537f84e9c-0065e60b79-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.301 DEBUG:boto:b'ppfahn-25999falseobjy476gJad3qtgVOCqSNyCSIsViYsdZnMtrue2024-03-04T17:56:35.937Zrgw-multisite-test-userMultisite Test UserNormalobjAytj.t3jFObNFOXGB1.vARQ1kbJTYQTfalse2024-03-04T17:56:35.902Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:13.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:13.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:13.301 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.301 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.302 DEBUG:boto:path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.302 DEBUG:boto:auth_path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.302 DEBUG:boto:Method: GET 2024-03-04T17:57:13.302 DEBUG:boto:Path: /ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.302 DEBUG:boto:Data: 2024-03-04T17:57:13.302 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.302 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.302 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.302 DEBUG:boto:Params: {} 2024-03-04T17:57:13.302 DEBUG:boto:Token: None 2024-03-04T17:57:13.302 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/?versions 2024-03-04T17:57:13.303 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8= 2024-03-04T17:57:13.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8=', 'Content-Length': '0'} 2024-03-04T17:57:13.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc31811a3b62af63-0065e60b79-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.305 DEBUG:boto:b'ppfahn-25999falseobjy476gJad3qtgVOCqSNyCSIsViYsdZnMtrue2024-03-04T17:56:35.937Zrgw-multisite-test-userMultisite Test UserNormalobjAytj.t3jFObNFOXGB1.vARQ1kbJTYQTfalse2024-03-04T17:56:35.902Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:13.306 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.306 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.306 DEBUG:boto:path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.306 DEBUG:boto:auth_path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.306 DEBUG:boto:Method: GET 2024-03-04T17:57:13.306 DEBUG:boto:Path: /ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.306 DEBUG:boto:Data: 2024-03-04T17:57:13.306 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.306 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.307 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.307 DEBUG:boto:Params: {} 2024-03-04T17:57:13.307 DEBUG:boto:Token: None 2024-03-04T17:57:13.307 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/?versions 2024-03-04T17:57:13.307 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8= 2024-03-04T17:57:13.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8=', 'Content-Length': '0'} 2024-03-04T17:57:13.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c25188086430c21e-0065e60b79-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.311 DEBUG:boto:b'ppfahn-25999falseobjy476gJad3qtgVOCqSNyCSIsViYsdZnMtrue2024-03-04T17:56:35.937Zrgw-multisite-test-userMultisite Test UserNormalobjAytj.t3jFObNFOXGB1.vARQ1kbJTYQTfalse2024-03-04T17:56:35.902Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:13.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:13.312 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.312 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.312 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.312 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:57:13.312 DEBUG:boto:Data: 2024-03-04T17:57:13.312 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.312 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.312 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.312 DEBUG:boto:Params: {} 2024-03-04T17:57:13.313 DEBUG:boto:Token: None 2024-03-04T17:57:13.313 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj 2024-03-04T17:57:13.313 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE= 2024-03-04T17:57:13.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE=', 'Content-Length': '0'} 2024-03-04T17:57:13.315 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006ee572b0f8057428-0065e60b79-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.315 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.315 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.315 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.315 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:57:13.315 DEBUG:boto:Data: 2024-03-04T17:57:13.316 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.316 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.316 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.316 DEBUG:boto:Params: {} 2024-03-04T17:57:13.316 DEBUG:boto:Token: None 2024-03-04T17:57:13.316 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj 2024-03-04T17:57:13.316 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE= 2024-03-04T17:57:13.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE=', 'Content-Length': '0'} 2024-03-04T17:57:13.320 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000044d3ad95b704fae8-0065e60b79-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:13.320 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.320 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.320 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.320 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.320 DEBUG:boto:Method: GET 2024-03-04T17:57:13.320 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.321 DEBUG:boto:Data: 2024-03-04T17:57:13.321 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.321 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.321 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.321 DEBUG:boto:Params: {} 2024-03-04T17:57:13.321 DEBUG:boto:Token: None 2024-03-04T17:57:13.321 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.321 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA= 2024-03-04T17:57:13.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA=', 'Content-Length': '0'} 2024-03-04T17:57:13.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000079a9442fc8d40ae9-0065e60b79-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.324 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.324 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.324 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.324 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.324 DEBUG:boto:Method: GET 2024-03-04T17:57:13.324 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.324 DEBUG:boto:Data: 2024-03-04T17:57:13.325 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.325 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.325 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.325 DEBUG:boto:Params: {} 2024-03-04T17:57:13.325 DEBUG:boto:Token: None 2024-03-04T17:57:13.325 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.325 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA= 2024-03-04T17:57:13.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA=', 'Content-Length': '0'} 2024-03-04T17:57:13.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-25:d96636ac-054d-40e3-90f3-e3912da784b2.4349.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000417776878e30e36a-0065e60b79-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.328 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.328 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.328 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.329 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.329 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.329 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.329 DEBUG:boto:Data: 2024-03-04T17:57:13.329 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.329 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.329 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.329 DEBUG:boto:Params: {} 2024-03-04T17:57:13.329 DEBUG:boto:Token: None 2024-03-04T17:57:13.329 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.330 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hBKNWOBB+tKhs/gXR+Q/CyUsND8= 2024-03-04T17:57:13.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hBKNWOBB+tKhs/gXR+Q/CyUsND8=', 'Content-Length': '0'} 2024-03-04T17:57:13.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002bb90c43d4c507cd-0065e60b79-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.333 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.333 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.333 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.333 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.333 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.333 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.333 DEBUG:boto:Data: 2024-03-04T17:57:13.333 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.333 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.333 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.333 DEBUG:boto:Params: {} 2024-03-04T17:57:13.334 DEBUG:boto:Token: None 2024-03-04T17:57:13.334 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.334 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hBKNWOBB+tKhs/gXR+Q/CyUsND8= 2024-03-04T17:57:13.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hBKNWOBB+tKhs/gXR+Q/CyUsND8=', 'Content-Length': '0'} 2024-03-04T17:57:13.337 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-25:d96636ac-054d-40e3-90f3-e3912da784b2.4349.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e8bdff36809be76-0065e60b79-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:13.337 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.337 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.337 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.337 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.337 DEBUG:boto:Method: GET 2024-03-04T17:57:13.337 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.337 DEBUG:boto:Data: 2024-03-04T17:57:13.338 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.338 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.338 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.338 DEBUG:boto:Params: {} 2024-03-04T17:57:13.338 DEBUG:boto:Token: None 2024-03-04T17:57:13.338 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.338 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA= 2024-03-04T17:57:13.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA=', 'Content-Length': '0'} 2024-03-04T17:57:13.341 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a253bfe9476c680b-0065e60b79-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.341 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.341 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.341 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.341 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.342 DEBUG:boto:Method: GET 2024-03-04T17:57:13.342 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.342 DEBUG:boto:Data: 2024-03-04T17:57:13.342 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.342 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.342 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.342 DEBUG:boto:Params: {} 2024-03-04T17:57:13.342 DEBUG:boto:Token: None 2024-03-04T17:57:13.342 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.342 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA= 2024-03-04T17:57:13.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA=', 'Content-Length': '0'} 2024-03-04T17:57:13.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-25:d96636ac-054d-40e3-90f3-e3912da784b2.4349.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b82e19bcd481f1be-0065e60b79-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.346 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.346 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.346 DEBUG:boto:path=/ppfahn-25/?prefix=obj 2024-03-04T17:57:13.346 DEBUG:boto:auth_path=/ppfahn-25/?prefix=obj 2024-03-04T17:57:13.346 DEBUG:boto:Method: GET 2024-03-04T17:57:13.346 DEBUG:boto:Path: /ppfahn-25/?prefix=obj 2024-03-04T17:57:13.346 DEBUG:boto:Data: 2024-03-04T17:57:13.347 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.347 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.347 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.347 DEBUG:boto:Params: {} 2024-03-04T17:57:13.347 DEBUG:boto:Token: None 2024-03-04T17:57:13.347 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/ 2024-03-04T17:57:13.347 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tU7etnz6HBnReVP23vsO1cQ20Lg= 2024-03-04T17:57:13.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tU7etnz6HBnReVP23vsO1cQ20Lg=', 'Content-Length': '0'} 2024-03-04T17:57:13.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbd8475e190d1cc0-0065e60b79-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.392 DEBUG:boto:b'ppfahn-25obj1000false' 2024-03-04T17:57:13.392 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.392 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.392 DEBUG:boto:path=/ppfahn-25/?prefix=obj 2024-03-04T17:57:13.393 DEBUG:boto:auth_path=/ppfahn-25/?prefix=obj 2024-03-04T17:57:13.393 DEBUG:boto:Method: GET 2024-03-04T17:57:13.393 DEBUG:boto:Path: /ppfahn-25/?prefix=obj 2024-03-04T17:57:13.393 DEBUG:boto:Data: 2024-03-04T17:57:13.393 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.393 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.393 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.393 DEBUG:boto:Params: {} 2024-03-04T17:57:13.393 DEBUG:boto:Token: None 2024-03-04T17:57:13.393 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/ 2024-03-04T17:57:13.393 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tU7etnz6HBnReVP23vsO1cQ20Lg= 2024-03-04T17:57:13.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tU7etnz6HBnReVP23vsO1cQ20Lg=', 'Content-Length': '0'} 2024-03-04T17:57:13.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eae96777d76e13c8-0065e60b79-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.398 DEBUG:boto:b'ppfahn-25obj1000false' 2024-03-04T17:57:13.399 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.399 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.399 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.399 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:57:13.399 DEBUG:boto:Data: 2024-03-04T17:57:13.399 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.399 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.399 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.399 DEBUG:boto:Params: {} 2024-03-04T17:57:13.399 DEBUG:boto:Token: None 2024-03-04T17:57:13.400 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj 2024-03-04T17:57:13.400 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE= 2024-03-04T17:57:13.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE=', 'Content-Length': '0'} 2024-03-04T17:57:13.403 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000016fe0ce2a60a8ab-0065e60b79-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.403 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.404 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.404 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.404 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:57:13.404 DEBUG:boto:Data: 2024-03-04T17:57:13.404 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.404 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.404 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.404 DEBUG:boto:Params: {} 2024-03-04T17:57:13.404 DEBUG:boto:Token: None 2024-03-04T17:57:13.404 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj 2024-03-04T17:57:13.405 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE= 2024-03-04T17:57:13.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE=', 'Content-Length': '0'} 2024-03-04T17:57:13.408 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000cc9e984bcdca4c58-0065e60b79-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.408 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-25 zones={test-zone2, test-zone1} 2024-03-04T17:57:13.408 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-25 zones={test-zone3, test-zone1} 2024-03-04T17:57:13.408 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.408 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.409 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.409 DEBUG:boto:Path: /ppfahn-25/ 2024-03-04T17:57:13.409 DEBUG:boto:Data: 2024-03-04T17:57:13.409 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.409 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.409 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.409 DEBUG:boto:Params: {} 2024-03-04T17:57:13.409 DEBUG:boto:Token: None 2024-03-04T17:57:13.409 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/ 2024-03-04T17:57:13.409 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1qyZtLhN100tl470Y0Et9S9C9rs= 2024-03-04T17:57:13.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1qyZtLhN100tl470Y0Et9S9C9rs=', 'Content-Length': '0'} 2024-03-04T17:57:13.414 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e629ea49fb05559d-0065e60b79-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.414 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.414 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.415 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.415 DEBUG:boto:Path: /ppfahn-25/ 2024-03-04T17:57:13.415 DEBUG:boto:Data: 2024-03-04T17:57:13.415 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.415 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.415 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.415 DEBUG:boto:Params: {} 2024-03-04T17:57:13.415 DEBUG:boto:Token: None 2024-03-04T17:57:13.415 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/ 2024-03-04T17:57:13.415 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1qyZtLhN100tl470Y0Et9S9C9rs= 2024-03-04T17:57:13.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1qyZtLhN100tl470Y0Et9S9C9rs=', 'Content-Length': '0'} 2024-03-04T17:57:13.420 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6390227b8978d9b-0065e60b79-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.420 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:13.420 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.420 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.420 DEBUG:boto:path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.420 DEBUG:boto:auth_path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.420 DEBUG:boto:Method: GET 2024-03-04T17:57:13.420 DEBUG:boto:Path: /ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.421 DEBUG:boto:Data: 2024-03-04T17:57:13.421 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.421 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.421 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.421 DEBUG:boto:Params: {} 2024-03-04T17:57:13.421 DEBUG:boto:Token: None 2024-03-04T17:57:13.421 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/?versions 2024-03-04T17:57:13.421 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8= 2024-03-04T17:57:13.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8=', 'Content-Length': '0'} 2024-03-04T17:57:13.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080a59d446f5b9127-0065e60b79-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.426 DEBUG:boto:b'ppfahn-25999falseobjy476gJad3qtgVOCqSNyCSIsViYsdZnMtrue2024-03-04T17:56:35.937Zrgw-multisite-test-userMultisite Test UserNormalobjAytj.t3jFObNFOXGB1.vARQ1kbJTYQTfalse2024-03-04T17:56:35.902Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:13.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:13.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:13.427 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:13.427 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.427 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.427 DEBUG:boto:path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.427 DEBUG:boto:auth_path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.427 DEBUG:boto:Method: GET 2024-03-04T17:57:13.427 DEBUG:boto:Path: /ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.427 DEBUG:boto:Data: 2024-03-04T17:57:13.427 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.427 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.428 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.428 DEBUG:boto:Params: {} 2024-03-04T17:57:13.428 DEBUG:boto:Token: None 2024-03-04T17:57:13.428 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/?versions 2024-03-04T17:57:13.428 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8= 2024-03-04T17:57:13.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8=', 'Content-Length': '0'} 2024-03-04T17:57:13.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fb36bf85a1db116-0065e60b79-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.476 DEBUG:boto:b'ppfahn-25999falseobjy476gJad3qtgVOCqSNyCSIsViYsdZnMtrue2024-03-04T17:56:35.937Zrgw-multisite-test-userMultisite Test UserNormalobjAytj.t3jFObNFOXGB1.vARQ1kbJTYQTfalse2024-03-04T17:56:35.902Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:13.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:13.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:13.476 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.477 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.477 DEBUG:boto:path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.477 DEBUG:boto:auth_path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.477 DEBUG:boto:Method: GET 2024-03-04T17:57:13.477 DEBUG:boto:Path: /ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.477 DEBUG:boto:Data: 2024-03-04T17:57:13.477 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.477 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.477 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.477 DEBUG:boto:Params: {} 2024-03-04T17:57:13.477 DEBUG:boto:Token: None 2024-03-04T17:57:13.478 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/?versions 2024-03-04T17:57:13.478 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8= 2024-03-04T17:57:13.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8=', 'Content-Length': '0'} 2024-03-04T17:57:13.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9c5feaf0492fdec-0065e60b79-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.482 DEBUG:boto:b'ppfahn-25999falseobjy476gJad3qtgVOCqSNyCSIsViYsdZnMtrue2024-03-04T17:56:35.937Zrgw-multisite-test-userMultisite Test UserNormalobjAytj.t3jFObNFOXGB1.vARQ1kbJTYQTfalse2024-03-04T17:56:35.902Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:13.483 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.483 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.483 DEBUG:boto:path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.483 DEBUG:boto:auth_path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.483 DEBUG:boto:Method: GET 2024-03-04T17:57:13.483 DEBUG:boto:Path: /ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.483 DEBUG:boto:Data: 2024-03-04T17:57:13.483 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.483 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.483 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.484 DEBUG:boto:Params: {} 2024-03-04T17:57:13.484 DEBUG:boto:Token: None 2024-03-04T17:57:13.484 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/?versions 2024-03-04T17:57:13.484 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8= 2024-03-04T17:57:13.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8=', 'Content-Length': '0'} 2024-03-04T17:57:13.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2846a3d5ca7f92c-0065e60b79-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.532 DEBUG:boto:b'ppfahn-25999falseobjy476gJad3qtgVOCqSNyCSIsViYsdZnMtrue2024-03-04T17:56:35.937Zrgw-multisite-test-userMultisite Test UserNormalobjAytj.t3jFObNFOXGB1.vARQ1kbJTYQTfalse2024-03-04T17:56:35.902Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:13.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:13.532 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.533 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.533 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.533 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:57:13.533 DEBUG:boto:Data: 2024-03-04T17:57:13.533 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.533 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.533 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.533 DEBUG:boto:Params: {} 2024-03-04T17:57:13.533 DEBUG:boto:Token: None 2024-03-04T17:57:13.533 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj 2024-03-04T17:57:13.534 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE= 2024-03-04T17:57:13.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE=', 'Content-Length': '0'} 2024-03-04T17:57:13.537 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c9c803fb2dddf704-0065e60b79-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.537 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.537 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.538 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.538 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:57:13.538 DEBUG:boto:Data: 2024-03-04T17:57:13.538 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.538 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.538 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.538 DEBUG:boto:Params: {} 2024-03-04T17:57:13.538 DEBUG:boto:Token: None 2024-03-04T17:57:13.538 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj 2024-03-04T17:57:13.538 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE= 2024-03-04T17:57:13.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE=', 'Content-Length': '0'} 2024-03-04T17:57:13.542 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000002ef62c8f8bf70372-0065e60b79-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:13.542 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.542 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.543 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.543 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.543 DEBUG:boto:Method: GET 2024-03-04T17:57:13.543 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.543 DEBUG:boto:Data: 2024-03-04T17:57:13.543 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.543 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.543 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.543 DEBUG:boto:Params: {} 2024-03-04T17:57:13.543 DEBUG:boto:Token: None 2024-03-04T17:57:13.544 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.544 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA= 2024-03-04T17:57:13.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA=', 'Content-Length': '0'} 2024-03-04T17:57:13.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-25:d96636ac-054d-40e3-90f3-e3912da784b2.4349.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008fb2eedf99f7ca0b-0065e60b79-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.547 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.547 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.547 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.547 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.547 DEBUG:boto:Method: GET 2024-03-04T17:57:13.547 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.547 DEBUG:boto:Data: 2024-03-04T17:57:13.547 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.547 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.547 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.548 DEBUG:boto:Params: {} 2024-03-04T17:57:13.548 DEBUG:boto:Token: None 2024-03-04T17:57:13.548 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.548 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA= 2024-03-04T17:57:13.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA=', 'Content-Length': '0'} 2024-03-04T17:57:13.551 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-25:d96636ac-054d-40e3-90f3-e3912da784b2.4349.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b73ed12a1d33bd4a-0065e60b79-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.551 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.551 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.551 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.551 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.551 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.551 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.551 DEBUG:boto:Data: 2024-03-04T17:57:13.551 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.552 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.552 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.552 DEBUG:boto:Params: {} 2024-03-04T17:57:13.552 DEBUG:boto:Token: None 2024-03-04T17:57:13.552 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.552 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hBKNWOBB+tKhs/gXR+Q/CyUsND8= 2024-03-04T17:57:13.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hBKNWOBB+tKhs/gXR+Q/CyUsND8=', 'Content-Length': '0'} 2024-03-04T17:57:13.555 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-25:d96636ac-054d-40e3-90f3-e3912da784b2.4349.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cbbf8ef96b8c621a-0065e60b79-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.555 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.555 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.555 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.555 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.555 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.555 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.555 DEBUG:boto:Data: 2024-03-04T17:57:13.556 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.556 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.556 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.556 DEBUG:boto:Params: {} 2024-03-04T17:57:13.556 DEBUG:boto:Token: None 2024-03-04T17:57:13.556 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.556 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hBKNWOBB+tKhs/gXR+Q/CyUsND8= 2024-03-04T17:57:13.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hBKNWOBB+tKhs/gXR+Q/CyUsND8=', 'Content-Length': '0'} 2024-03-04T17:57:13.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-25:d96636ac-054d-40e3-90f3-e3912da784b2.4349.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002821ba57dc1b5040-0065e60b79-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:13.559 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.560 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.560 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.560 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.560 DEBUG:boto:Method: GET 2024-03-04T17:57:13.560 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.560 DEBUG:boto:Data: 2024-03-04T17:57:13.560 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.560 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.560 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.560 DEBUG:boto:Params: {} 2024-03-04T17:57:13.560 DEBUG:boto:Token: None 2024-03-04T17:57:13.561 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.561 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA= 2024-03-04T17:57:13.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA=', 'Content-Length': '0'} 2024-03-04T17:57:13.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-25:d96636ac-054d-40e3-90f3-e3912da784b2.4349.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000862ecd88bcb8a55-0065e60b79-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.564 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.564 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.564 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.564 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.564 DEBUG:boto:Method: GET 2024-03-04T17:57:13.564 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.564 DEBUG:boto:Data: 2024-03-04T17:57:13.565 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.565 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.565 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.565 DEBUG:boto:Params: {} 2024-03-04T17:57:13.565 DEBUG:boto:Token: None 2024-03-04T17:57:13.565 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.565 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA= 2024-03-04T17:57:13.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA=', 'Content-Length': '0'} 2024-03-04T17:57:13.568 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-25:d96636ac-054d-40e3-90f3-e3912da784b2.4349.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a01ad8f604dfce6-0065e60b79-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.568 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.569 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.569 DEBUG:boto:path=/ppfahn-25/?prefix=obj 2024-03-04T17:57:13.569 DEBUG:boto:auth_path=/ppfahn-25/?prefix=obj 2024-03-04T17:57:13.569 DEBUG:boto:Method: GET 2024-03-04T17:57:13.569 DEBUG:boto:Path: /ppfahn-25/?prefix=obj 2024-03-04T17:57:13.569 DEBUG:boto:Data: 2024-03-04T17:57:13.569 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.569 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.569 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.569 DEBUG:boto:Params: {} 2024-03-04T17:57:13.569 DEBUG:boto:Token: None 2024-03-04T17:57:13.570 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/ 2024-03-04T17:57:13.570 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tU7etnz6HBnReVP23vsO1cQ20Lg= 2024-03-04T17:57:13.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tU7etnz6HBnReVP23vsO1cQ20Lg=', 'Content-Length': '0'} 2024-03-04T17:57:13.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002991aa718e56d96-0065e60b79-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.616 DEBUG:boto:b'ppfahn-25obj1000false' 2024-03-04T17:57:13.616 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.616 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.617 DEBUG:boto:path=/ppfahn-25/?prefix=obj 2024-03-04T17:57:13.617 DEBUG:boto:auth_path=/ppfahn-25/?prefix=obj 2024-03-04T17:57:13.617 DEBUG:boto:Method: GET 2024-03-04T17:57:13.617 DEBUG:boto:Path: /ppfahn-25/?prefix=obj 2024-03-04T17:57:13.617 DEBUG:boto:Data: 2024-03-04T17:57:13.617 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.617 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.617 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.617 DEBUG:boto:Params: {} 2024-03-04T17:57:13.617 DEBUG:boto:Token: None 2024-03-04T17:57:13.617 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/ 2024-03-04T17:57:13.618 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tU7etnz6HBnReVP23vsO1cQ20Lg= 2024-03-04T17:57:13.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tU7etnz6HBnReVP23vsO1cQ20Lg=', 'Content-Length': '0'} 2024-03-04T17:57:13.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023b56e6729f146b2-0065e60b79-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.664 DEBUG:boto:b'ppfahn-25obj1000false' 2024-03-04T17:57:13.664 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.664 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.664 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.665 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:57:13.665 DEBUG:boto:Data: 2024-03-04T17:57:13.665 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.665 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.665 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.665 DEBUG:boto:Params: {} 2024-03-04T17:57:13.665 DEBUG:boto:Token: None 2024-03-04T17:57:13.665 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj 2024-03-04T17:57:13.665 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE= 2024-03-04T17:57:13.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE=', 'Content-Length': '0'} 2024-03-04T17:57:13.669 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e834b385971cf568-0065e60b79-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.669 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.669 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.669 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.669 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:57:13.670 DEBUG:boto:Data: 2024-03-04T17:57:13.670 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.670 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:13.670 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.670 DEBUG:boto:Params: {} 2024-03-04T17:57:13.670 DEBUG:boto:Token: None 2024-03-04T17:57:13.670 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj 2024-03-04T17:57:13.670 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE= 2024-03-04T17:57:13.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE=', 'Content-Length': '0'} 2024-03-04T17:57:13.674 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000079b100d2c097fb0f-0065e60b79-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.674 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-25 zones={test-zone3, test-zone1} 2024-03-04T17:57:13.674 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-25 zones={test-zone3, test-zone2} 2024-03-04T17:57:13.674 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.674 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.674 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.674 DEBUG:boto:Path: /ppfahn-25/ 2024-03-04T17:57:13.674 DEBUG:boto:Data: 2024-03-04T17:57:13.675 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.675 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.675 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.675 DEBUG:boto:Params: {} 2024-03-04T17:57:13.675 DEBUG:boto:Token: None 2024-03-04T17:57:13.675 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/ 2024-03-04T17:57:13.675 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1qyZtLhN100tl470Y0Et9S9C9rs= 2024-03-04T17:57:13.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1qyZtLhN100tl470Y0Et9S9C9rs=', 'Content-Length': '0'} 2024-03-04T17:57:13.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000675a5a061c30c9fd-0065e60b79-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.679 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.679 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.680 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.680 DEBUG:boto:Path: /ppfahn-25/ 2024-03-04T17:57:13.680 DEBUG:boto:Data: 2024-03-04T17:57:13.680 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.680 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.680 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.680 DEBUG:boto:Params: {} 2024-03-04T17:57:13.680 DEBUG:boto:Token: None 2024-03-04T17:57:13.680 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/ 2024-03-04T17:57:13.680 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:1qyZtLhN100tl470Y0Et9S9C9rs= 2024-03-04T17:57:13.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1qyZtLhN100tl470Y0Et9S9C9rs=', 'Content-Length': '0'} 2024-03-04T17:57:13.685 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000180406ff00bedcfe-0065e60b79-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.685 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:13.685 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.685 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.685 DEBUG:boto:path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.686 DEBUG:boto:auth_path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.686 DEBUG:boto:Method: GET 2024-03-04T17:57:13.686 DEBUG:boto:Path: /ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.686 DEBUG:boto:Data: 2024-03-04T17:57:13.686 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.686 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.686 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.686 DEBUG:boto:Params: {} 2024-03-04T17:57:13.686 DEBUG:boto:Token: None 2024-03-04T17:57:13.686 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/?versions 2024-03-04T17:57:13.686 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8= 2024-03-04T17:57:13.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8=', 'Content-Length': '0'} 2024-03-04T17:57:13.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba1b16e0288c4a7b-0065e60b79-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.691 DEBUG:boto:b'ppfahn-25999falseobjy476gJad3qtgVOCqSNyCSIsViYsdZnMtrue2024-03-04T17:56:35.937Zrgw-multisite-test-userMultisite Test UserNormalobjAytj.t3jFObNFOXGB1.vARQ1kbJTYQTfalse2024-03-04T17:56:35.902Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:13.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:13.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:13.692 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:13.692 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.692 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.692 DEBUG:boto:path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.692 DEBUG:boto:auth_path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.692 DEBUG:boto:Method: GET 2024-03-04T17:57:13.692 DEBUG:boto:Path: /ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.692 DEBUG:boto:Data: 2024-03-04T17:57:13.692 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.692 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.693 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.693 DEBUG:boto:Params: {} 2024-03-04T17:57:13.693 DEBUG:boto:Token: None 2024-03-04T17:57:13.693 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/?versions 2024-03-04T17:57:13.693 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8= 2024-03-04T17:57:13.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8=', 'Content-Length': '0'} 2024-03-04T17:57:13.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce9916c90eecd581-0065e60b79-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.698 DEBUG:boto:b'ppfahn-25999falseobjy476gJad3qtgVOCqSNyCSIsViYsdZnMtrue2024-03-04T17:56:35.937Zrgw-multisite-test-userMultisite Test UserNormalobjAytj.t3jFObNFOXGB1.vARQ1kbJTYQTfalse2024-03-04T17:56:35.902Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:13.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:13.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:13.699 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.699 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.699 DEBUG:boto:path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.699 DEBUG:boto:auth_path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.699 DEBUG:boto:Method: GET 2024-03-04T17:57:13.699 DEBUG:boto:Path: /ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.699 DEBUG:boto:Data: 2024-03-04T17:57:13.699 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.700 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.700 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.700 DEBUG:boto:Params: {} 2024-03-04T17:57:13.700 DEBUG:boto:Token: None 2024-03-04T17:57:13.700 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/?versions 2024-03-04T17:57:13.700 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8= 2024-03-04T17:57:13.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8=', 'Content-Length': '0'} 2024-03-04T17:57:13.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084a121d9cae868f4-0065e60b79-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.748 DEBUG:boto:b'ppfahn-25999falseobjy476gJad3qtgVOCqSNyCSIsViYsdZnMtrue2024-03-04T17:56:35.937Zrgw-multisite-test-userMultisite Test UserNormalobjAytj.t3jFObNFOXGB1.vARQ1kbJTYQTfalse2024-03-04T17:56:35.902Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:13.748 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.749 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.749 DEBUG:boto:path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.749 DEBUG:boto:auth_path=/ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.749 DEBUG:boto:Method: GET 2024-03-04T17:57:13.749 DEBUG:boto:Path: /ppfahn-25/?versions&max-keys=999 2024-03-04T17:57:13.749 DEBUG:boto:Data: 2024-03-04T17:57:13.749 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.749 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.749 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.749 DEBUG:boto:Params: {} 2024-03-04T17:57:13.749 DEBUG:boto:Token: None 2024-03-04T17:57:13.750 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/?versions 2024-03-04T17:57:13.750 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8= 2024-03-04T17:57:13.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ApCaEa1wW9WriPFHqppGWQ2NH/8=', 'Content-Length': '0'} 2024-03-04T17:57:13.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db61968cda431d46-0065e60b79-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.755 DEBUG:boto:b'ppfahn-25999falseobjy476gJad3qtgVOCqSNyCSIsViYsdZnMtrue2024-03-04T17:56:35.937Zrgw-multisite-test-userMultisite Test UserNormalobjAytj.t3jFObNFOXGB1.vARQ1kbJTYQTfalse2024-03-04T17:56:35.902Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:13.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:13.755 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.755 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.755 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.755 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:57:13.755 DEBUG:boto:Data: 2024-03-04T17:57:13.756 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.756 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.756 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.756 DEBUG:boto:Params: {} 2024-03-04T17:57:13.756 DEBUG:boto:Token: None 2024-03-04T17:57:13.756 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj 2024-03-04T17:57:13.756 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE= 2024-03-04T17:57:13.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE=', 'Content-Length': '0'} 2024-03-04T17:57:13.760 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000005e893d088de8a72b-0065e60b79-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.760 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.760 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.760 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.760 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:57:13.761 DEBUG:boto:Data: 2024-03-04T17:57:13.761 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.761 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.761 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.761 DEBUG:boto:Params: {} 2024-03-04T17:57:13.761 DEBUG:boto:Token: None 2024-03-04T17:57:13.761 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj 2024-03-04T17:57:13.761 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE= 2024-03-04T17:57:13.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE=', 'Content-Length': '0'} 2024-03-04T17:57:13.764 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000f59b05295277888f-0065e60b79-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:13.765 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.765 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.765 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.765 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.765 DEBUG:boto:Method: GET 2024-03-04T17:57:13.765 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.765 DEBUG:boto:Data: 2024-03-04T17:57:13.765 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.765 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.766 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.766 DEBUG:boto:Params: {} 2024-03-04T17:57:13.766 DEBUG:boto:Token: None 2024-03-04T17:57:13.766 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.766 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA= 2024-03-04T17:57:13.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA=', 'Content-Length': '0'} 2024-03-04T17:57:13.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-25:d96636ac-054d-40e3-90f3-e3912da784b2.4349.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e35547f25a7b5d45-0065e60b79-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.769 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.769 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.769 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.769 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.769 DEBUG:boto:Method: GET 2024-03-04T17:57:13.769 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.770 DEBUG:boto:Data: 2024-03-04T17:57:13.770 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.770 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.770 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.770 DEBUG:boto:Params: {} 2024-03-04T17:57:13.770 DEBUG:boto:Token: None 2024-03-04T17:57:13.770 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.770 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA= 2024-03-04T17:57:13.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA=', 'Content-Length': '0'} 2024-03-04T17:57:13.773 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d2890732fa4e962f-0065e60b79-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.773 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.774 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.774 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.774 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.774 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.774 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.774 DEBUG:boto:Data: 2024-03-04T17:57:13.774 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.774 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.774 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.774 DEBUG:boto:Params: {} 2024-03-04T17:57:13.775 DEBUG:boto:Token: None 2024-03-04T17:57:13.775 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.775 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hBKNWOBB+tKhs/gXR+Q/CyUsND8= 2024-03-04T17:57:13.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hBKNWOBB+tKhs/gXR+Q/CyUsND8=', 'Content-Length': '0'} 2024-03-04T17:57:13.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-25:d96636ac-054d-40e3-90f3-e3912da784b2.4349.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000027c54d327589d430-0065e60b79-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.778 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.778 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.778 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.778 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.779 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.779 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.779 DEBUG:boto:Data: 2024-03-04T17:57:13.779 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.779 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.779 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.779 DEBUG:boto:Params: {} 2024-03-04T17:57:13.779 DEBUG:boto:Token: None 2024-03-04T17:57:13.779 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.779 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hBKNWOBB+tKhs/gXR+Q/CyUsND8= 2024-03-04T17:57:13.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hBKNWOBB+tKhs/gXR+Q/CyUsND8=', 'Content-Length': '0'} 2024-03-04T17:57:13.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fc07ea01e70c7f3b-0065e60b79-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:13.783 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.783 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.783 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.783 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.783 DEBUG:boto:Method: GET 2024-03-04T17:57:13.783 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.783 DEBUG:boto:Data: 2024-03-04T17:57:13.783 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.783 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.784 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.784 DEBUG:boto:Params: {} 2024-03-04T17:57:13.784 DEBUG:boto:Token: None 2024-03-04T17:57:13.784 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.784 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA= 2024-03-04T17:57:13.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA=', 'Content-Length': '0'} 2024-03-04T17:57:13.787 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-25:d96636ac-054d-40e3-90f3-e3912da784b2.4349.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003a4374a9dc337633-0065e60b79-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.787 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.788 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.788 DEBUG:boto:path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.788 DEBUG:boto:auth_path=/ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.788 DEBUG:boto:Method: GET 2024-03-04T17:57:13.788 DEBUG:boto:Path: /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.788 DEBUG:boto:Data: 2024-03-04T17:57:13.788 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.788 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.788 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.789 DEBUG:boto:Params: {} 2024-03-04T17:57:13.789 DEBUG:boto:Token: None 2024-03-04T17:57:13.789 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj?versionId=Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT 2024-03-04T17:57:13.789 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA= 2024-03-04T17:57:13.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Hpz59Oj91uNEf9ypuauJnZkeVrA=', 'Content-Length': '0'} 2024-03-04T17:57:13.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', 'Aytj.t3jFObNFOXGB1.vARQ1kbJTYQT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000382ebfb0299b7772-0065e60b79-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.792 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.792 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.792 DEBUG:boto:path=/ppfahn-25/?prefix=obj 2024-03-04T17:57:13.792 DEBUG:boto:auth_path=/ppfahn-25/?prefix=obj 2024-03-04T17:57:13.793 DEBUG:boto:Method: GET 2024-03-04T17:57:13.793 DEBUG:boto:Path: /ppfahn-25/?prefix=obj 2024-03-04T17:57:13.793 DEBUG:boto:Data: 2024-03-04T17:57:13.793 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.793 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.793 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.793 DEBUG:boto:Params: {} 2024-03-04T17:57:13.793 DEBUG:boto:Token: None 2024-03-04T17:57:13.793 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/ 2024-03-04T17:57:13.794 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tU7etnz6HBnReVP23vsO1cQ20Lg= 2024-03-04T17:57:13.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tU7etnz6HBnReVP23vsO1cQ20Lg=', 'Content-Length': '0'} 2024-03-04T17:57:13.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3b582895447e45d-0065e60b79-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.840 DEBUG:boto:b'ppfahn-25obj1000false' 2024-03-04T17:57:13.840 DEBUG:boto:path=/ppfahn-25/ 2024-03-04T17:57:13.840 DEBUG:boto:auth_path=/ppfahn-25/ 2024-03-04T17:57:13.841 DEBUG:boto:path=/ppfahn-25/?prefix=obj 2024-03-04T17:57:13.841 DEBUG:boto:auth_path=/ppfahn-25/?prefix=obj 2024-03-04T17:57:13.841 DEBUG:boto:Method: GET 2024-03-04T17:57:13.841 DEBUG:boto:Path: /ppfahn-25/?prefix=obj 2024-03-04T17:57:13.841 DEBUG:boto:Data: 2024-03-04T17:57:13.841 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.841 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.841 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.841 DEBUG:boto:Params: {} 2024-03-04T17:57:13.841 DEBUG:boto:Token: None 2024-03-04T17:57:13.842 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/ 2024-03-04T17:57:13.842 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tU7etnz6HBnReVP23vsO1cQ20Lg= 2024-03-04T17:57:13.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tU7etnz6HBnReVP23vsO1cQ20Lg=', 'Content-Length': '0'} 2024-03-04T17:57:13.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b71946e8dcbde6d-0065e60b79-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.846 DEBUG:boto:b'ppfahn-25obj1000false' 2024-03-04T17:57:13.847 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.847 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.847 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.847 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:57:13.847 DEBUG:boto:Data: 2024-03-04T17:57:13.847 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.847 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:13.847 DEBUG:boto:Port: 8001 2024-03-04T17:57:13.847 DEBUG:boto:Params: {} 2024-03-04T17:57:13.847 DEBUG:boto:Token: None 2024-03-04T17:57:13.847 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj 2024-03-04T17:57:13.848 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE= 2024-03-04T17:57:13.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE=', 'Content-Length': '0'} 2024-03-04T17:57:13.851 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000007d0663dee8f5c3f2-0065e60b79-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.852 DEBUG:boto:path=/ppfahn-25/obj 2024-03-04T17:57:13.852 DEBUG:boto:auth_path=/ppfahn-25/obj 2024-03-04T17:57:13.852 DEBUG:boto:Method: HEAD 2024-03-04T17:57:13.852 DEBUG:boto:Path: /ppfahn-25/obj 2024-03-04T17:57:13.852 DEBUG:boto:Data: 2024-03-04T17:57:13.852 DEBUG:boto:Headers: {} 2024-03-04T17:57:13.852 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:13.852 DEBUG:boto:Port: 8000 2024-03-04T17:57:13.852 DEBUG:boto:Params: {} 2024-03-04T17:57:13.852 DEBUG:boto:Token: None 2024-03-04T17:57:13.853 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:13 GMT /ppfahn-25/obj 2024-03-04T17:57:13.853 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE= 2024-03-04T17:57:13.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:13 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+mmlQA9HHoGAqSGDdFGhEMmZxlE=', 'Content-Length': '0'} 2024-03-04T17:57:13.856 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e723937c84c516b6-0065e60b79-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:13.856 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-25 zones={test-zone3, test-zone2} 2024-03-04T17:57:13.856 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-26 2024-03-04T17:57:13.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:13.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:13.857 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:13.907 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:13.907 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:13.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:13.976+0000 7fbd1bc34980 1 bucket sync caught up with source: 2024-03-04T17:57:13.973 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000002.13.6, , ] 2024-03-04T17:57:13.973 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000002.13.6, , ] 2024-03-04T17:57:13.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:13.976+0000 7fbd1bc34980 0 bucket checkpoint complete 2024-03-04T17:57:13.995 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-26 2024-03-04T17:57:13.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:13.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:13.996 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:14.045 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:14.045 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:14.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:14.105+0000 7fc78a69e980 1 bucket sync caught up with source: 2024-03-04T17:57:14.106 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000002.13.6, , ] 2024-03-04T17:57:14.106 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000002.13.6, , ] 2024-03-04T17:57:14.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:14.105+0000 7fc78a69e980 0 bucket checkpoint complete 2024-03-04T17:57:14.128 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-26 2024-03-04T17:57:14.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:14.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:14.128 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:14.169 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:14.169 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:14.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:14.229+0000 7f4372e08980 1 bucket sync caught up with source: 2024-03-04T17:57:14.231 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000002.21.6, , ] 2024-03-04T17:57:14.231 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000002.21.6, , ] 2024-03-04T17:57:14.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:14.229+0000 7f4372e08980 0 bucket checkpoint complete 2024-03-04T17:57:14.253 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-26 2024-03-04T17:57:14.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:14.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:14.253 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:14.337 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:14.338 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:14.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:14.397+0000 7fa6fda77980 1 bucket sync caught up with source: 2024-03-04T17:57:14.399 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000002.21.6, , ] 2024-03-04T17:57:14.399 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000002.21.6, , ] 2024-03-04T17:57:14.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:14.397+0000 7fa6fda77980 0 bucket checkpoint complete 2024-03-04T17:57:14.421 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-26 2024-03-04T17:57:14.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:14.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:14.421 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:14.510 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:14.510 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:14.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:14.569+0000 7f0baf219980 1 bucket sync caught up with source: 2024-03-04T17:57:14.568 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000003.60.12, , ] 2024-03-04T17:57:14.568 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.60.12, , ] 2024-03-04T17:57:14.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:14.569+0000 7f0baf219980 0 bucket checkpoint complete 2024-03-04T17:57:14.590 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-26 2024-03-04T17:57:14.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:14.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:14.591 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:14.641 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:14.641 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:14.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:14.708+0000 7fe4d071c980 1 bucket sync caught up with source: 2024-03-04T17:57:14.705 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000003.60.12, , ] 2024-03-04T17:57:14.705 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000003.60.12, , ] 2024-03-04T17:57:14.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:14.708+0000 7fe4d071c980 0 bucket checkpoint complete 2024-03-04T17:57:14.729 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-26 zones={test-zone2, test-zone1} 2024-03-04T17:57:14.729 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.729 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.729 DEBUG:boto:Method: HEAD 2024-03-04T17:57:14.729 DEBUG:boto:Path: /ppfahn-26/ 2024-03-04T17:57:14.729 DEBUG:boto:Data: 2024-03-04T17:57:14.729 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.729 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:14.730 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.730 DEBUG:boto:Params: {} 2024-03-04T17:57:14.730 DEBUG:boto:Token: None 2024-03-04T17:57:14.730 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/ 2024-03-04T17:57:14.730 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hq3jCG9y8UjnIsPC/q56g5+gNp0= 2024-03-04T17:57:14.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hq3jCG9y8UjnIsPC/q56g5+gNp0=', 'Content-Length': '0'} 2024-03-04T17:57:14.732 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074ca84d4e485249c-0065e60b7a-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.733 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.733 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.733 DEBUG:boto:Method: HEAD 2024-03-04T17:57:14.733 DEBUG:boto:Path: /ppfahn-26/ 2024-03-04T17:57:14.733 DEBUG:boto:Data: 2024-03-04T17:57:14.733 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.733 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:14.733 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.733 DEBUG:boto:Params: {} 2024-03-04T17:57:14.733 DEBUG:boto:Token: None 2024-03-04T17:57:14.734 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/ 2024-03-04T17:57:14.734 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hq3jCG9y8UjnIsPC/q56g5+gNp0= 2024-03-04T17:57:14.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hq3jCG9y8UjnIsPC/q56g5+gNp0=', 'Content-Length': '0'} 2024-03-04T17:57:14.737 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d92e0d99586a821-0065e60b7a-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.738 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:14.738 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.738 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.738 DEBUG:boto:path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.738 DEBUG:boto:auth_path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.738 DEBUG:boto:Method: GET 2024-03-04T17:57:14.738 DEBUG:boto:Path: /ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.738 DEBUG:boto:Data: 2024-03-04T17:57:14.738 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.738 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:14.738 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.739 DEBUG:boto:Params: {} 2024-03-04T17:57:14.739 DEBUG:boto:Token: None 2024-03-04T17:57:14.739 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/?versions 2024-03-04T17:57:14.739 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8= 2024-03-04T17:57:14.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8=', 'Content-Length': '0'} 2024-03-04T17:57:14.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab795d8eabf7b3c4-0065e60b7a-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.742 DEBUG:boto:b'ppfahn-26999falseobj4qnz-Ulcde6cJDXxKloMY8QwW.xzlq2true2024-03-04T17:56:36.015Zrgw-multisite-test-userMultisite Test UserNormalobj0WamgmIvOh.4g-Qrzmkk5TwhQskoikPfalse2024-03-04T17:56:35.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:14.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:14.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:14.742 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:14.742 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.742 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.743 DEBUG:boto:path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.743 DEBUG:boto:auth_path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.743 DEBUG:boto:Method: GET 2024-03-04T17:57:14.743 DEBUG:boto:Path: /ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.743 DEBUG:boto:Data: 2024-03-04T17:57:14.743 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.743 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:14.743 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.743 DEBUG:boto:Params: {} 2024-03-04T17:57:14.743 DEBUG:boto:Token: None 2024-03-04T17:57:14.744 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/?versions 2024-03-04T17:57:14.744 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8= 2024-03-04T17:57:14.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8=', 'Content-Length': '0'} 2024-03-04T17:57:14.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089fab97a2f656456-0065e60b7a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.748 DEBUG:boto:b'ppfahn-26999falseobj4qnz-Ulcde6cJDXxKloMY8QwW.xzlq2true2024-03-04T17:56:36.015Zrgw-multisite-test-userMultisite Test UserNormalobj0WamgmIvOh.4g-Qrzmkk5TwhQskoikPfalse2024-03-04T17:56:35.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:14.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:14.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:14.749 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.749 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.749 DEBUG:boto:path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.749 DEBUG:boto:auth_path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.749 DEBUG:boto:Method: GET 2024-03-04T17:57:14.749 DEBUG:boto:Path: /ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.749 DEBUG:boto:Data: 2024-03-04T17:57:14.749 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.749 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:14.750 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.750 DEBUG:boto:Params: {} 2024-03-04T17:57:14.750 DEBUG:boto:Token: None 2024-03-04T17:57:14.750 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/?versions 2024-03-04T17:57:14.750 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8= 2024-03-04T17:57:14.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8=', 'Content-Length': '0'} 2024-03-04T17:57:14.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000385f5e2f18c403b4-0065e60b7a-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.796 DEBUG:boto:b'ppfahn-26999falseobj4qnz-Ulcde6cJDXxKloMY8QwW.xzlq2true2024-03-04T17:56:36.015Zrgw-multisite-test-userMultisite Test UserNormalobj0WamgmIvOh.4g-Qrzmkk5TwhQskoikPfalse2024-03-04T17:56:35.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:14.796 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.796 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.797 DEBUG:boto:path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.797 DEBUG:boto:auth_path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.797 DEBUG:boto:Method: GET 2024-03-04T17:57:14.797 DEBUG:boto:Path: /ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.797 DEBUG:boto:Data: 2024-03-04T17:57:14.797 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.797 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:14.797 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.797 DEBUG:boto:Params: {} 2024-03-04T17:57:14.797 DEBUG:boto:Token: None 2024-03-04T17:57:14.797 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/?versions 2024-03-04T17:57:14.798 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8= 2024-03-04T17:57:14.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8=', 'Content-Length': '0'} 2024-03-04T17:57:14.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfe07ac815bb8d96-0065e60b7a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.802 DEBUG:boto:b'ppfahn-26999falseobj4qnz-Ulcde6cJDXxKloMY8QwW.xzlq2true2024-03-04T17:56:36.015Zrgw-multisite-test-userMultisite Test UserNormalobj0WamgmIvOh.4g-Qrzmkk5TwhQskoikPfalse2024-03-04T17:56:35.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:14.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:14.802 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:14.802 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:14.803 DEBUG:boto:Method: HEAD 2024-03-04T17:57:14.803 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:57:14.803 DEBUG:boto:Data: 2024-03-04T17:57:14.803 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.803 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:14.803 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.803 DEBUG:boto:Params: {} 2024-03-04T17:57:14.803 DEBUG:boto:Token: None 2024-03-04T17:57:14.803 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/obj 2024-03-04T17:57:14.803 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rhhpCiT4ZPfECF+HmANIo7aa/+Q= 2024-03-04T17:57:14.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rhhpCiT4ZPfECF+HmANIo7aa/+Q=', 'Content-Length': '0'} 2024-03-04T17:57:14.806 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000000cadf49c2e74d7bf-0065e60b7a-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.807 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:14.807 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:14.807 DEBUG:boto:Method: HEAD 2024-03-04T17:57:14.807 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:57:14.807 DEBUG:boto:Data: 2024-03-04T17:57:14.807 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.807 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:14.807 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.807 DEBUG:boto:Params: {} 2024-03-04T17:57:14.807 DEBUG:boto:Token: None 2024-03-04T17:57:14.808 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/obj 2024-03-04T17:57:14.808 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rhhpCiT4ZPfECF+HmANIo7aa/+Q= 2024-03-04T17:57:14.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rhhpCiT4ZPfECF+HmANIo7aa/+Q=', 'Content-Length': '0'} 2024-03-04T17:57:14.811 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000002c221491b261171f-0065e60b7a-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:14.812 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:14.812 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:14.812 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.812 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.812 DEBUG:boto:Method: GET 2024-03-04T17:57:14.812 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.812 DEBUG:boto:Data: 2024-03-04T17:57:14.812 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.812 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:14.812 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.812 DEBUG:boto:Params: {} 2024-03-04T17:57:14.813 DEBUG:boto:Token: None 2024-03-04T17:57:14.813 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.813 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OaTpCgHmz3wKdvfv0Vi/EOfBpwM= 2024-03-04T17:57:14.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OaTpCgHmz3wKdvfv0Vi/EOfBpwM=', 'Content-Length': '0'} 2024-03-04T17:57:14.816 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c6c133955141846d-0065e60b7a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.816 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:14.816 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:14.816 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.817 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.817 DEBUG:boto:Method: GET 2024-03-04T17:57:14.817 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.817 DEBUG:boto:Data: 2024-03-04T17:57:14.817 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.817 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:14.817 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.817 DEBUG:boto:Params: {} 2024-03-04T17:57:14.817 DEBUG:boto:Token: None 2024-03-04T17:57:14.817 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.818 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OaTpCgHmz3wKdvfv0Vi/EOfBpwM= 2024-03-04T17:57:14.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OaTpCgHmz3wKdvfv0Vi/EOfBpwM=', 'Content-Length': '0'} 2024-03-04T17:57:14.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000543185632d939492-0065e60b7a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.821 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:14.821 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:14.821 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.821 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.821 DEBUG:boto:Method: HEAD 2024-03-04T17:57:14.821 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.821 DEBUG:boto:Data: 2024-03-04T17:57:14.822 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.822 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:14.822 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.822 DEBUG:boto:Params: {} 2024-03-04T17:57:14.822 DEBUG:boto:Token: None 2024-03-04T17:57:14.822 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.822 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:b6Dbf8y0VKQAQpWcxVAy5e6VTCs= 2024-03-04T17:57:14.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:b6Dbf8y0VKQAQpWcxVAy5e6VTCs=', 'Content-Length': '0'} 2024-03-04T17:57:14.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000087a589e16b6a965d-0065e60b7a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.825 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:14.825 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:14.825 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.825 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.825 DEBUG:boto:Method: HEAD 2024-03-04T17:57:14.825 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.825 DEBUG:boto:Data: 2024-03-04T17:57:14.826 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.826 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:14.826 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.826 DEBUG:boto:Params: {} 2024-03-04T17:57:14.826 DEBUG:boto:Token: None 2024-03-04T17:57:14.826 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.826 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:b6Dbf8y0VKQAQpWcxVAy5e6VTCs= 2024-03-04T17:57:14.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:b6Dbf8y0VKQAQpWcxVAy5e6VTCs=', 'Content-Length': '0'} 2024-03-04T17:57:14.829 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000096a18c143563617a-0065e60b7a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:14.829 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:14.830 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:14.830 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.830 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.830 DEBUG:boto:Method: GET 2024-03-04T17:57:14.830 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.830 DEBUG:boto:Data: 2024-03-04T17:57:14.830 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.830 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:14.830 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.830 DEBUG:boto:Params: {} 2024-03-04T17:57:14.831 DEBUG:boto:Token: None 2024-03-04T17:57:14.831 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.831 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OaTpCgHmz3wKdvfv0Vi/EOfBpwM= 2024-03-04T17:57:14.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OaTpCgHmz3wKdvfv0Vi/EOfBpwM=', 'Content-Length': '0'} 2024-03-04T17:57:14.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f510db1a7701b2ab-0065e60b7a-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.834 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:14.834 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:14.834 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.834 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.834 DEBUG:boto:Method: GET 2024-03-04T17:57:14.834 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.834 DEBUG:boto:Data: 2024-03-04T17:57:14.835 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.835 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:14.835 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.835 DEBUG:boto:Params: {} 2024-03-04T17:57:14.835 DEBUG:boto:Token: None 2024-03-04T17:57:14.835 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:14.835 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:OaTpCgHmz3wKdvfv0Vi/EOfBpwM= 2024-03-04T17:57:14.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OaTpCgHmz3wKdvfv0Vi/EOfBpwM=', 'Content-Length': '0'} 2024-03-04T17:57:14.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002812b48827ca73b6-0065e60b7a-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.838 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.838 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.839 DEBUG:boto:path=/ppfahn-26/?prefix=obj 2024-03-04T17:57:14.839 DEBUG:boto:auth_path=/ppfahn-26/?prefix=obj 2024-03-04T17:57:14.839 DEBUG:boto:Method: GET 2024-03-04T17:57:14.839 DEBUG:boto:Path: /ppfahn-26/?prefix=obj 2024-03-04T17:57:14.839 DEBUG:boto:Data: 2024-03-04T17:57:14.839 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.839 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:14.839 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.839 DEBUG:boto:Params: {} 2024-03-04T17:57:14.839 DEBUG:boto:Token: None 2024-03-04T17:57:14.839 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/ 2024-03-04T17:57:14.840 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LwXZHn3BjUKbLj+yhOfF11OkeQk= 2024-03-04T17:57:14.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LwXZHn3BjUKbLj+yhOfF11OkeQk=', 'Content-Length': '0'} 2024-03-04T17:57:14.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b21e26daea9c1032-0065e60b7a-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.884 DEBUG:boto:b'ppfahn-26obj1000false' 2024-03-04T17:57:14.884 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.884 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.884 DEBUG:boto:path=/ppfahn-26/?prefix=obj 2024-03-04T17:57:14.884 DEBUG:boto:auth_path=/ppfahn-26/?prefix=obj 2024-03-04T17:57:14.885 DEBUG:boto:Method: GET 2024-03-04T17:57:14.885 DEBUG:boto:Path: /ppfahn-26/?prefix=obj 2024-03-04T17:57:14.885 DEBUG:boto:Data: 2024-03-04T17:57:14.885 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.885 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:14.885 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.885 DEBUG:boto:Params: {} 2024-03-04T17:57:14.885 DEBUG:boto:Token: None 2024-03-04T17:57:14.885 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/ 2024-03-04T17:57:14.885 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LwXZHn3BjUKbLj+yhOfF11OkeQk= 2024-03-04T17:57:14.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LwXZHn3BjUKbLj+yhOfF11OkeQk=', 'Content-Length': '0'} 2024-03-04T17:57:14.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000982b87dec31ecb17-0065e60b7a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.890 DEBUG:boto:b'ppfahn-26obj1000false' 2024-03-04T17:57:14.890 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:14.890 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:14.890 DEBUG:boto:Method: HEAD 2024-03-04T17:57:14.890 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:57:14.890 DEBUG:boto:Data: 2024-03-04T17:57:14.890 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.890 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:14.891 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.891 DEBUG:boto:Params: {} 2024-03-04T17:57:14.891 DEBUG:boto:Token: None 2024-03-04T17:57:14.891 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/obj 2024-03-04T17:57:14.891 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rhhpCiT4ZPfECF+HmANIo7aa/+Q= 2024-03-04T17:57:14.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rhhpCiT4ZPfECF+HmANIo7aa/+Q=', 'Content-Length': '0'} 2024-03-04T17:57:14.895 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000003dd91b00f9941406-0065e60b7a-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.895 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:14.895 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:14.895 DEBUG:boto:Method: HEAD 2024-03-04T17:57:14.895 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:57:14.895 DEBUG:boto:Data: 2024-03-04T17:57:14.895 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.895 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:14.895 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.896 DEBUG:boto:Params: {} 2024-03-04T17:57:14.896 DEBUG:boto:Token: None 2024-03-04T17:57:14.896 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/obj 2024-03-04T17:57:14.896 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:rhhpCiT4ZPfECF+HmANIo7aa/+Q= 2024-03-04T17:57:14.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rhhpCiT4ZPfECF+HmANIo7aa/+Q=', 'Content-Length': '0'} 2024-03-04T17:57:14.899 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c74c5467ab44b955-0065e60b7a-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.899 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-26 zones={test-zone2, test-zone1} 2024-03-04T17:57:14.900 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-26 zones={test-zone3, test-zone1} 2024-03-04T17:57:14.900 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.900 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.900 DEBUG:boto:Method: HEAD 2024-03-04T17:57:14.900 DEBUG:boto:Path: /ppfahn-26/ 2024-03-04T17:57:14.900 DEBUG:boto:Data: 2024-03-04T17:57:14.900 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.900 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:14.900 DEBUG:boto:Port: 8001 2024-03-04T17:57:14.900 DEBUG:boto:Params: {} 2024-03-04T17:57:14.901 DEBUG:boto:Token: None 2024-03-04T17:57:14.901 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/ 2024-03-04T17:57:14.901 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hq3jCG9y8UjnIsPC/q56g5+gNp0= 2024-03-04T17:57:14.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hq3jCG9y8UjnIsPC/q56g5+gNp0=', 'Content-Length': '0'} 2024-03-04T17:57:14.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000186458966656f9ac-0065e60b7a-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.905 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.905 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.906 DEBUG:boto:Method: HEAD 2024-03-04T17:57:14.906 DEBUG:boto:Path: /ppfahn-26/ 2024-03-04T17:57:14.906 DEBUG:boto:Data: 2024-03-04T17:57:14.906 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.906 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:14.906 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.906 DEBUG:boto:Params: {} 2024-03-04T17:57:14.906 DEBUG:boto:Token: None 2024-03-04T17:57:14.906 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/ 2024-03-04T17:57:14.906 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hq3jCG9y8UjnIsPC/q56g5+gNp0= 2024-03-04T17:57:14.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hq3jCG9y8UjnIsPC/q56g5+gNp0=', 'Content-Length': '0'} 2024-03-04T17:57:14.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008716021a4f723f9e-0065e60b7a-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.911 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:14.911 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.911 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.911 DEBUG:boto:path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.911 DEBUG:boto:auth_path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.911 DEBUG:boto:Method: GET 2024-03-04T17:57:14.911 DEBUG:boto:Path: /ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.911 DEBUG:boto:Data: 2024-03-04T17:57:14.912 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.912 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:14.912 DEBUG:boto:Port: 8001 2024-03-04T17:57:14.912 DEBUG:boto:Params: {} 2024-03-04T17:57:14.912 DEBUG:boto:Token: None 2024-03-04T17:57:14.912 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/?versions 2024-03-04T17:57:14.912 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8= 2024-03-04T17:57:14.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8=', 'Content-Length': '0'} 2024-03-04T17:57:14.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000561f55a4ede38fa1-0065e60b7a-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.917 DEBUG:boto:b'ppfahn-26999falseobj4qnz-Ulcde6cJDXxKloMY8QwW.xzlq2true2024-03-04T17:56:36.015Zrgw-multisite-test-userMultisite Test UserNormalobj0WamgmIvOh.4g-Qrzmkk5TwhQskoikPfalse2024-03-04T17:56:35.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:14.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:14.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:14.918 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:14.918 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.918 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.918 DEBUG:boto:path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.918 DEBUG:boto:auth_path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.918 DEBUG:boto:Method: GET 2024-03-04T17:57:14.918 DEBUG:boto:Path: /ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.918 DEBUG:boto:Data: 2024-03-04T17:57:14.918 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.918 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:14.919 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.919 DEBUG:boto:Params: {} 2024-03-04T17:57:14.919 DEBUG:boto:Token: None 2024-03-04T17:57:14.919 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/?versions 2024-03-04T17:57:14.919 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8= 2024-03-04T17:57:14.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8=', 'Content-Length': '0'} 2024-03-04T17:57:14.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000116c8a0293ff7fbe-0065e60b7a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.923 DEBUG:boto:b'ppfahn-26999falseobj4qnz-Ulcde6cJDXxKloMY8QwW.xzlq2true2024-03-04T17:56:36.015Zrgw-multisite-test-userMultisite Test UserNormalobj0WamgmIvOh.4g-Qrzmkk5TwhQskoikPfalse2024-03-04T17:56:35.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:14.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:14.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:14.923 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.924 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.924 DEBUG:boto:path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.924 DEBUG:boto:auth_path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.924 DEBUG:boto:Method: GET 2024-03-04T17:57:14.924 DEBUG:boto:Path: /ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.924 DEBUG:boto:Data: 2024-03-04T17:57:14.924 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.924 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:14.924 DEBUG:boto:Port: 8001 2024-03-04T17:57:14.925 DEBUG:boto:Params: {} 2024-03-04T17:57:14.925 DEBUG:boto:Token: None 2024-03-04T17:57:14.925 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/?versions 2024-03-04T17:57:14.925 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8= 2024-03-04T17:57:14.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8=', 'Content-Length': '0'} 2024-03-04T17:57:14.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e59485245d13a607-0065e60b7a-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:14.972 DEBUG:boto:b'ppfahn-26999falseobj4qnz-Ulcde6cJDXxKloMY8QwW.xzlq2true2024-03-04T17:56:36.015Zrgw-multisite-test-userMultisite Test UserNormalobj0WamgmIvOh.4g-Qrzmkk5TwhQskoikPfalse2024-03-04T17:56:35.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:14.972 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:14.973 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:14.973 DEBUG:boto:path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.973 DEBUG:boto:auth_path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.973 DEBUG:boto:Method: GET 2024-03-04T17:57:14.974 DEBUG:boto:Path: /ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:14.974 DEBUG:boto:Data: 2024-03-04T17:57:14.974 DEBUG:boto:Headers: {} 2024-03-04T17:57:14.974 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:14.974 DEBUG:boto:Port: 8000 2024-03-04T17:57:14.974 DEBUG:boto:Params: {} 2024-03-04T17:57:14.974 DEBUG:boto:Token: None 2024-03-04T17:57:14.975 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:14 GMT /ppfahn-26/?versions 2024-03-04T17:57:14.975 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8= 2024-03-04T17:57:14.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sTW6T8pL2lnVwozrkAWxelUa/J8=', 'Content-Length': '0'} 2024-03-04T17:57:14.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000508d1221790fbb67-0065e60b7a-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.020 DEBUG:boto:b'ppfahn-26999falseobj4qnz-Ulcde6cJDXxKloMY8QwW.xzlq2true2024-03-04T17:56:36.015Zrgw-multisite-test-userMultisite Test UserNormalobj0WamgmIvOh.4g-Qrzmkk5TwhQskoikPfalse2024-03-04T17:56:35.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:15.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:15.020 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.020 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.021 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.021 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:57:15.021 DEBUG:boto:Data: 2024-03-04T17:57:15.021 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.021 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.021 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.021 DEBUG:boto:Params: {} 2024-03-04T17:57:15.021 DEBUG:boto:Token: None 2024-03-04T17:57:15.022 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj 2024-03-04T17:57:15.022 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw= 2024-03-04T17:57:15.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw=', 'Content-Length': '0'} 2024-03-04T17:57:15.026 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000016fc39a7b669fbca-0065e60b7b-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.026 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.026 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.026 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.026 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:57:15.026 DEBUG:boto:Data: 2024-03-04T17:57:15.026 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.026 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:15.027 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.027 DEBUG:boto:Params: {} 2024-03-04T17:57:15.027 DEBUG:boto:Token: None 2024-03-04T17:57:15.027 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj 2024-03-04T17:57:15.027 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw= 2024-03-04T17:57:15.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw=', 'Content-Length': '0'} 2024-03-04T17:57:15.030 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a80c9170021113f0-0065e60b7b-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:15.031 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.031 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.031 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.031 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.031 DEBUG:boto:Method: GET 2024-03-04T17:57:15.031 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.031 DEBUG:boto:Data: 2024-03-04T17:57:15.032 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.032 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.032 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.032 DEBUG:boto:Params: {} 2024-03-04T17:57:15.032 DEBUG:boto:Token: None 2024-03-04T17:57:15.032 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.032 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI= 2024-03-04T17:57:15.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI=', 'Content-Length': '0'} 2024-03-04T17:57:15.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca893a2b447231aa-0065e60b7b-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.035 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.035 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.035 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.035 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.036 DEBUG:boto:Method: GET 2024-03-04T17:57:15.036 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.036 DEBUG:boto:Data: 2024-03-04T17:57:15.036 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.036 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:15.036 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.036 DEBUG:boto:Params: {} 2024-03-04T17:57:15.036 DEBUG:boto:Token: None 2024-03-04T17:57:15.036 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.036 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI= 2024-03-04T17:57:15.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI=', 'Content-Length': '0'} 2024-03-04T17:57:15.039 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003842826acbf730ef-0065e60b7b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.039 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.040 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.040 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.040 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.040 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.040 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.040 DEBUG:boto:Data: 2024-03-04T17:57:15.040 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.040 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.040 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.040 DEBUG:boto:Params: {} 2024-03-04T17:57:15.041 DEBUG:boto:Token: None 2024-03-04T17:57:15.041 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.041 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CkaV4sUQymmZssrEBQEaNv74gHA= 2024-03-04T17:57:15.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CkaV4sUQymmZssrEBQEaNv74gHA=', 'Content-Length': '0'} 2024-03-04T17:57:15.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eae8e80696122878-0065e60b7b-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.044 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.044 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.044 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.044 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.044 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.044 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.045 DEBUG:boto:Data: 2024-03-04T17:57:15.045 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.045 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:15.045 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.045 DEBUG:boto:Params: {} 2024-03-04T17:57:15.045 DEBUG:boto:Token: None 2024-03-04T17:57:15.045 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.045 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CkaV4sUQymmZssrEBQEaNv74gHA= 2024-03-04T17:57:15.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CkaV4sUQymmZssrEBQEaNv74gHA=', 'Content-Length': '0'} 2024-03-04T17:57:15.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7572a45d9d7779b-0065e60b7b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:15.048 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.048 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.048 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.049 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.049 DEBUG:boto:Method: GET 2024-03-04T17:57:15.049 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.049 DEBUG:boto:Data: 2024-03-04T17:57:15.049 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.049 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.049 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.049 DEBUG:boto:Params: {} 2024-03-04T17:57:15.049 DEBUG:boto:Token: None 2024-03-04T17:57:15.049 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.049 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI= 2024-03-04T17:57:15.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI=', 'Content-Length': '0'} 2024-03-04T17:57:15.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000715a89c318ac57e1-0065e60b7b-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.052 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.052 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.053 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.053 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.053 DEBUG:boto:Method: GET 2024-03-04T17:57:15.053 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.053 DEBUG:boto:Data: 2024-03-04T17:57:15.053 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.053 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:15.053 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.053 DEBUG:boto:Params: {} 2024-03-04T17:57:15.053 DEBUG:boto:Token: None 2024-03-04T17:57:15.053 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.053 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI= 2024-03-04T17:57:15.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI=', 'Content-Length': '0'} 2024-03-04T17:57:15.056 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('x-rgw-replicated-from', '185c722f-6c40-48e4-8855-bf605b5498c7:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000edafbe46232e982c-0065e60b7b-4349-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.056 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:15.057 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:15.057 DEBUG:boto:path=/ppfahn-26/?prefix=obj 2024-03-04T17:57:15.057 DEBUG:boto:auth_path=/ppfahn-26/?prefix=obj 2024-03-04T17:57:15.057 DEBUG:boto:Method: GET 2024-03-04T17:57:15.057 DEBUG:boto:Path: /ppfahn-26/?prefix=obj 2024-03-04T17:57:15.057 DEBUG:boto:Data: 2024-03-04T17:57:15.057 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.057 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.057 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.057 DEBUG:boto:Params: {} 2024-03-04T17:57:15.057 DEBUG:boto:Token: None 2024-03-04T17:57:15.057 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/ 2024-03-04T17:57:15.058 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3KD45qhn4VyM+ela5TsVjva4aUs= 2024-03-04T17:57:15.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3KD45qhn4VyM+ela5TsVjva4aUs=', 'Content-Length': '0'} 2024-03-04T17:57:15.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007686cb55bd058aae-0065e60b7b-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.062 DEBUG:boto:b'ppfahn-26obj1000false' 2024-03-04T17:57:15.062 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:15.062 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:15.062 DEBUG:boto:path=/ppfahn-26/?prefix=obj 2024-03-04T17:57:15.063 DEBUG:boto:auth_path=/ppfahn-26/?prefix=obj 2024-03-04T17:57:15.063 DEBUG:boto:Method: GET 2024-03-04T17:57:15.063 DEBUG:boto:Path: /ppfahn-26/?prefix=obj 2024-03-04T17:57:15.063 DEBUG:boto:Data: 2024-03-04T17:57:15.063 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.063 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:15.063 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.063 DEBUG:boto:Params: {} 2024-03-04T17:57:15.063 DEBUG:boto:Token: None 2024-03-04T17:57:15.063 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/ 2024-03-04T17:57:15.063 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3KD45qhn4VyM+ela5TsVjva4aUs= 2024-03-04T17:57:15.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3KD45qhn4VyM+ela5TsVjva4aUs=', 'Content-Length': '0'} 2024-03-04T17:57:15.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000922dd40fb4c82cf-0065e60b7b-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.108 DEBUG:boto:b'ppfahn-26obj1000false' 2024-03-04T17:57:15.108 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.108 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.108 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.109 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:57:15.109 DEBUG:boto:Data: 2024-03-04T17:57:15.109 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.109 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.109 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.109 DEBUG:boto:Params: {} 2024-03-04T17:57:15.109 DEBUG:boto:Token: None 2024-03-04T17:57:15.109 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj 2024-03-04T17:57:15.109 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw= 2024-03-04T17:57:15.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw=', 'Content-Length': '0'} 2024-03-04T17:57:15.113 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000008f61cbd67d960e1e-0065e60b7b-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.113 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.113 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.113 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.114 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:57:15.114 DEBUG:boto:Data: 2024-03-04T17:57:15.114 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.114 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:15.114 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.114 DEBUG:boto:Params: {} 2024-03-04T17:57:15.114 DEBUG:boto:Token: None 2024-03-04T17:57:15.114 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj 2024-03-04T17:57:15.114 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw= 2024-03-04T17:57:15.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw=', 'Content-Length': '0'} 2024-03-04T17:57:15.118 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d6edb3efc3cd4053-0065e60b7b-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.118 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-26 zones={test-zone3, test-zone1} 2024-03-04T17:57:15.118 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-26 zones={test-zone3, test-zone2} 2024-03-04T17:57:15.118 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:15.118 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:15.118 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.118 DEBUG:boto:Path: /ppfahn-26/ 2024-03-04T17:57:15.118 DEBUG:boto:Data: 2024-03-04T17:57:15.118 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.118 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.119 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.119 DEBUG:boto:Params: {} 2024-03-04T17:57:15.119 DEBUG:boto:Token: None 2024-03-04T17:57:15.119 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/ 2024-03-04T17:57:15.119 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gTSw6abWJTmBCi+oEiF2+nyLusk= 2024-03-04T17:57:15.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gTSw6abWJTmBCi+oEiF2+nyLusk=', 'Content-Length': '0'} 2024-03-04T17:57:15.123 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0a4251ace72f14e-0065e60b7b-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.123 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:15.123 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:15.123 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.123 DEBUG:boto:Path: /ppfahn-26/ 2024-03-04T17:57:15.123 DEBUG:boto:Data: 2024-03-04T17:57:15.123 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.123 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:15.123 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.123 DEBUG:boto:Params: {} 2024-03-04T17:57:15.124 DEBUG:boto:Token: None 2024-03-04T17:57:15.124 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/ 2024-03-04T17:57:15.124 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gTSw6abWJTmBCi+oEiF2+nyLusk= 2024-03-04T17:57:15.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gTSw6abWJTmBCi+oEiF2+nyLusk=', 'Content-Length': '0'} 2024-03-04T17:57:15.128 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f9fbe180273ef15-0065e60b7b-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.128 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:15.128 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:15.128 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:15.128 DEBUG:boto:path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:15.128 DEBUG:boto:auth_path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:15.129 DEBUG:boto:Method: GET 2024-03-04T17:57:15.129 DEBUG:boto:Path: /ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:15.129 DEBUG:boto:Data: 2024-03-04T17:57:15.129 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.129 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.129 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.129 DEBUG:boto:Params: {} 2024-03-04T17:57:15.129 DEBUG:boto:Token: None 2024-03-04T17:57:15.129 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/?versions 2024-03-04T17:57:15.129 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+UvzJGgRoy0sDKkduIzgkyEU+vM= 2024-03-04T17:57:15.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+UvzJGgRoy0sDKkduIzgkyEU+vM=', 'Content-Length': '0'} 2024-03-04T17:57:15.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf77b2f4992feb68-0065e60b7b-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.176 DEBUG:boto:b'ppfahn-26999falseobj4qnz-Ulcde6cJDXxKloMY8QwW.xzlq2true2024-03-04T17:56:36.015Zrgw-multisite-test-userMultisite Test UserNormalobj0WamgmIvOh.4g-Qrzmkk5TwhQskoikPfalse2024-03-04T17:56:35.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:15.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:15.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:15.176 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:15.177 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:15.177 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:15.177 DEBUG:boto:path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:15.177 DEBUG:boto:auth_path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:15.177 DEBUG:boto:Method: GET 2024-03-04T17:57:15.177 DEBUG:boto:Path: /ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:15.177 DEBUG:boto:Data: 2024-03-04T17:57:15.177 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.177 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:15.177 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.177 DEBUG:boto:Params: {} 2024-03-04T17:57:15.178 DEBUG:boto:Token: None 2024-03-04T17:57:15.178 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/?versions 2024-03-04T17:57:15.178 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+UvzJGgRoy0sDKkduIzgkyEU+vM= 2024-03-04T17:57:15.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+UvzJGgRoy0sDKkduIzgkyEU+vM=', 'Content-Length': '0'} 2024-03-04T17:57:15.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006241ee4ea402bb38-0065e60b7b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.182 DEBUG:boto:b'ppfahn-26999falseobj4qnz-Ulcde6cJDXxKloMY8QwW.xzlq2true2024-03-04T17:56:36.015Zrgw-multisite-test-userMultisite Test UserNormalobj0WamgmIvOh.4g-Qrzmkk5TwhQskoikPfalse2024-03-04T17:56:35.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:15.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:15.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:15.183 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:15.183 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:15.183 DEBUG:boto:path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:15.183 DEBUG:boto:auth_path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:15.183 DEBUG:boto:Method: GET 2024-03-04T17:57:15.183 DEBUG:boto:Path: /ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:15.183 DEBUG:boto:Data: 2024-03-04T17:57:15.183 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.184 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.184 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.184 DEBUG:boto:Params: {} 2024-03-04T17:57:15.184 DEBUG:boto:Token: None 2024-03-04T17:57:15.184 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/?versions 2024-03-04T17:57:15.184 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+UvzJGgRoy0sDKkduIzgkyEU+vM= 2024-03-04T17:57:15.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+UvzJGgRoy0sDKkduIzgkyEU+vM=', 'Content-Length': '0'} 2024-03-04T17:57:15.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ff05d987b61e30e-0065e60b7b-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.189 DEBUG:boto:b'ppfahn-26999falseobj4qnz-Ulcde6cJDXxKloMY8QwW.xzlq2true2024-03-04T17:56:36.015Zrgw-multisite-test-userMultisite Test UserNormalobj0WamgmIvOh.4g-Qrzmkk5TwhQskoikPfalse2024-03-04T17:56:35.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:15.190 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:15.190 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:15.190 DEBUG:boto:path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:15.190 DEBUG:boto:auth_path=/ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:15.190 DEBUG:boto:Method: GET 2024-03-04T17:57:15.190 DEBUG:boto:Path: /ppfahn-26/?versions&max-keys=999 2024-03-04T17:57:15.190 DEBUG:boto:Data: 2024-03-04T17:57:15.190 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.190 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:15.190 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.190 DEBUG:boto:Params: {} 2024-03-04T17:57:15.191 DEBUG:boto:Token: None 2024-03-04T17:57:15.191 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/?versions 2024-03-04T17:57:15.191 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+UvzJGgRoy0sDKkduIzgkyEU+vM= 2024-03-04T17:57:15.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+UvzJGgRoy0sDKkduIzgkyEU+vM=', 'Content-Length': '0'} 2024-03-04T17:57:15.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d86f736e984c57ef-0065e60b7b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.195 DEBUG:boto:b'ppfahn-26999falseobj4qnz-Ulcde6cJDXxKloMY8QwW.xzlq2true2024-03-04T17:56:36.015Zrgw-multisite-test-userMultisite Test UserNormalobj0WamgmIvOh.4g-Qrzmkk5TwhQskoikPfalse2024-03-04T17:56:35.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:15.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:15.195 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.196 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.196 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.196 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:57:15.196 DEBUG:boto:Data: 2024-03-04T17:57:15.196 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.196 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.196 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.196 DEBUG:boto:Params: {} 2024-03-04T17:57:15.196 DEBUG:boto:Token: None 2024-03-04T17:57:15.197 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj 2024-03-04T17:57:15.197 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw= 2024-03-04T17:57:15.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw=', 'Content-Length': '0'} 2024-03-04T17:57:15.200 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000aad3326773da44d4-0065e60b7b-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.201 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.201 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.201 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.201 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:57:15.201 DEBUG:boto:Data: 2024-03-04T17:57:15.201 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.201 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:15.201 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.201 DEBUG:boto:Params: {} 2024-03-04T17:57:15.201 DEBUG:boto:Token: None 2024-03-04T17:57:15.201 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj 2024-03-04T17:57:15.202 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw= 2024-03-04T17:57:15.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw=', 'Content-Length': '0'} 2024-03-04T17:57:15.205 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000002c9559a3d00b866f-0065e60b7b-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:15.205 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.206 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.206 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.206 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.206 DEBUG:boto:Method: GET 2024-03-04T17:57:15.206 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.206 DEBUG:boto:Data: 2024-03-04T17:57:15.206 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.206 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.206 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.206 DEBUG:boto:Params: {} 2024-03-04T17:57:15.206 DEBUG:boto:Token: None 2024-03-04T17:57:15.206 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.207 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI= 2024-03-04T17:57:15.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI=', 'Content-Length': '0'} 2024-03-04T17:57:15.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000275d633b2ac68bba-0065e60b7b-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.210 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.210 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.210 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.210 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.210 DEBUG:boto:Method: GET 2024-03-04T17:57:15.210 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.210 DEBUG:boto:Data: 2024-03-04T17:57:15.210 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.210 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:15.211 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.211 DEBUG:boto:Params: {} 2024-03-04T17:57:15.211 DEBUG:boto:Token: None 2024-03-04T17:57:15.211 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.211 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI= 2024-03-04T17:57:15.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI=', 'Content-Length': '0'} 2024-03-04T17:57:15.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057f73f2872cb5c08-0065e60b7b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.214 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.214 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.214 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.215 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.215 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.215 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.215 DEBUG:boto:Data: 2024-03-04T17:57:15.215 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.215 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.215 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.215 DEBUG:boto:Params: {} 2024-03-04T17:57:15.215 DEBUG:boto:Token: None 2024-03-04T17:57:15.215 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.215 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CkaV4sUQymmZssrEBQEaNv74gHA= 2024-03-04T17:57:15.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CkaV4sUQymmZssrEBQEaNv74gHA=', 'Content-Length': '0'} 2024-03-04T17:57:15.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f6f2f61ab62ec43f-0065e60b7b-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.219 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.219 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.219 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.219 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.219 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.219 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.219 DEBUG:boto:Data: 2024-03-04T17:57:15.219 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.219 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:15.219 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.220 DEBUG:boto:Params: {} 2024-03-04T17:57:15.220 DEBUG:boto:Token: None 2024-03-04T17:57:15.220 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.220 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CkaV4sUQymmZssrEBQEaNv74gHA= 2024-03-04T17:57:15.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CkaV4sUQymmZssrEBQEaNv74gHA=', 'Content-Length': '0'} 2024-03-04T17:57:15.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7ece5cba4153ef1-0065e60b7b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:15.223 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.224 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.224 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.224 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.224 DEBUG:boto:Method: GET 2024-03-04T17:57:15.224 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.224 DEBUG:boto:Data: 2024-03-04T17:57:15.224 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.224 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.224 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.224 DEBUG:boto:Params: {} 2024-03-04T17:57:15.224 DEBUG:boto:Token: None 2024-03-04T17:57:15.224 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.225 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI= 2024-03-04T17:57:15.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI=', 'Content-Length': '0'} 2024-03-04T17:57:15.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000217d3722c1693d1e-0065e60b7b-4356-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.228 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.228 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.228 DEBUG:boto:path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.228 DEBUG:boto:auth_path=/ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.228 DEBUG:boto:Method: GET 2024-03-04T17:57:15.228 DEBUG:boto:Path: /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.228 DEBUG:boto:Data: 2024-03-04T17:57:15.228 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.228 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:15.229 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.229 DEBUG:boto:Params: {} 2024-03-04T17:57:15.229 DEBUG:boto:Token: None 2024-03-04T17:57:15.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj?versionId=0WamgmIvOh.4g-Qrzmkk5TwhQskoikP 2024-03-04T17:57:15.229 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI= 2024-03-04T17:57:15.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CnreFXuBnClEg4hOXsTz7R//IpI=', 'Content-Length': '0'} 2024-03-04T17:57:15.232 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 17:56:35 GMT'), ('x-amz-version-id', '0WamgmIvOh.4g-Qrzmkk5TwhQskoikP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c146b41a-1ed1-469f-afe6-2dddc3577925:ppfahn-26:d96636ac-054d-40e3-90f3-e3912da784b2.4349.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cef6f0d830a38b37-0065e60b7b-4306-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.232 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:15.232 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:15.232 DEBUG:boto:path=/ppfahn-26/?prefix=obj 2024-03-04T17:57:15.233 DEBUG:boto:auth_path=/ppfahn-26/?prefix=obj 2024-03-04T17:57:15.233 DEBUG:boto:Method: GET 2024-03-04T17:57:15.233 DEBUG:boto:Path: /ppfahn-26/?prefix=obj 2024-03-04T17:57:15.233 DEBUG:boto:Data: 2024-03-04T17:57:15.233 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.233 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.233 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.233 DEBUG:boto:Params: {} 2024-03-04T17:57:15.233 DEBUG:boto:Token: None 2024-03-04T17:57:15.233 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/ 2024-03-04T17:57:15.233 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3KD45qhn4VyM+ela5TsVjva4aUs= 2024-03-04T17:57:15.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3KD45qhn4VyM+ela5TsVjva4aUs=', 'Content-Length': '0'} 2024-03-04T17:57:15.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d93535c603e94c85-0065e60b7b-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.280 DEBUG:boto:b'ppfahn-26obj1000false' 2024-03-04T17:57:15.280 DEBUG:boto:path=/ppfahn-26/ 2024-03-04T17:57:15.280 DEBUG:boto:auth_path=/ppfahn-26/ 2024-03-04T17:57:15.280 DEBUG:boto:path=/ppfahn-26/?prefix=obj 2024-03-04T17:57:15.280 DEBUG:boto:auth_path=/ppfahn-26/?prefix=obj 2024-03-04T17:57:15.281 DEBUG:boto:Method: GET 2024-03-04T17:57:15.281 DEBUG:boto:Path: /ppfahn-26/?prefix=obj 2024-03-04T17:57:15.281 DEBUG:boto:Data: 2024-03-04T17:57:15.281 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.281 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:15.281 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.281 DEBUG:boto:Params: {} 2024-03-04T17:57:15.281 DEBUG:boto:Token: None 2024-03-04T17:57:15.281 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/ 2024-03-04T17:57:15.281 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3KD45qhn4VyM+ela5TsVjva4aUs= 2024-03-04T17:57:15.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3KD45qhn4VyM+ela5TsVjva4aUs=', 'Content-Length': '0'} 2024-03-04T17:57:15.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e4ae9a4173717c6-0065e60b7b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.286 DEBUG:boto:b'ppfahn-26obj1000false' 2024-03-04T17:57:15.286 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.286 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.286 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.286 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:57:15.286 DEBUG:boto:Data: 2024-03-04T17:57:15.286 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.286 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.286 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.287 DEBUG:boto:Params: {} 2024-03-04T17:57:15.287 DEBUG:boto:Token: None 2024-03-04T17:57:15.287 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj 2024-03-04T17:57:15.287 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw= 2024-03-04T17:57:15.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw=', 'Content-Length': '0'} 2024-03-04T17:57:15.291 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000000e5ae514a9818fb-0065e60b7b-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.291 DEBUG:boto:path=/ppfahn-26/obj 2024-03-04T17:57:15.291 DEBUG:boto:auth_path=/ppfahn-26/obj 2024-03-04T17:57:15.291 DEBUG:boto:Method: HEAD 2024-03-04T17:57:15.291 DEBUG:boto:Path: /ppfahn-26/obj 2024-03-04T17:57:15.291 DEBUG:boto:Data: 2024-03-04T17:57:15.291 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.292 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:15.292 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.292 DEBUG:boto:Params: {} 2024-03-04T17:57:15.292 DEBUG:boto:Token: None 2024-03-04T17:57:15.292 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-26/obj 2024-03-04T17:57:15.292 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw= 2024-03-04T17:57:15.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Vwkttywq34yI7fR2WDVqYBjgdAw=', 'Content-Length': '0'} 2024-03-04T17:57:15.296 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000008942fd4ef4fefe2c-0065e60b7b-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.296 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-26 zones={test-zone3, test-zone2} 2024-03-04T17:57:15.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2024-03-04T17:57:15.297 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-27 2024-03-04T17:57:15.297 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:15.297 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:15.297 DEBUG:boto:Method: PUT 2024-03-04T17:57:15.298 DEBUG:boto:Path: /ppfahn-27/ 2024-03-04T17:57:15.298 DEBUG:boto:Data: 2024-03-04T17:57:15.298 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.298 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:15.298 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.298 DEBUG:boto:Params: {} 2024-03-04T17:57:15.298 DEBUG:boto:Token: None 2024-03-04T17:57:15.298 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-27/ 2024-03-04T17:57:15.298 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:sNC9adfsFFReQuZ16QrlVNpk6uY= 2024-03-04T17:57:15.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:sNC9adfsFFReQuZ16QrlVNpk6uY=', 'Content-Length': '0'} 2024-03-04T17:57:15.338 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007a5a9facbde974c1-0065e60b7b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.338 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-28 2024-03-04T17:57:15.338 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:15.339 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:15.339 DEBUG:boto:Method: PUT 2024-03-04T17:57:15.339 DEBUG:boto:Path: /ppfahn-28/ 2024-03-04T17:57:15.339 DEBUG:boto:Data: 2024-03-04T17:57:15.339 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.339 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:15.339 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.339 DEBUG:boto:Params: {} 2024-03-04T17:57:15.339 DEBUG:boto:Token: None 2024-03-04T17:57:15.339 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-28/ 2024-03-04T17:57:15.339 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lpUXEyv2/mQCVMPrkJxONyfgdr0= 2024-03-04T17:57:15.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lpUXEyv2/mQCVMPrkJxONyfgdr0=', 'Content-Length': '0'} 2024-03-04T17:57:15.435 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c9673cb10be97d3d-0065e60b7b-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.435 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-29 2024-03-04T17:57:15.435 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:15.435 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:15.435 DEBUG:boto:Method: PUT 2024-03-04T17:57:15.435 DEBUG:boto:Path: /ppfahn-29/ 2024-03-04T17:57:15.435 DEBUG:boto:Data: 2024-03-04T17:57:15.435 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.435 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.436 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.436 DEBUG:boto:Params: {} 2024-03-04T17:57:15.436 DEBUG:boto:Token: None 2024-03-04T17:57:15.436 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-29/ 2024-03-04T17:57:15.436 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vzDJce8GiM5wrDMaIS94GLtqDqc= 2024-03-04T17:57:15.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vzDJce8GiM5wrDMaIS94GLtqDqc=', 'Content-Length': '0'} 2024-03-04T17:57:15.500 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d6975a537b074a1-0065e60b7b-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.500 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:15.500 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:15.501 DEBUG:boto:path=/ppfahn-27/?versioning 2024-03-04T17:57:15.501 DEBUG:boto:auth_path=/ppfahn-27/?versioning 2024-03-04T17:57:15.501 DEBUG:boto:Method: PUT 2024-03-04T17:57:15.501 DEBUG:boto:Path: /ppfahn-27/?versioning 2024-03-04T17:57:15.501 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:57:15.501 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.501 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:15.501 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.501 DEBUG:boto:Params: {} 2024-03-04T17:57:15.501 DEBUG:boto:Token: None 2024-03-04T17:57:15.502 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-27/?versioning 2024-03-04T17:57:15.502 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DENm/1Juf0ZeZss56hM654PDFcU= 2024-03-04T17:57:15.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DENm/1Juf0ZeZss56hM654PDFcU=', 'Content-Length': '228'} 2024-03-04T17:57:15.512 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005c4df14390feeeef-0065e60b7b-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.512 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:15.512 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:15.512 DEBUG:boto:path=/ppfahn-27/?versioning 2024-03-04T17:57:15.512 DEBUG:boto:auth_path=/ppfahn-27/?versioning 2024-03-04T17:57:15.513 DEBUG:boto:Method: PUT 2024-03-04T17:57:15.513 DEBUG:boto:Path: /ppfahn-27/?versioning 2024-03-04T17:57:15.513 DEBUG:boto:Data: Suspended Disabled 2024-03-04T17:57:15.513 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.513 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:15.513 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.513 DEBUG:boto:Params: {} 2024-03-04T17:57:15.513 DEBUG:boto:Token: None 2024-03-04T17:57:15.513 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-27/?versioning 2024-03-04T17:57:15.514 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DENm/1Juf0ZeZss56hM654PDFcU= 2024-03-04T17:57:15.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DENm/1Juf0ZeZss56hM654PDFcU=', 'Content-Length': '230'} 2024-03-04T17:57:15.523 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c070218a0196a5c5-0065e60b7b-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.523 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:15.524 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:15.524 DEBUG:boto:path=/ppfahn-28/?versioning 2024-03-04T17:57:15.524 DEBUG:boto:auth_path=/ppfahn-28/?versioning 2024-03-04T17:57:15.524 DEBUG:boto:Method: PUT 2024-03-04T17:57:15.524 DEBUG:boto:Path: /ppfahn-28/?versioning 2024-03-04T17:57:15.524 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:57:15.524 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.524 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:15.524 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.524 DEBUG:boto:Params: {} 2024-03-04T17:57:15.525 DEBUG:boto:Token: None 2024-03-04T17:57:15.525 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-28/?versioning 2024-03-04T17:57:15.525 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dZVfZTSxTkjl2NsoUrgJo3ddM8E= 2024-03-04T17:57:15.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dZVfZTSxTkjl2NsoUrgJo3ddM8E=', 'Content-Length': '228'} 2024-03-04T17:57:15.541 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ef3014eb3e9e1468-0065e60b7b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.542 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:15.542 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:15.542 DEBUG:boto:path=/ppfahn-28/?versioning 2024-03-04T17:57:15.542 DEBUG:boto:auth_path=/ppfahn-28/?versioning 2024-03-04T17:57:15.542 DEBUG:boto:Method: PUT 2024-03-04T17:57:15.542 DEBUG:boto:Path: /ppfahn-28/?versioning 2024-03-04T17:57:15.542 DEBUG:boto:Data: Suspended Disabled 2024-03-04T17:57:15.542 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.542 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:15.542 DEBUG:boto:Port: 8000 2024-03-04T17:57:15.542 DEBUG:boto:Params: {} 2024-03-04T17:57:15.542 DEBUG:boto:Token: None 2024-03-04T17:57:15.543 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-28/?versioning 2024-03-04T17:57:15.543 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dZVfZTSxTkjl2NsoUrgJo3ddM8E= 2024-03-04T17:57:15.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dZVfZTSxTkjl2NsoUrgJo3ddM8E=', 'Content-Length': '230'} 2024-03-04T17:57:15.571 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dd4a5163ff252661-0065e60b7b-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.571 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:15.571 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:15.571 DEBUG:boto:path=/ppfahn-29/?versioning 2024-03-04T17:57:15.571 DEBUG:boto:auth_path=/ppfahn-29/?versioning 2024-03-04T17:57:15.572 DEBUG:boto:Method: PUT 2024-03-04T17:57:15.572 DEBUG:boto:Path: /ppfahn-29/?versioning 2024-03-04T17:57:15.572 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:57:15.572 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.572 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.572 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.572 DEBUG:boto:Params: {} 2024-03-04T17:57:15.572 DEBUG:boto:Token: None 2024-03-04T17:57:15.572 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-29/?versioning 2024-03-04T17:57:15.572 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3CyJoUMhQPy+QBSYNm+9UWFWSlU= 2024-03-04T17:57:15.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3CyJoUMhQPy+QBSYNm+9UWFWSlU=', 'Content-Length': '228'} 2024-03-04T17:57:15.583 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000107d8a64021b7f78-0065e60b7b-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.583 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:15.583 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:15.583 DEBUG:boto:path=/ppfahn-29/?versioning 2024-03-04T17:57:15.583 DEBUG:boto:auth_path=/ppfahn-29/?versioning 2024-03-04T17:57:15.583 DEBUG:boto:Method: PUT 2024-03-04T17:57:15.583 DEBUG:boto:Path: /ppfahn-29/?versioning 2024-03-04T17:57:15.583 DEBUG:boto:Data: Suspended Disabled 2024-03-04T17:57:15.583 DEBUG:boto:Headers: {} 2024-03-04T17:57:15.584 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:15.584 DEBUG:boto:Port: 8001 2024-03-04T17:57:15.584 DEBUG:boto:Params: {} 2024-03-04T17:57:15.584 DEBUG:boto:Token: None 2024-03-04T17:57:15.584 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:15 GMT /ppfahn-29/?versioning 2024-03-04T17:57:15.584 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3CyJoUMhQPy+QBSYNm+9UWFWSlU= 2024-03-04T17:57:15.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3CyJoUMhQPy+QBSYNm+9UWFWSlU=', 'Content-Length': '230'} 2024-03-04T17:57:15.595 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d47748007469346-0065e60b7b-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:15.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:15.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:15.595 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:15.632 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:15.632 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:15.683 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:57:15.706 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709575035.596198_263.1","last_update":"2024-03-04T17:57:15.596198Z"}] 2024-03-04T17:57:15.706 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575035.596198_263.1'} 2024-03-04T17:57:15.706 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:57:15.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:15.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:15.707 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:15.749 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:15.749 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:15.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:15.824+0000 7f8c5e1c7980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:15.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:15.824+0000 7f8c5e1c7980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:15.846 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575035.520467_254.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:15.520467Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:15Z"} 2024-03-04T17:57:15.846 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575035.520467_254.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:15.520467Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:15Z"} 2024-03-04T17:57:15.846 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575035.520467_254.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:15.520467Z', 'realm_epoch': 2}}] 2024-03-04T17:57:15.846 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575035.596198_263.1'} 2024-03-04T17:57:15.846 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575035.520467_254.1'} 2024-03-04T17:57:15.847 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575035.596198_263.1 target=1_1709575035.520467_254.1 2024-03-04T17:57:20.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:20.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:20.848 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:20.892 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:20.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:20.948+0000 7f1944a04980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:20.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:20.952+0000 7f1944a04980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:20.972 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575035.596198_263.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:15.596198Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:20Z"} 2024-03-04T17:57:20.972 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575035.596198_263.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:15.596198Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:20Z"} 2024-03-04T17:57:20.972 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575035.596198_263.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:15.596198Z', 'realm_epoch': 2}}] 2024-03-04T17:57:20.972 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575035.596198_263.1'} 2024-03-04T17:57:20.973 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575035.596198_263.1'} 2024-03-04T17:57:20.973 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T17:57:20.973 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T17:57:20.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:20.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:20.973 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:21.027 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:21.027 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:21.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:21.089+0000 7f6a5ae7e980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:21.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:21.093+0000 7f6a5ae7e980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:21.115 INFO:teuthology.orchestra.run.smithi136.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575035.596198_263.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:15.596198Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:21Z"} 2024-03-04T17:57:21.115 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575035.596198_263.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:15.596198Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:21Z"} 2024-03-04T17:57:21.115 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575035.596198_263.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:15.596198Z', 'realm_epoch': 2}}] 2024-03-04T17:57:21.115 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575035.596198_263.1'} 2024-03-04T17:57:21.116 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575035.596198_263.1'} 2024-03-04T17:57:21.116 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T17:57:21.116 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:21.116 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:21.116 DEBUG:boto:Method: HEAD 2024-03-04T17:57:21.116 DEBUG:boto:Path: /ppfahn-27/ 2024-03-04T17:57:21.116 DEBUG:boto:Data: 2024-03-04T17:57:21.116 DEBUG:boto:Headers: {} 2024-03-04T17:57:21.116 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:21.116 DEBUG:boto:Port: 8000 2024-03-04T17:57:21.116 DEBUG:boto:Params: {} 2024-03-04T17:57:21.117 DEBUG:boto:Token: None 2024-03-04T17:57:21.117 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:21 GMT /ppfahn-27/ 2024-03-04T17:57:21.117 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GMr3jC2pB8UNZzL9Effb1B2m0MU= 2024-03-04T17:57:21.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GMr3jC2pB8UNZzL9Effb1B2m0MU=', 'Content-Length': '0'} 2024-03-04T17:57:21.119 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a6d836e8debe2f9-0065e60b81-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:21.120 DEBUG:boto:path=/ppfahn-27/obj 2024-03-04T17:57:21.120 DEBUG:boto:auth_path=/ppfahn-27/obj 2024-03-04T17:57:21.120 DEBUG:boto:Method: PUT 2024-03-04T17:57:21.120 DEBUG:boto:Path: /ppfahn-27/obj 2024-03-04T17:57:21.120 DEBUG:boto:Data: 2024-03-04T17:57:21.120 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T17:57:21.120 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:21.120 DEBUG:boto:Port: 8000 2024-03-04T17:57:21.120 DEBUG:boto:Params: {} 2024-03-04T17:57:21.120 DEBUG:boto:Token: None 2024-03-04T17:57:21.121 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 17:57:21 GMT /ppfahn-27/obj 2024-03-04T17:57:21.121 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Wy6pKsJzwfZZ+/j/O6tYbs9xF8E= 2024-03-04T17:57:21.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:57:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Wy6pKsJzwfZZ+/j/O6tYbs9xF8E='} 2024-03-04T17:57:21.127 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031170080f5e8ddf0-0065e60b81-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:21.127 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:21.127 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:21.127 DEBUG:boto:Method: HEAD 2024-03-04T17:57:21.127 DEBUG:boto:Path: /ppfahn-27/ 2024-03-04T17:57:21.127 DEBUG:boto:Data: 2024-03-04T17:57:21.128 DEBUG:boto:Headers: {} 2024-03-04T17:57:21.128 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:21.128 DEBUG:boto:Port: 8000 2024-03-04T17:57:21.128 DEBUG:boto:Params: {} 2024-03-04T17:57:21.128 DEBUG:boto:Token: None 2024-03-04T17:57:21.128 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:21 GMT /ppfahn-27/ 2024-03-04T17:57:21.128 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GMr3jC2pB8UNZzL9Effb1B2m0MU= 2024-03-04T17:57:21.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GMr3jC2pB8UNZzL9Effb1B2m0MU=', 'Content-Length': '0'} 2024-03-04T17:57:21.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3e9fb1c3d24f179-0065e60b81-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:21.131 DEBUG:boto:path=/ppfahn-27/obj 2024-03-04T17:57:21.131 DEBUG:boto:auth_path=/ppfahn-27/obj 2024-03-04T17:57:21.131 DEBUG:boto:Method: DELETE 2024-03-04T17:57:21.131 DEBUG:boto:Path: /ppfahn-27/obj 2024-03-04T17:57:21.131 DEBUG:boto:Data: 2024-03-04T17:57:21.131 DEBUG:boto:Headers: {} 2024-03-04T17:57:21.131 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:21.131 DEBUG:boto:Port: 8000 2024-03-04T17:57:21.131 DEBUG:boto:Params: {} 2024-03-04T17:57:21.131 DEBUG:boto:Token: None 2024-03-04T17:57:21.132 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:57:21 GMT /ppfahn-27/obj 2024-03-04T17:57:21.132 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IxW1btnPoPp7CpdCqjoV3VRTwxY= 2024-03-04T17:57:21.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IxW1btnPoPp7CpdCqjoV3VRTwxY=', 'Content-Length': '0'} 2024-03-04T17:57:21.141 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000ad388e750341cbb0-0065e60b81-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:21.141 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:21.141 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:21.142 DEBUG:boto:Method: HEAD 2024-03-04T17:57:21.142 DEBUG:boto:Path: /ppfahn-28/ 2024-03-04T17:57:21.142 DEBUG:boto:Data: 2024-03-04T17:57:21.142 DEBUG:boto:Headers: {} 2024-03-04T17:57:21.142 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:21.142 DEBUG:boto:Port: 8000 2024-03-04T17:57:21.142 DEBUG:boto:Params: {} 2024-03-04T17:57:21.142 DEBUG:boto:Token: None 2024-03-04T17:57:21.142 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:21 GMT /ppfahn-28/ 2024-03-04T17:57:21.142 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MBXDTi7RowYEGhI1RjJfgag7SS4= 2024-03-04T17:57:21.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MBXDTi7RowYEGhI1RjJfgag7SS4=', 'Content-Length': '0'} 2024-03-04T17:57:21.147 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009531e2d4278071db-0065e60b81-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:21.147 DEBUG:boto:path=/ppfahn-28/obj 2024-03-04T17:57:21.147 DEBUG:boto:auth_path=/ppfahn-28/obj 2024-03-04T17:57:21.147 DEBUG:boto:Method: PUT 2024-03-04T17:57:21.147 DEBUG:boto:Path: /ppfahn-28/obj 2024-03-04T17:57:21.147 DEBUG:boto:Data: 2024-03-04T17:57:21.148 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T17:57:21.148 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:21.148 DEBUG:boto:Port: 8000 2024-03-04T17:57:21.148 DEBUG:boto:Params: {} 2024-03-04T17:57:21.148 DEBUG:boto:Token: None 2024-03-04T17:57:21.148 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 17:57:21 GMT /ppfahn-28/obj 2024-03-04T17:57:21.148 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vWEp2HzR1Yc4pvbulpTZg1iO3OM= 2024-03-04T17:57:21.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:57:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vWEp2HzR1Yc4pvbulpTZg1iO3OM='} 2024-03-04T17:57:21.192 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a3110a1cdb221c1-0065e60b81-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:21.192 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:21.192 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:21.192 DEBUG:boto:Method: HEAD 2024-03-04T17:57:21.192 DEBUG:boto:Path: /ppfahn-28/ 2024-03-04T17:57:21.193 DEBUG:boto:Data: 2024-03-04T17:57:21.193 DEBUG:boto:Headers: {} 2024-03-04T17:57:21.193 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:21.193 DEBUG:boto:Port: 8000 2024-03-04T17:57:21.193 DEBUG:boto:Params: {} 2024-03-04T17:57:21.193 DEBUG:boto:Token: None 2024-03-04T17:57:21.193 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:21 GMT /ppfahn-28/ 2024-03-04T17:57:21.193 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MBXDTi7RowYEGhI1RjJfgag7SS4= 2024-03-04T17:57:21.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MBXDTi7RowYEGhI1RjJfgag7SS4=', 'Content-Length': '0'} 2024-03-04T17:57:21.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019fc14748155e37a-0065e60b81-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:21.197 DEBUG:boto:path=/ppfahn-28/obj 2024-03-04T17:57:21.197 DEBUG:boto:auth_path=/ppfahn-28/obj 2024-03-04T17:57:21.197 DEBUG:boto:Method: DELETE 2024-03-04T17:57:21.197 DEBUG:boto:Path: /ppfahn-28/obj 2024-03-04T17:57:21.198 DEBUG:boto:Data: 2024-03-04T17:57:21.198 DEBUG:boto:Headers: {} 2024-03-04T17:57:21.198 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:21.198 DEBUG:boto:Port: 8000 2024-03-04T17:57:21.198 DEBUG:boto:Params: {} 2024-03-04T17:57:21.198 DEBUG:boto:Token: None 2024-03-04T17:57:21.198 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:57:21 GMT /ppfahn-28/obj 2024-03-04T17:57:21.198 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HzHB2dDJI7AOOXMpcIWcIC8qprc= 2024-03-04T17:57:21.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HzHB2dDJI7AOOXMpcIWcIC8qprc=', 'Content-Length': '0'} 2024-03-04T17:57:21.221 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000da7a661cc9a5cbf9-0065e60b81-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:21.221 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:21.221 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:21.221 DEBUG:boto:Method: HEAD 2024-03-04T17:57:21.221 DEBUG:boto:Path: /ppfahn-29/ 2024-03-04T17:57:21.222 DEBUG:boto:Data: 2024-03-04T17:57:21.222 DEBUG:boto:Headers: {} 2024-03-04T17:57:21.222 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:21.222 DEBUG:boto:Port: 8001 2024-03-04T17:57:21.222 DEBUG:boto:Params: {} 2024-03-04T17:57:21.222 DEBUG:boto:Token: None 2024-03-04T17:57:21.222 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:21 GMT /ppfahn-29/ 2024-03-04T17:57:21.222 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:d/i7hcj63BsehlXN+uD/3fPxeDg= 2024-03-04T17:57:21.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:d/i7hcj63BsehlXN+uD/3fPxeDg=', 'Content-Length': '0'} 2024-03-04T17:57:21.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071e766ad567033bb-0065e60b81-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:21.227 DEBUG:boto:path=/ppfahn-29/obj 2024-03-04T17:57:21.227 DEBUG:boto:auth_path=/ppfahn-29/obj 2024-03-04T17:57:21.227 DEBUG:boto:Method: PUT 2024-03-04T17:57:21.227 DEBUG:boto:Path: /ppfahn-29/obj 2024-03-04T17:57:21.228 DEBUG:boto:Data: 2024-03-04T17:57:21.228 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T17:57:21.228 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:21.228 DEBUG:boto:Port: 8001 2024-03-04T17:57:21.228 DEBUG:boto:Params: {} 2024-03-04T17:57:21.228 DEBUG:boto:Token: None 2024-03-04T17:57:21.228 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 17:57:21 GMT /ppfahn-29/obj 2024-03-04T17:57:21.228 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:UzwxbAawjc+h53KvmWwjcBoVJ1I= 2024-03-04T17:57:21.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:57:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UzwxbAawjc+h53KvmWwjcBoVJ1I='} 2024-03-04T17:57:21.272 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f23c3893d65fe163-0065e60b81-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:21.272 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:21.272 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:21.272 DEBUG:boto:Method: HEAD 2024-03-04T17:57:21.273 DEBUG:boto:Path: /ppfahn-29/ 2024-03-04T17:57:21.273 DEBUG:boto:Data: 2024-03-04T17:57:21.273 DEBUG:boto:Headers: {} 2024-03-04T17:57:21.273 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:21.273 DEBUG:boto:Port: 8001 2024-03-04T17:57:21.273 DEBUG:boto:Params: {} 2024-03-04T17:57:21.273 DEBUG:boto:Token: None 2024-03-04T17:57:21.273 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:21 GMT /ppfahn-29/ 2024-03-04T17:57:21.273 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:d/i7hcj63BsehlXN+uD/3fPxeDg= 2024-03-04T17:57:21.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:d/i7hcj63BsehlXN+uD/3fPxeDg=', 'Content-Length': '0'} 2024-03-04T17:57:21.278 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001062006de28cbc40-0065e60b81-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:21.278 DEBUG:boto:path=/ppfahn-29/obj 2024-03-04T17:57:21.278 DEBUG:boto:auth_path=/ppfahn-29/obj 2024-03-04T17:57:21.278 DEBUG:boto:Method: DELETE 2024-03-04T17:57:21.278 DEBUG:boto:Path: /ppfahn-29/obj 2024-03-04T17:57:21.278 DEBUG:boto:Data: 2024-03-04T17:57:21.278 DEBUG:boto:Headers: {} 2024-03-04T17:57:21.278 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:21.278 DEBUG:boto:Port: 8001 2024-03-04T17:57:21.278 DEBUG:boto:Params: {} 2024-03-04T17:57:21.279 DEBUG:boto:Token: None 2024-03-04T17:57:21.279 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 17:57:21 GMT /ppfahn-29/obj 2024-03-04T17:57:21.279 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8vJAWEw6v0cuMkCcOsedWXNIrbU= 2024-03-04T17:57:21.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:21 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8vJAWEw6v0cuMkCcOsedWXNIrbU=', 'Content-Length': '0'} 2024-03-04T17:57:21.302 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000f62c510ed64de9fb-0065e60b81-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:21.302 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-27 2024-03-04T17:57:21.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:21.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:21.303 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-27 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:21.353 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:21.353 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:21.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:21.416+0000 7f2e43fc7980 1 waiting to reach incremental sync.. 2024-03-04T17:57:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:26.420+0000 7f2e43fc7980 1 waiting to reach incremental sync.. 2024-03-04T17:57:31.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:31.419+0000 7f2e43fc7980 1 waiting to reach incremental sync.. 2024-03-04T17:57:35.367 INFO:tasks.rgw.c2.client.0.smithi153.stdout:2024-03-04T17:57:35.367+0000 7fa53962a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:57:36.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:36.423+0000 7f2e43fc7980 1 bucket sync caught up with source: 2024-03-04T17:57:36.422 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000003.32.7, , ] 2024-03-04T17:57:36.422 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000003.32.7, , ] 2024-03-04T17:57:36.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:36.423+0000 7f2e43fc7980 0 bucket checkpoint complete 2024-03-04T17:57:36.460 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-27 2024-03-04T17:57:36.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:36.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:36.460 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-27 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:36.511 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:36.511 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:36.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:36.573+0000 7fe13c074980 1 bucket sync caught up with source: 2024-03-04T17:57:36.572 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000003.32.7, , ] 2024-03-04T17:57:36.572 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.32.7, , ] 2024-03-04T17:57:36.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:36.573+0000 7fe13c074980 0 bucket checkpoint complete 2024-03-04T17:57:36.595 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-27 2024-03-04T17:57:36.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:36.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:36.595 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-27 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:36.636 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:36.636 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:36.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:36.701+0000 7fcbb7528980 1 waiting to reach incremental sync.. 2024-03-04T17:57:41.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:41.700+0000 7fcbb7528980 1 waiting to reach incremental sync.. 2024-03-04T17:57:46.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:46.704+0000 7fcbb7528980 1 bucket sync caught up with source: 2024-03-04T17:57:46.703 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.41.14, , ] 2024-03-04T17:57:46.704 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000001.41.14, , ] 2024-03-04T17:57:46.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:46.704+0000 7fcbb7528980 0 bucket checkpoint complete 2024-03-04T17:57:46.749 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-27 2024-03-04T17:57:46.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:46.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:46.749 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-27 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:46.840 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:46.840 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:46.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:46.908+0000 7f31a10c0980 1 waiting to reach incremental sync.. 2024-03-04T17:57:51.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:51.908+0000 7f31a10c0980 1 bucket sync caught up with source: 2024-03-04T17:57:51.908 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.41.14, , ] 2024-03-04T17:57:51.908 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000001.41.14, , ] 2024-03-04T17:57:51.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:51.908+0000 7f31a10c0980 0 bucket checkpoint complete 2024-03-04T17:57:51.955 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-27 2024-03-04T17:57:51.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:51.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:51.955 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-27 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:52.012 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:52.013 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:52.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:52.092+0000 7f8a0d6d0980 1 bucket sync caught up with source: 2024-03-04T17:57:52.091 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.76.14, , ] 2024-03-04T17:57:52.092 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000001.76.14, , ] 2024-03-04T17:57:52.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:52.092+0000 7f8a0d6d0980 0 bucket checkpoint complete 2024-03-04T17:57:52.113 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-27 2024-03-04T17:57:52.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:52.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:52.114 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-27 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:52.163 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:52.163 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:52.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:52.231+0000 7f8ec7e2c980 1 bucket sync caught up with source: 2024-03-04T17:57:52.229 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000001.76.14, , ] 2024-03-04T17:57:52.229 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000001.76.14, , ] 2024-03-04T17:57:52.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:52.231+0000 7f8ec7e2c980 0 bucket checkpoint complete 2024-03-04T17:57:52.253 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-27 zones={test-zone2, test-zone1} 2024-03-04T17:57:52.254 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.254 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.254 DEBUG:boto:Method: HEAD 2024-03-04T17:57:52.254 DEBUG:boto:Path: /ppfahn-27/ 2024-03-04T17:57:52.254 DEBUG:boto:Data: 2024-03-04T17:57:52.254 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.254 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:52.254 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.254 DEBUG:boto:Params: {} 2024-03-04T17:57:52.255 DEBUG:boto:Token: None 2024-03-04T17:57:52.255 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/ 2024-03-04T17:57:52.255 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LOH6WVXV0JSGTXcQSAqa0ynKz3s= 2024-03-04T17:57:52.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LOH6WVXV0JSGTXcQSAqa0ynKz3s=', 'Content-Length': '0'} 2024-03-04T17:57:52.258 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f35a9129b091129-0065e60ba0-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.258 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.258 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.258 DEBUG:boto:Method: HEAD 2024-03-04T17:57:52.258 DEBUG:boto:Path: /ppfahn-27/ 2024-03-04T17:57:52.258 DEBUG:boto:Data: 2024-03-04T17:57:52.258 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.258 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:52.258 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.258 DEBUG:boto:Params: {} 2024-03-04T17:57:52.259 DEBUG:boto:Token: None 2024-03-04T17:57:52.259 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/ 2024-03-04T17:57:52.259 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LOH6WVXV0JSGTXcQSAqa0ynKz3s= 2024-03-04T17:57:52.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LOH6WVXV0JSGTXcQSAqa0ynKz3s=', 'Content-Length': '0'} 2024-03-04T17:57:52.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db1b6f96e00bf7e2-0065e60ba0-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.263 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:52.263 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.263 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.263 DEBUG:boto:path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.263 DEBUG:boto:auth_path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.264 DEBUG:boto:Method: GET 2024-03-04T17:57:52.264 DEBUG:boto:Path: /ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.264 DEBUG:boto:Data: 2024-03-04T17:57:52.264 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.264 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:52.264 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.264 DEBUG:boto:Params: {} 2024-03-04T17:57:52.264 DEBUG:boto:Token: None 2024-03-04T17:57:52.264 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/?versions 2024-03-04T17:57:52.264 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g= 2024-03-04T17:57:52.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g=', 'Content-Length': '0'} 2024-03-04T17:57:52.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fdcef282440f8ab-0065e60ba0-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.308 DEBUG:boto:b'ppfahn-27999falseobjnulltrue2024-03-04T17:57:21.138Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:52.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:52.309 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:52.309 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.309 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.309 DEBUG:boto:path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.309 DEBUG:boto:auth_path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.309 DEBUG:boto:Method: GET 2024-03-04T17:57:52.309 DEBUG:boto:Path: /ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.309 DEBUG:boto:Data: 2024-03-04T17:57:52.309 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.309 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:52.309 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.310 DEBUG:boto:Params: {} 2024-03-04T17:57:52.310 DEBUG:boto:Token: None 2024-03-04T17:57:52.310 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/?versions 2024-03-04T17:57:52.310 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g= 2024-03-04T17:57:52.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g=', 'Content-Length': '0'} 2024-03-04T17:57:52.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3b75d52a7c4e677-0065e60ba0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.315 DEBUG:boto:b'ppfahn-27999falseobjnulltrue2024-03-04T17:57:21.138Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:52.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:52.315 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.315 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.315 DEBUG:boto:path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.315 DEBUG:boto:auth_path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.315 DEBUG:boto:Method: GET 2024-03-04T17:57:52.315 DEBUG:boto:Path: /ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.316 DEBUG:boto:Data: 2024-03-04T17:57:52.316 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.316 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:52.316 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.316 DEBUG:boto:Params: {} 2024-03-04T17:57:52.316 DEBUG:boto:Token: None 2024-03-04T17:57:52.316 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/?versions 2024-03-04T17:57:52.316 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g= 2024-03-04T17:57:52.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g=', 'Content-Length': '0'} 2024-03-04T17:57:52.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a3261e25aa99cae-0065e60ba0-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.319 DEBUG:boto:b'ppfahn-27999falseobjnulltrue2024-03-04T17:57:21.138Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:52.319 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.319 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.320 DEBUG:boto:path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.320 DEBUG:boto:auth_path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.320 DEBUG:boto:Method: GET 2024-03-04T17:57:52.320 DEBUG:boto:Path: /ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.320 DEBUG:boto:Data: 2024-03-04T17:57:52.320 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.320 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:52.320 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.320 DEBUG:boto:Params: {} 2024-03-04T17:57:52.320 DEBUG:boto:Token: None 2024-03-04T17:57:52.320 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/?versions 2024-03-04T17:57:52.321 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g= 2024-03-04T17:57:52.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g=', 'Content-Length': '0'} 2024-03-04T17:57:52.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066ff4afd692d17ab-0065e60ba0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.328 DEBUG:boto:b'ppfahn-27999falseobjnulltrue2024-03-04T17:57:21.138Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:52.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:52.328 DEBUG:boto:path=/ppfahn-27/obj 2024-03-04T17:57:52.328 DEBUG:boto:auth_path=/ppfahn-27/obj 2024-03-04T17:57:52.328 DEBUG:boto:Method: HEAD 2024-03-04T17:57:52.328 DEBUG:boto:Path: /ppfahn-27/obj 2024-03-04T17:57:52.328 DEBUG:boto:Data: 2024-03-04T17:57:52.328 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.329 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:52.329 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.329 DEBUG:boto:Params: {} 2024-03-04T17:57:52.329 DEBUG:boto:Token: None 2024-03-04T17:57:52.329 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/obj 2024-03-04T17:57:52.329 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EwDb5FDPhN/Sb2hbeDuYpj/O5CY= 2024-03-04T17:57:52.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EwDb5FDPhN/Sb2hbeDuYpj/O5CY=', 'Content-Length': '0'} 2024-03-04T17:57:52.331 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000090de6a94b0d72c90-0065e60ba0-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.332 DEBUG:boto:path=/ppfahn-27/obj 2024-03-04T17:57:52.332 DEBUG:boto:auth_path=/ppfahn-27/obj 2024-03-04T17:57:52.332 DEBUG:boto:Method: HEAD 2024-03-04T17:57:52.332 DEBUG:boto:Path: /ppfahn-27/obj 2024-03-04T17:57:52.332 DEBUG:boto:Data: 2024-03-04T17:57:52.332 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.332 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:52.332 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.332 DEBUG:boto:Params: {} 2024-03-04T17:57:52.332 DEBUG:boto:Token: None 2024-03-04T17:57:52.332 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/obj 2024-03-04T17:57:52.333 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EwDb5FDPhN/Sb2hbeDuYpj/O5CY= 2024-03-04T17:57:52.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EwDb5FDPhN/Sb2hbeDuYpj/O5CY=', 'Content-Length': '0'} 2024-03-04T17:57:52.335 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000060681d83a5347da4-0065e60ba0-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.335 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-27 zones={test-zone2, test-zone1} 2024-03-04T17:57:52.335 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-27 zones={test-zone3, test-zone1} 2024-03-04T17:57:52.335 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.335 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.335 DEBUG:boto:Method: HEAD 2024-03-04T17:57:52.335 DEBUG:boto:Path: /ppfahn-27/ 2024-03-04T17:57:52.336 DEBUG:boto:Data: 2024-03-04T17:57:52.336 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.336 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:52.336 DEBUG:boto:Port: 8001 2024-03-04T17:57:52.336 DEBUG:boto:Params: {} 2024-03-04T17:57:52.336 DEBUG:boto:Token: None 2024-03-04T17:57:52.336 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/ 2024-03-04T17:57:52.336 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LOH6WVXV0JSGTXcQSAqa0ynKz3s= 2024-03-04T17:57:52.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LOH6WVXV0JSGTXcQSAqa0ynKz3s=', 'Content-Length': '0'} 2024-03-04T17:57:52.340 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000451d892597b922d4-0065e60ba0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.341 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.341 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.341 DEBUG:boto:Method: HEAD 2024-03-04T17:57:52.341 DEBUG:boto:Path: /ppfahn-27/ 2024-03-04T17:57:52.341 DEBUG:boto:Data: 2024-03-04T17:57:52.341 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.341 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:52.341 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.341 DEBUG:boto:Params: {} 2024-03-04T17:57:52.341 DEBUG:boto:Token: None 2024-03-04T17:57:52.342 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/ 2024-03-04T17:57:52.342 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LOH6WVXV0JSGTXcQSAqa0ynKz3s= 2024-03-04T17:57:52.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LOH6WVXV0JSGTXcQSAqa0ynKz3s=', 'Content-Length': '0'} 2024-03-04T17:57:52.345 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093414b0bfa49380e-0065e60ba0-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.346 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:52.346 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.346 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.346 DEBUG:boto:path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.346 DEBUG:boto:auth_path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.346 DEBUG:boto:Method: GET 2024-03-04T17:57:52.346 DEBUG:boto:Path: /ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.346 DEBUG:boto:Data: 2024-03-04T17:57:52.346 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.346 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:52.346 DEBUG:boto:Port: 8001 2024-03-04T17:57:52.346 DEBUG:boto:Params: {} 2024-03-04T17:57:52.347 DEBUG:boto:Token: None 2024-03-04T17:57:52.347 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/?versions 2024-03-04T17:57:52.347 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g= 2024-03-04T17:57:52.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g=', 'Content-Length': '0'} 2024-03-04T17:57:52.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009580b16891fe2699-0065e60ba0-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.392 DEBUG:boto:b'ppfahn-27999falseobjnulltrue2024-03-04T17:57:21.138Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:52.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:52.392 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:52.392 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.393 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.393 DEBUG:boto:path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.393 DEBUG:boto:auth_path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.393 DEBUG:boto:Method: GET 2024-03-04T17:57:52.393 DEBUG:boto:Path: /ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.393 DEBUG:boto:Data: 2024-03-04T17:57:52.393 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.393 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:52.393 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.393 DEBUG:boto:Params: {} 2024-03-04T17:57:52.393 DEBUG:boto:Token: None 2024-03-04T17:57:52.393 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/?versions 2024-03-04T17:57:52.394 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g= 2024-03-04T17:57:52.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g=', 'Content-Length': '0'} 2024-03-04T17:57:52.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6a4b24032d5a2be-0065e60ba0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.398 DEBUG:boto:b'ppfahn-27999falseobjnulltrue2024-03-04T17:57:21.138Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:52.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:52.398 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.398 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.398 DEBUG:boto:path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.398 DEBUG:boto:auth_path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.398 DEBUG:boto:Method: GET 2024-03-04T17:57:52.398 DEBUG:boto:Path: /ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.399 DEBUG:boto:Data: 2024-03-04T17:57:52.399 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.399 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:52.399 DEBUG:boto:Port: 8001 2024-03-04T17:57:52.399 DEBUG:boto:Params: {} 2024-03-04T17:57:52.399 DEBUG:boto:Token: None 2024-03-04T17:57:52.399 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/?versions 2024-03-04T17:57:52.399 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g= 2024-03-04T17:57:52.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g=', 'Content-Length': '0'} 2024-03-04T17:57:52.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000155a2a114f97751c-0065e60ba0-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.403 DEBUG:boto:b'ppfahn-27999falseobjnulltrue2024-03-04T17:57:21.138Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:52.404 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.404 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.404 DEBUG:boto:path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.404 DEBUG:boto:auth_path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.404 DEBUG:boto:Method: GET 2024-03-04T17:57:52.404 DEBUG:boto:Path: /ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.404 DEBUG:boto:Data: 2024-03-04T17:57:52.404 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.404 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:52.404 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.404 DEBUG:boto:Params: {} 2024-03-04T17:57:52.405 DEBUG:boto:Token: None 2024-03-04T17:57:52.405 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/?versions 2024-03-04T17:57:52.405 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g= 2024-03-04T17:57:52.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g=', 'Content-Length': '0'} 2024-03-04T17:57:52.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3d4253d8a6401ec-0065e60ba0-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.452 DEBUG:boto:b'ppfahn-27999falseobjnulltrue2024-03-04T17:57:21.138Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:52.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:52.452 DEBUG:boto:path=/ppfahn-27/obj 2024-03-04T17:57:52.453 DEBUG:boto:auth_path=/ppfahn-27/obj 2024-03-04T17:57:52.453 DEBUG:boto:Method: HEAD 2024-03-04T17:57:52.453 DEBUG:boto:Path: /ppfahn-27/obj 2024-03-04T17:57:52.453 DEBUG:boto:Data: 2024-03-04T17:57:52.453 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.453 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:52.453 DEBUG:boto:Port: 8001 2024-03-04T17:57:52.453 DEBUG:boto:Params: {} 2024-03-04T17:57:52.453 DEBUG:boto:Token: None 2024-03-04T17:57:52.454 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/obj 2024-03-04T17:57:52.454 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EwDb5FDPhN/Sb2hbeDuYpj/O5CY= 2024-03-04T17:57:52.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EwDb5FDPhN/Sb2hbeDuYpj/O5CY=', 'Content-Length': '0'} 2024-03-04T17:57:52.456 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000003ea0b1d5d574ad65-0065e60ba0-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.456 DEBUG:boto:path=/ppfahn-27/obj 2024-03-04T17:57:52.457 DEBUG:boto:auth_path=/ppfahn-27/obj 2024-03-04T17:57:52.457 DEBUG:boto:Method: HEAD 2024-03-04T17:57:52.457 DEBUG:boto:Path: /ppfahn-27/obj 2024-03-04T17:57:52.457 DEBUG:boto:Data: 2024-03-04T17:57:52.457 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.457 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:52.457 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.457 DEBUG:boto:Params: {} 2024-03-04T17:57:52.457 DEBUG:boto:Token: None 2024-03-04T17:57:52.457 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/obj 2024-03-04T17:57:52.457 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EwDb5FDPhN/Sb2hbeDuYpj/O5CY= 2024-03-04T17:57:52.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EwDb5FDPhN/Sb2hbeDuYpj/O5CY=', 'Content-Length': '0'} 2024-03-04T17:57:52.460 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d57ec7e016f0d23a-0065e60ba0-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.460 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-27 zones={test-zone3, test-zone1} 2024-03-04T17:57:52.460 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-27 zones={test-zone3, test-zone2} 2024-03-04T17:57:52.460 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.460 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.460 DEBUG:boto:Method: HEAD 2024-03-04T17:57:52.461 DEBUG:boto:Path: /ppfahn-27/ 2024-03-04T17:57:52.461 DEBUG:boto:Data: 2024-03-04T17:57:52.461 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.461 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:52.461 DEBUG:boto:Port: 8001 2024-03-04T17:57:52.461 DEBUG:boto:Params: {} 2024-03-04T17:57:52.461 DEBUG:boto:Token: None 2024-03-04T17:57:52.461 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/ 2024-03-04T17:57:52.461 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LOH6WVXV0JSGTXcQSAqa0ynKz3s= 2024-03-04T17:57:52.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LOH6WVXV0JSGTXcQSAqa0ynKz3s=', 'Content-Length': '0'} 2024-03-04T17:57:52.465 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe9c8a51b40bc901-0065e60ba0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.465 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.465 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.465 DEBUG:boto:Method: HEAD 2024-03-04T17:57:52.465 DEBUG:boto:Path: /ppfahn-27/ 2024-03-04T17:57:52.465 DEBUG:boto:Data: 2024-03-04T17:57:52.465 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.465 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:52.466 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.466 DEBUG:boto:Params: {} 2024-03-04T17:57:52.466 DEBUG:boto:Token: None 2024-03-04T17:57:52.466 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/ 2024-03-04T17:57:52.466 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LOH6WVXV0JSGTXcQSAqa0ynKz3s= 2024-03-04T17:57:52.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LOH6WVXV0JSGTXcQSAqa0ynKz3s=', 'Content-Length': '0'} 2024-03-04T17:57:52.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000373808fc22cc48c7-0065e60ba0-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.470 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:52.470 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.471 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.471 DEBUG:boto:path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.471 DEBUG:boto:auth_path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.471 DEBUG:boto:Method: GET 2024-03-04T17:57:52.471 DEBUG:boto:Path: /ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.471 DEBUG:boto:Data: 2024-03-04T17:57:52.471 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.471 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:52.471 DEBUG:boto:Port: 8001 2024-03-04T17:57:52.471 DEBUG:boto:Params: {} 2024-03-04T17:57:52.471 DEBUG:boto:Token: None 2024-03-04T17:57:52.472 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/?versions 2024-03-04T17:57:52.472 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g= 2024-03-04T17:57:52.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g=', 'Content-Length': '0'} 2024-03-04T17:57:52.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082798c38e9e5c334-0065e60ba0-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.520 DEBUG:boto:b'ppfahn-27999falseobjnulltrue2024-03-04T17:57:21.138Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:52.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:52.520 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:52.521 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.521 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.521 DEBUG:boto:path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.521 DEBUG:boto:auth_path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.521 DEBUG:boto:Method: GET 2024-03-04T17:57:52.521 DEBUG:boto:Path: /ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.521 DEBUG:boto:Data: 2024-03-04T17:57:52.521 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.521 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:52.521 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.521 DEBUG:boto:Params: {} 2024-03-04T17:57:52.521 DEBUG:boto:Token: None 2024-03-04T17:57:52.522 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/?versions 2024-03-04T17:57:52.522 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g= 2024-03-04T17:57:52.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g=', 'Content-Length': '0'} 2024-03-04T17:57:52.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f15c03745a6166e2-0065e60ba0-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.526 DEBUG:boto:b'ppfahn-27999falseobjnulltrue2024-03-04T17:57:21.138Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:52.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:52.526 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.527 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.527 DEBUG:boto:path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.527 DEBUG:boto:auth_path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.527 DEBUG:boto:Method: GET 2024-03-04T17:57:52.527 DEBUG:boto:Path: /ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.527 DEBUG:boto:Data: 2024-03-04T17:57:52.527 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.527 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:52.527 DEBUG:boto:Port: 8001 2024-03-04T17:57:52.527 DEBUG:boto:Params: {} 2024-03-04T17:57:52.528 DEBUG:boto:Token: None 2024-03-04T17:57:52.528 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/?versions 2024-03-04T17:57:52.528 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g= 2024-03-04T17:57:52.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g=', 'Content-Length': '0'} 2024-03-04T17:57:52.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee133e44aa9ae1f-0065e60ba0-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.532 DEBUG:boto:b'ppfahn-27999falseobjnulltrue2024-03-04T17:57:21.138Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:52.533 DEBUG:boto:path=/ppfahn-27/ 2024-03-04T17:57:52.533 DEBUG:boto:auth_path=/ppfahn-27/ 2024-03-04T17:57:52.533 DEBUG:boto:path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.533 DEBUG:boto:auth_path=/ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.533 DEBUG:boto:Method: GET 2024-03-04T17:57:52.533 DEBUG:boto:Path: /ppfahn-27/?versions&max-keys=999 2024-03-04T17:57:52.533 DEBUG:boto:Data: 2024-03-04T17:57:52.533 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.534 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:52.534 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.534 DEBUG:boto:Params: {} 2024-03-04T17:57:52.534 DEBUG:boto:Token: None 2024-03-04T17:57:52.534 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/?versions 2024-03-04T17:57:52.534 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g= 2024-03-04T17:57:52.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:WLgaJMb89OXLWLCFMLQCB6Upg6g=', 'Content-Length': '0'} 2024-03-04T17:57:52.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ed54cd30ffade1a-0065e60ba0-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.538 DEBUG:boto:b'ppfahn-27999falseobjnulltrue2024-03-04T17:57:21.138Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:52.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:52.539 DEBUG:boto:path=/ppfahn-27/obj 2024-03-04T17:57:52.539 DEBUG:boto:auth_path=/ppfahn-27/obj 2024-03-04T17:57:52.539 DEBUG:boto:Method: HEAD 2024-03-04T17:57:52.539 DEBUG:boto:Path: /ppfahn-27/obj 2024-03-04T17:57:52.539 DEBUG:boto:Data: 2024-03-04T17:57:52.540 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.540 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:52.540 DEBUG:boto:Port: 8001 2024-03-04T17:57:52.540 DEBUG:boto:Params: {} 2024-03-04T17:57:52.540 DEBUG:boto:Token: None 2024-03-04T17:57:52.540 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/obj 2024-03-04T17:57:52.540 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EwDb5FDPhN/Sb2hbeDuYpj/O5CY= 2024-03-04T17:57:52.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EwDb5FDPhN/Sb2hbeDuYpj/O5CY=', 'Content-Length': '0'} 2024-03-04T17:57:52.543 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000ed248cf440b55fe9-0065e60ba0-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.543 DEBUG:boto:path=/ppfahn-27/obj 2024-03-04T17:57:52.543 DEBUG:boto:auth_path=/ppfahn-27/obj 2024-03-04T17:57:52.544 DEBUG:boto:Method: HEAD 2024-03-04T17:57:52.544 DEBUG:boto:Path: /ppfahn-27/obj 2024-03-04T17:57:52.544 DEBUG:boto:Data: 2024-03-04T17:57:52.544 DEBUG:boto:Headers: {} 2024-03-04T17:57:52.544 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:52.544 DEBUG:boto:Port: 8000 2024-03-04T17:57:52.544 DEBUG:boto:Params: {} 2024-03-04T17:57:52.544 DEBUG:boto:Token: None 2024-03-04T17:57:52.545 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:52 GMT /ppfahn-27/obj 2024-03-04T17:57:52.545 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EwDb5FDPhN/Sb2hbeDuYpj/O5CY= 2024-03-04T17:57:52.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EwDb5FDPhN/Sb2hbeDuYpj/O5CY=', 'Content-Length': '0'} 2024-03-04T17:57:52.547 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000339b01403c731660-0065e60ba0-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:52.548 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-27 zones={test-zone3, test-zone2} 2024-03-04T17:57:52.548 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-28 2024-03-04T17:57:52.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:52.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:52.548 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-28 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:52.598 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:52.599 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:52.663+0000 7f14cfa41980 1 bucket sync caught up with source: 2024-03-04T17:57:52.662 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000001.35.14, , ] 2024-03-04T17:57:52.662 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000001.35.14, , ] 2024-03-04T17:57:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:52.663+0000 7f14cfa41980 0 bucket checkpoint complete 2024-03-04T17:57:52.685 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-28 2024-03-04T17:57:52.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:52.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:52.685 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-28 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:52.736 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:52.736 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:52.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:52.800+0000 7f02619e8980 1 bucket sync caught up with source: 2024-03-04T17:57:52.796 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.35.14, , ] 2024-03-04T17:57:52.797 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000001.35.14, , ] 2024-03-04T17:57:52.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:52.800+0000 7f02619e8980 0 bucket checkpoint complete 2024-03-04T17:57:52.819 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-28 2024-03-04T17:57:52.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:52.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:52.820 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-28 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:52.862 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:52.862 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:52.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:52.924+0000 7fb33d8f7980 1 bucket sync caught up with source: 2024-03-04T17:57:52.923 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000003.50.7, , ] 2024-03-04T17:57:52.923 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.50.7, , ] 2024-03-04T17:57:52.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:52.924+0000 7fb33d8f7980 0 bucket checkpoint complete 2024-03-04T17:57:52.945 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-28 2024-03-04T17:57:52.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:52.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:52.946 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-28 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:53.033 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:53.033 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:53.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:53.096+0000 7f5062f77980 1 bucket sync caught up with source: 2024-03-04T17:57:53.094 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000003.50.7, , ] 2024-03-04T17:57:53.094 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.50.7, , ] 2024-03-04T17:57:53.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:53.096+0000 7f5062f77980 0 bucket checkpoint complete 2024-03-04T17:57:53.115 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-28 2024-03-04T17:57:53.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:53.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:53.116 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-28 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:53.156 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:53.156 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:53.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:53.216+0000 7f7370e6d980 1 bucket sync caught up with source: 2024-03-04T17:57:53.213 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.68.14, , ] 2024-03-04T17:57:53.213 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000001.68.14, , ] 2024-03-04T17:57:53.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:53.216+0000 7f7370e6d980 0 bucket checkpoint complete 2024-03-04T17:57:53.234 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-28 2024-03-04T17:57:53.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:53.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:53.234 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-28 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:53.284 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:53.285 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:53.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:53.347+0000 7fdeb7772980 1 bucket sync caught up with source: 2024-03-04T17:57:53.349 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000001.68.14, , ] 2024-03-04T17:57:53.349 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000001.68.14, , ] 2024-03-04T17:57:53.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:53.347+0000 7fdeb7772980 0 bucket checkpoint complete 2024-03-04T17:57:53.371 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-28 zones={test-zone2, test-zone1} 2024-03-04T17:57:53.371 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.371 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.371 DEBUG:boto:Method: HEAD 2024-03-04T17:57:53.371 DEBUG:boto:Path: /ppfahn-28/ 2024-03-04T17:57:53.372 DEBUG:boto:Data: 2024-03-04T17:57:53.372 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.372 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:53.372 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.372 DEBUG:boto:Params: {} 2024-03-04T17:57:53.372 DEBUG:boto:Token: None 2024-03-04T17:57:53.372 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/ 2024-03-04T17:57:53.372 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IxWH447XHYttWMV9xe9qw8vKP+w= 2024-03-04T17:57:53.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IxWH447XHYttWMV9xe9qw8vKP+w=', 'Content-Length': '0'} 2024-03-04T17:57:53.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043d26c1d4827638b-0065e60ba1-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.375 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.375 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.375 DEBUG:boto:Method: HEAD 2024-03-04T17:57:53.375 DEBUG:boto:Path: /ppfahn-28/ 2024-03-04T17:57:53.375 DEBUG:boto:Data: 2024-03-04T17:57:53.375 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.375 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:53.375 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.376 DEBUG:boto:Params: {} 2024-03-04T17:57:53.376 DEBUG:boto:Token: None 2024-03-04T17:57:53.376 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/ 2024-03-04T17:57:53.376 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IxWH447XHYttWMV9xe9qw8vKP+w= 2024-03-04T17:57:53.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IxWH447XHYttWMV9xe9qw8vKP+w=', 'Content-Length': '0'} 2024-03-04T17:57:53.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000179eac5f86130bfb-0065e60ba1-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.380 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:53.380 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.381 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.381 DEBUG:boto:path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.381 DEBUG:boto:auth_path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.381 DEBUG:boto:Method: GET 2024-03-04T17:57:53.381 DEBUG:boto:Path: /ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.381 DEBUG:boto:Data: 2024-03-04T17:57:53.381 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.381 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:53.381 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.381 DEBUG:boto:Params: {} 2024-03-04T17:57:53.381 DEBUG:boto:Token: None 2024-03-04T17:57:53.382 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/?versions 2024-03-04T17:57:53.382 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI= 2024-03-04T17:57:53.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI=', 'Content-Length': '0'} 2024-03-04T17:57:53.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f55fd92ef9dd6df-0065e60ba1-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.384 DEBUG:boto:b'ppfahn-28999falseobjnulltrue2024-03-04T17:57:21.207Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:53.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:53.384 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:53.385 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.385 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.385 DEBUG:boto:path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.385 DEBUG:boto:auth_path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.385 DEBUG:boto:Method: GET 2024-03-04T17:57:53.385 DEBUG:boto:Path: /ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.385 DEBUG:boto:Data: 2024-03-04T17:57:53.385 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.385 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:53.385 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.386 DEBUG:boto:Params: {} 2024-03-04T17:57:53.386 DEBUG:boto:Token: None 2024-03-04T17:57:53.386 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/?versions 2024-03-04T17:57:53.386 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI= 2024-03-04T17:57:53.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI=', 'Content-Length': '0'} 2024-03-04T17:57:53.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ac05cf1cb2b2174-0065e60ba1-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.391 DEBUG:boto:b'ppfahn-28999falseobjnulltrue2024-03-04T17:57:21.207Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:53.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:53.391 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.391 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.391 DEBUG:boto:path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.391 DEBUG:boto:auth_path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.391 DEBUG:boto:Method: GET 2024-03-04T17:57:53.391 DEBUG:boto:Path: /ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.391 DEBUG:boto:Data: 2024-03-04T17:57:53.392 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.392 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:53.392 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.392 DEBUG:boto:Params: {} 2024-03-04T17:57:53.392 DEBUG:boto:Token: None 2024-03-04T17:57:53.392 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/?versions 2024-03-04T17:57:53.392 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI= 2024-03-04T17:57:53.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI=', 'Content-Length': '0'} 2024-03-04T17:57:53.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eecf1f5084e905dc-0065e60ba1-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.395 DEBUG:boto:b'ppfahn-28999falseobjnulltrue2024-03-04T17:57:21.207Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:53.395 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.395 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.395 DEBUG:boto:path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.395 DEBUG:boto:auth_path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.395 DEBUG:boto:Method: GET 2024-03-04T17:57:53.395 DEBUG:boto:Path: /ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.395 DEBUG:boto:Data: 2024-03-04T17:57:53.396 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.396 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:53.396 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.396 DEBUG:boto:Params: {} 2024-03-04T17:57:53.396 DEBUG:boto:Token: None 2024-03-04T17:57:53.399 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/?versions 2024-03-04T17:57:53.399 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI= 2024-03-04T17:57:53.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI=', 'Content-Length': '0'} 2024-03-04T17:57:53.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f66438ea8237f42-0065e60ba1-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.444 DEBUG:boto:b'ppfahn-28999falseobjnulltrue2024-03-04T17:57:21.207Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:53.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:53.444 DEBUG:boto:path=/ppfahn-28/obj 2024-03-04T17:57:53.445 DEBUG:boto:auth_path=/ppfahn-28/obj 2024-03-04T17:57:53.445 DEBUG:boto:Method: HEAD 2024-03-04T17:57:53.445 DEBUG:boto:Path: /ppfahn-28/obj 2024-03-04T17:57:53.445 DEBUG:boto:Data: 2024-03-04T17:57:53.445 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.445 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:53.445 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.445 DEBUG:boto:Params: {} 2024-03-04T17:57:53.445 DEBUG:boto:Token: None 2024-03-04T17:57:53.446 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/obj 2024-03-04T17:57:53.446 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUJFzwfsW1NCjUWMU2sAwaJ3xgg= 2024-03-04T17:57:53.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUJFzwfsW1NCjUWMU2sAwaJ3xgg=', 'Content-Length': '0'} 2024-03-04T17:57:53.448 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a41d5589f3430d6b-0065e60ba1-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.448 DEBUG:boto:path=/ppfahn-28/obj 2024-03-04T17:57:53.448 DEBUG:boto:auth_path=/ppfahn-28/obj 2024-03-04T17:57:53.448 DEBUG:boto:Method: HEAD 2024-03-04T17:57:53.449 DEBUG:boto:Path: /ppfahn-28/obj 2024-03-04T17:57:53.449 DEBUG:boto:Data: 2024-03-04T17:57:53.449 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.449 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:53.449 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.449 DEBUG:boto:Params: {} 2024-03-04T17:57:53.449 DEBUG:boto:Token: None 2024-03-04T17:57:53.449 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/obj 2024-03-04T17:57:53.449 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUJFzwfsW1NCjUWMU2sAwaJ3xgg= 2024-03-04T17:57:53.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUJFzwfsW1NCjUWMU2sAwaJ3xgg=', 'Content-Length': '0'} 2024-03-04T17:57:53.452 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000ef71192196912fec-0065e60ba1-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.452 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-28 zones={test-zone2, test-zone1} 2024-03-04T17:57:53.452 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-28 zones={test-zone3, test-zone1} 2024-03-04T17:57:53.453 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.453 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.453 DEBUG:boto:Method: HEAD 2024-03-04T17:57:53.453 DEBUG:boto:Path: /ppfahn-28/ 2024-03-04T17:57:53.453 DEBUG:boto:Data: 2024-03-04T17:57:53.453 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.453 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:53.453 DEBUG:boto:Port: 8001 2024-03-04T17:57:53.453 DEBUG:boto:Params: {} 2024-03-04T17:57:53.453 DEBUG:boto:Token: None 2024-03-04T17:57:53.453 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/ 2024-03-04T17:57:53.454 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IxWH447XHYttWMV9xe9qw8vKP+w= 2024-03-04T17:57:53.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IxWH447XHYttWMV9xe9qw8vKP+w=', 'Content-Length': '0'} 2024-03-04T17:57:53.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c86d86617b701ac2-0065e60ba1-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.458 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.459 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.459 DEBUG:boto:Method: HEAD 2024-03-04T17:57:53.459 DEBUG:boto:Path: /ppfahn-28/ 2024-03-04T17:57:53.459 DEBUG:boto:Data: 2024-03-04T17:57:53.459 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.459 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:53.459 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.459 DEBUG:boto:Params: {} 2024-03-04T17:57:53.459 DEBUG:boto:Token: None 2024-03-04T17:57:53.459 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/ 2024-03-04T17:57:53.460 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IxWH447XHYttWMV9xe9qw8vKP+w= 2024-03-04T17:57:53.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IxWH447XHYttWMV9xe9qw8vKP+w=', 'Content-Length': '0'} 2024-03-04T17:57:53.464 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004195dff926f072d8-0065e60ba1-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.464 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:53.464 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.464 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.464 DEBUG:boto:path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.465 DEBUG:boto:auth_path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.465 DEBUG:boto:Method: GET 2024-03-04T17:57:53.465 DEBUG:boto:Path: /ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.465 DEBUG:boto:Data: 2024-03-04T17:57:53.465 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.465 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:53.465 DEBUG:boto:Port: 8001 2024-03-04T17:57:53.465 DEBUG:boto:Params: {} 2024-03-04T17:57:53.465 DEBUG:boto:Token: None 2024-03-04T17:57:53.465 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/?versions 2024-03-04T17:57:53.466 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI= 2024-03-04T17:57:53.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI=', 'Content-Length': '0'} 2024-03-04T17:57:53.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b93587cfee141b8-0065e60ba1-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.512 DEBUG:boto:b'ppfahn-28999falseobjnulltrue2024-03-04T17:57:21.207Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:53.512 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:53.512 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:53.512 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.513 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.513 DEBUG:boto:path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.513 DEBUG:boto:auth_path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.513 DEBUG:boto:Method: GET 2024-03-04T17:57:53.513 DEBUG:boto:Path: /ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.513 DEBUG:boto:Data: 2024-03-04T17:57:53.513 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.513 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:53.513 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.513 DEBUG:boto:Params: {} 2024-03-04T17:57:53.513 DEBUG:boto:Token: None 2024-03-04T17:57:53.514 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/?versions 2024-03-04T17:57:53.514 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI= 2024-03-04T17:57:53.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI=', 'Content-Length': '0'} 2024-03-04T17:57:53.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001363744bff42d781-0065e60ba1-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.519 DEBUG:boto:b'ppfahn-28999falseobjnulltrue2024-03-04T17:57:21.207Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:53.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:53.519 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.519 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.519 DEBUG:boto:path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.520 DEBUG:boto:auth_path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.520 DEBUG:boto:Method: GET 2024-03-04T17:57:53.520 DEBUG:boto:Path: /ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.520 DEBUG:boto:Data: 2024-03-04T17:57:53.520 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.520 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:53.520 DEBUG:boto:Port: 8001 2024-03-04T17:57:53.520 DEBUG:boto:Params: {} 2024-03-04T17:57:53.520 DEBUG:boto:Token: None 2024-03-04T17:57:53.520 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/?versions 2024-03-04T17:57:53.521 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI= 2024-03-04T17:57:53.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI=', 'Content-Length': '0'} 2024-03-04T17:57:53.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baa73efe89973db6-0065e60ba1-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.525 DEBUG:boto:b'ppfahn-28999falseobjnulltrue2024-03-04T17:57:21.207Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:53.525 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.525 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.526 DEBUG:boto:path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.526 DEBUG:boto:auth_path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.526 DEBUG:boto:Method: GET 2024-03-04T17:57:53.526 DEBUG:boto:Path: /ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.526 DEBUG:boto:Data: 2024-03-04T17:57:53.526 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.526 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:53.526 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.526 DEBUG:boto:Params: {} 2024-03-04T17:57:53.526 DEBUG:boto:Token: None 2024-03-04T17:57:53.526 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/?versions 2024-03-04T17:57:53.527 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI= 2024-03-04T17:57:53.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI=', 'Content-Length': '0'} 2024-03-04T17:57:53.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f9b288e88d78bc-0065e60ba1-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.572 DEBUG:boto:b'ppfahn-28999falseobjnulltrue2024-03-04T17:57:21.207Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:53.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:53.572 DEBUG:boto:path=/ppfahn-28/obj 2024-03-04T17:57:53.572 DEBUG:boto:auth_path=/ppfahn-28/obj 2024-03-04T17:57:53.573 DEBUG:boto:Method: HEAD 2024-03-04T17:57:53.573 DEBUG:boto:Path: /ppfahn-28/obj 2024-03-04T17:57:53.573 DEBUG:boto:Data: 2024-03-04T17:57:53.573 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.573 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:53.573 DEBUG:boto:Port: 8001 2024-03-04T17:57:53.573 DEBUG:boto:Params: {} 2024-03-04T17:57:53.573 DEBUG:boto:Token: None 2024-03-04T17:57:53.573 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/obj 2024-03-04T17:57:53.573 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUJFzwfsW1NCjUWMU2sAwaJ3xgg= 2024-03-04T17:57:53.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUJFzwfsW1NCjUWMU2sAwaJ3xgg=', 'Content-Length': '0'} 2024-03-04T17:57:53.576 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000009b1fa0fe068c438-0065e60ba1-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.576 DEBUG:boto:path=/ppfahn-28/obj 2024-03-04T17:57:53.576 DEBUG:boto:auth_path=/ppfahn-28/obj 2024-03-04T17:57:53.576 DEBUG:boto:Method: HEAD 2024-03-04T17:57:53.577 DEBUG:boto:Path: /ppfahn-28/obj 2024-03-04T17:57:53.577 DEBUG:boto:Data: 2024-03-04T17:57:53.577 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.577 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:53.577 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.577 DEBUG:boto:Params: {} 2024-03-04T17:57:53.577 DEBUG:boto:Token: None 2024-03-04T17:57:53.577 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/obj 2024-03-04T17:57:53.577 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUJFzwfsW1NCjUWMU2sAwaJ3xgg= 2024-03-04T17:57:53.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUJFzwfsW1NCjUWMU2sAwaJ3xgg=', 'Content-Length': '0'} 2024-03-04T17:57:53.580 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000002f41de165db069c6-0065e60ba1-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.580 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-28 zones={test-zone3, test-zone1} 2024-03-04T17:57:53.580 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-28 zones={test-zone3, test-zone2} 2024-03-04T17:57:53.580 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.580 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.580 DEBUG:boto:Method: HEAD 2024-03-04T17:57:53.581 DEBUG:boto:Path: /ppfahn-28/ 2024-03-04T17:57:53.581 DEBUG:boto:Data: 2024-03-04T17:57:53.581 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.581 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:53.581 DEBUG:boto:Port: 8001 2024-03-04T17:57:53.581 DEBUG:boto:Params: {} 2024-03-04T17:57:53.581 DEBUG:boto:Token: None 2024-03-04T17:57:53.581 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/ 2024-03-04T17:57:53.581 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IxWH447XHYttWMV9xe9qw8vKP+w= 2024-03-04T17:57:53.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IxWH447XHYttWMV9xe9qw8vKP+w=', 'Content-Length': '0'} 2024-03-04T17:57:53.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031f0e27833859535-0065e60ba1-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.585 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.586 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.586 DEBUG:boto:Method: HEAD 2024-03-04T17:57:53.586 DEBUG:boto:Path: /ppfahn-28/ 2024-03-04T17:57:53.586 DEBUG:boto:Data: 2024-03-04T17:57:53.586 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.586 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:53.586 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.586 DEBUG:boto:Params: {} 2024-03-04T17:57:53.586 DEBUG:boto:Token: None 2024-03-04T17:57:53.586 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/ 2024-03-04T17:57:53.587 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IxWH447XHYttWMV9xe9qw8vKP+w= 2024-03-04T17:57:53.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IxWH447XHYttWMV9xe9qw8vKP+w=', 'Content-Length': '0'} 2024-03-04T17:57:53.591 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a21ed219a26e7b5a-0065e60ba1-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.591 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:53.591 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.591 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.591 DEBUG:boto:path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.591 DEBUG:boto:auth_path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.591 DEBUG:boto:Method: GET 2024-03-04T17:57:53.591 DEBUG:boto:Path: /ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.591 DEBUG:boto:Data: 2024-03-04T17:57:53.591 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.592 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:53.592 DEBUG:boto:Port: 8001 2024-03-04T17:57:53.592 DEBUG:boto:Params: {} 2024-03-04T17:57:53.592 DEBUG:boto:Token: None 2024-03-04T17:57:53.592 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/?versions 2024-03-04T17:57:53.592 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI= 2024-03-04T17:57:53.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI=', 'Content-Length': '0'} 2024-03-04T17:57:53.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010cdf46fbf9e4b43-0065e60ba1-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.640 DEBUG:boto:b'ppfahn-28999falseobjnulltrue2024-03-04T17:57:21.207Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:53.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:53.640 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:53.641 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.641 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.641 DEBUG:boto:path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.641 DEBUG:boto:auth_path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.641 DEBUG:boto:Method: GET 2024-03-04T17:57:53.641 DEBUG:boto:Path: /ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.641 DEBUG:boto:Data: 2024-03-04T17:57:53.641 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.641 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:53.641 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.641 DEBUG:boto:Params: {} 2024-03-04T17:57:53.642 DEBUG:boto:Token: None 2024-03-04T17:57:53.642 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/?versions 2024-03-04T17:57:53.642 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI= 2024-03-04T17:57:53.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI=', 'Content-Length': '0'} 2024-03-04T17:57:53.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cde49c758028be25-0065e60ba1-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.646 DEBUG:boto:b'ppfahn-28999falseobjnulltrue2024-03-04T17:57:21.207Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:53.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:53.647 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.647 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.647 DEBUG:boto:path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.647 DEBUG:boto:auth_path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.647 DEBUG:boto:Method: GET 2024-03-04T17:57:53.647 DEBUG:boto:Path: /ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.647 DEBUG:boto:Data: 2024-03-04T17:57:53.647 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.647 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:53.647 DEBUG:boto:Port: 8001 2024-03-04T17:57:53.647 DEBUG:boto:Params: {} 2024-03-04T17:57:53.647 DEBUG:boto:Token: None 2024-03-04T17:57:53.648 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/?versions 2024-03-04T17:57:53.648 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI= 2024-03-04T17:57:53.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI=', 'Content-Length': '0'} 2024-03-04T17:57:53.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000033726faa6e3b8db-0065e60ba1-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.652 DEBUG:boto:b'ppfahn-28999falseobjnulltrue2024-03-04T17:57:21.207Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:53.652 DEBUG:boto:path=/ppfahn-28/ 2024-03-04T17:57:53.653 DEBUG:boto:auth_path=/ppfahn-28/ 2024-03-04T17:57:53.653 DEBUG:boto:path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.653 DEBUG:boto:auth_path=/ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.653 DEBUG:boto:Method: GET 2024-03-04T17:57:53.653 DEBUG:boto:Path: /ppfahn-28/?versions&max-keys=999 2024-03-04T17:57:53.653 DEBUG:boto:Data: 2024-03-04T17:57:53.653 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.653 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:53.653 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.653 DEBUG:boto:Params: {} 2024-03-04T17:57:53.653 DEBUG:boto:Token: None 2024-03-04T17:57:53.654 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/?versions 2024-03-04T17:57:53.654 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI= 2024-03-04T17:57:53.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VYprQ4DP1CqThpGeoFLBhWysfLI=', 'Content-Length': '0'} 2024-03-04T17:57:53.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fccb2fbd0c302b3-0065e60ba1-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.658 DEBUG:boto:b'ppfahn-28999falseobjnulltrue2024-03-04T17:57:21.207Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:53.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:53.658 DEBUG:boto:path=/ppfahn-28/obj 2024-03-04T17:57:53.659 DEBUG:boto:auth_path=/ppfahn-28/obj 2024-03-04T17:57:53.659 DEBUG:boto:Method: HEAD 2024-03-04T17:57:53.659 DEBUG:boto:Path: /ppfahn-28/obj 2024-03-04T17:57:53.659 DEBUG:boto:Data: 2024-03-04T17:57:53.659 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.659 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:53.659 DEBUG:boto:Port: 8001 2024-03-04T17:57:53.659 DEBUG:boto:Params: {} 2024-03-04T17:57:53.659 DEBUG:boto:Token: None 2024-03-04T17:57:53.659 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/obj 2024-03-04T17:57:53.659 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUJFzwfsW1NCjUWMU2sAwaJ3xgg= 2024-03-04T17:57:53.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUJFzwfsW1NCjUWMU2sAwaJ3xgg=', 'Content-Length': '0'} 2024-03-04T17:57:53.662 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000b20db348fab6f1ed-0065e60ba1-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.662 DEBUG:boto:path=/ppfahn-28/obj 2024-03-04T17:57:53.662 DEBUG:boto:auth_path=/ppfahn-28/obj 2024-03-04T17:57:53.662 DEBUG:boto:Method: HEAD 2024-03-04T17:57:53.663 DEBUG:boto:Path: /ppfahn-28/obj 2024-03-04T17:57:53.663 DEBUG:boto:Data: 2024-03-04T17:57:53.663 DEBUG:boto:Headers: {} 2024-03-04T17:57:53.663 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:53.663 DEBUG:boto:Port: 8000 2024-03-04T17:57:53.663 DEBUG:boto:Params: {} 2024-03-04T17:57:53.663 DEBUG:boto:Token: None 2024-03-04T17:57:53.663 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:53 GMT /ppfahn-28/obj 2024-03-04T17:57:53.663 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YUJFzwfsW1NCjUWMU2sAwaJ3xgg= 2024-03-04T17:57:53.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:53 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YUJFzwfsW1NCjUWMU2sAwaJ3xgg=', 'Content-Length': '0'} 2024-03-04T17:57:53.666 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000000d8961513682e3f2-0065e60ba1-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:53.666 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-28 zones={test-zone3, test-zone2} 2024-03-04T17:57:53.666 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=ppfahn-29 2024-03-04T17:57:53.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:53.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:53.666 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-29 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:53.717 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:53.717 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:53.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:53.783+0000 7fcfca58a980 1 bucket sync caught up with source: 2024-03-04T17:57:53.782 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000001.28.14, , ] 2024-03-04T17:57:53.782 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000001.28.14, , ] 2024-03-04T17:57:53.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:53.783+0000 7fcfca58a980 0 bucket checkpoint complete 2024-03-04T17:57:53.804 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=ppfahn-29 2024-03-04T17:57:53.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:53.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:53.808 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-29 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:53.858 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:53.859 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:53.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:53.920+0000 7fd39fab6980 1 bucket sync caught up with source: 2024-03-04T17:57:53.920 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.28.14, , ] 2024-03-04T17:57:53.920 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000001.28.14, , ] 2024-03-04T17:57:53.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:53.920+0000 7fd39fab6980 0 bucket checkpoint complete 2024-03-04T17:57:53.942 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=ppfahn-29 2024-03-04T17:57:53.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:53.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:53.942 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-29 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:53.983 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:53.983 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:54.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:54.044+0000 7f6876e77980 1 bucket sync caught up with source: 2024-03-04T17:57:54.043 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.54.14, , ] 2024-03-04T17:57:54.043 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000001.54.14, , ] 2024-03-04T17:57:54.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:54.044+0000 7f6876e77980 0 bucket checkpoint complete 2024-03-04T17:57:54.066 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=ppfahn-29 2024-03-04T17:57:54.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:54.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:54.066 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-29 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:54.149 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:54.149 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:54.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:54.212+0000 7fa50142e980 1 bucket sync caught up with source: 2024-03-04T17:57:54.211 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000001.54.14, , ] 2024-03-04T17:57:54.211 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000001.54.14, , ] 2024-03-04T17:57:54.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:54.212+0000 7fa50142e980 0 bucket checkpoint complete 2024-03-04T17:57:54.233 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=ppfahn-29 2024-03-04T17:57:54.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:54.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:54.234 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ppfahn-29 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:54.325 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:54.325 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:54.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:54.388+0000 7f3d60ada980 1 bucket sync caught up with source: 2024-03-04T17:57:54.385 INFO:teuthology.orchestra.run.smithi136.stderr: local status: [, , , , , , , , 00000000003.210.7, , ] 2024-03-04T17:57:54.385 INFO:teuthology.orchestra.run.smithi136.stderr: remote markers: [, , , , , , , , 00000000003.210.7, , ] 2024-03-04T17:57:54.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T17:57:54.388+0000 7f3d60ada980 0 bucket checkpoint complete 2024-03-04T17:57:54.408 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=ppfahn-29 2024-03-04T17:57:54.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:54.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T17:57:54.408 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-29 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T17:57:54.458 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:54.458 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:54.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:54.527+0000 7f417f1e5980 1 bucket sync caught up with source: 2024-03-04T17:57:54.521 INFO:teuthology.orchestra.run.smithi153.stderr: local status: [, , , , , , , , 00000000003.210.7, , ] 2024-03-04T17:57:54.521 INFO:teuthology.orchestra.run.smithi153.stderr: remote markers: [, , , , , , , , 00000000003.210.7, , ] 2024-03-04T17:57:54.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:54.527+0000 7f417f1e5980 0 bucket checkpoint complete 2024-03-04T17:57:54.545 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-29 zones={test-zone2, test-zone1} 2024-03-04T17:57:54.545 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.545 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.545 DEBUG:boto:Method: HEAD 2024-03-04T17:57:54.545 DEBUG:boto:Path: /ppfahn-29/ 2024-03-04T17:57:54.545 DEBUG:boto:Data: 2024-03-04T17:57:54.545 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.545 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:54.545 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.545 DEBUG:boto:Params: {} 2024-03-04T17:57:54.546 DEBUG:boto:Token: None 2024-03-04T17:57:54.546 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/ 2024-03-04T17:57:54.546 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l8nPEu+phulD0el8H1+yWCMtDN0= 2024-03-04T17:57:54.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l8nPEu+phulD0el8H1+yWCMtDN0=', 'Content-Length': '0'} 2024-03-04T17:57:54.549 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068aac3e6a39d47bb-0065e60ba2-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.549 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.549 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.549 DEBUG:boto:Method: HEAD 2024-03-04T17:57:54.549 DEBUG:boto:Path: /ppfahn-29/ 2024-03-04T17:57:54.549 DEBUG:boto:Data: 2024-03-04T17:57:54.549 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.549 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:54.549 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.549 DEBUG:boto:Params: {} 2024-03-04T17:57:54.550 DEBUG:boto:Token: None 2024-03-04T17:57:54.550 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/ 2024-03-04T17:57:54.550 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l8nPEu+phulD0el8H1+yWCMtDN0= 2024-03-04T17:57:54.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l8nPEu+phulD0el8H1+yWCMtDN0=', 'Content-Length': '0'} 2024-03-04T17:57:54.555 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5ffc66e1e7d0161-0065e60ba2-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.555 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:54.555 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.556 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.556 DEBUG:boto:path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.556 DEBUG:boto:auth_path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.556 DEBUG:boto:Method: GET 2024-03-04T17:57:54.556 DEBUG:boto:Path: /ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.556 DEBUG:boto:Data: 2024-03-04T17:57:54.556 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.556 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:54.556 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.556 DEBUG:boto:Params: {} 2024-03-04T17:57:54.556 DEBUG:boto:Token: None 2024-03-04T17:57:54.556 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/?versions 2024-03-04T17:57:54.557 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ= 2024-03-04T17:57:54.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ=', 'Content-Length': '0'} 2024-03-04T17:57:54.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ba3e494fb590221-0065e60ba2-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.559 DEBUG:boto:b'ppfahn-29999falseobjnulltrue2024-03-04T17:57:21.286Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:54.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:54.560 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:54.560 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.560 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.560 DEBUG:boto:path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.560 DEBUG:boto:auth_path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.560 DEBUG:boto:Method: GET 2024-03-04T17:57:54.560 DEBUG:boto:Path: /ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.560 DEBUG:boto:Data: 2024-03-04T17:57:54.560 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.560 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:54.561 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.561 DEBUG:boto:Params: {} 2024-03-04T17:57:54.561 DEBUG:boto:Token: None 2024-03-04T17:57:54.561 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/?versions 2024-03-04T17:57:54.561 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ= 2024-03-04T17:57:54.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ=', 'Content-Length': '0'} 2024-03-04T17:57:54.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f50accacdbdf7e7e-0065e60ba2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.566 DEBUG:boto:b'ppfahn-29999falseobjnulltrue2024-03-04T17:57:21.286Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:54.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:54.566 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.566 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.566 DEBUG:boto:path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.566 DEBUG:boto:auth_path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.566 DEBUG:boto:Method: GET 2024-03-04T17:57:54.566 DEBUG:boto:Path: /ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.567 DEBUG:boto:Data: 2024-03-04T17:57:54.567 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.567 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:54.567 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.567 DEBUG:boto:Params: {} 2024-03-04T17:57:54.567 DEBUG:boto:Token: None 2024-03-04T17:57:54.567 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/?versions 2024-03-04T17:57:54.567 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ= 2024-03-04T17:57:54.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ=', 'Content-Length': '0'} 2024-03-04T17:57:54.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d00fe9604921169-0065e60ba2-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.570 DEBUG:boto:b'ppfahn-29999falseobjnulltrue2024-03-04T17:57:21.286Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:54.571 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.571 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.571 DEBUG:boto:path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.571 DEBUG:boto:auth_path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.571 DEBUG:boto:Method: GET 2024-03-04T17:57:54.571 DEBUG:boto:Path: /ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.571 DEBUG:boto:Data: 2024-03-04T17:57:54.571 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.571 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:54.571 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.571 DEBUG:boto:Params: {} 2024-03-04T17:57:54.572 DEBUG:boto:Token: None 2024-03-04T17:57:54.572 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/?versions 2024-03-04T17:57:54.572 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ= 2024-03-04T17:57:54.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ=', 'Content-Length': '0'} 2024-03-04T17:57:54.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9a700f97fc8afd4-0065e60ba2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.620 DEBUG:boto:b'ppfahn-29999falseobjnulltrue2024-03-04T17:57:21.286Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:54.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:54.620 DEBUG:boto:path=/ppfahn-29/obj 2024-03-04T17:57:54.621 DEBUG:boto:auth_path=/ppfahn-29/obj 2024-03-04T17:57:54.621 DEBUG:boto:Method: HEAD 2024-03-04T17:57:54.621 DEBUG:boto:Path: /ppfahn-29/obj 2024-03-04T17:57:54.621 DEBUG:boto:Data: 2024-03-04T17:57:54.621 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.621 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:54.621 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.621 DEBUG:boto:Params: {} 2024-03-04T17:57:54.621 DEBUG:boto:Token: None 2024-03-04T17:57:54.621 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/obj 2024-03-04T17:57:54.621 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pVvyzEbrEc7VjF1mBnTTYHhIQsw= 2024-03-04T17:57:54.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pVvyzEbrEc7VjF1mBnTTYHhIQsw=', 'Content-Length': '0'} 2024-03-04T17:57:54.623 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000870b0b212fe57099-0065e60ba2-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.623 DEBUG:boto:path=/ppfahn-29/obj 2024-03-04T17:57:54.624 DEBUG:boto:auth_path=/ppfahn-29/obj 2024-03-04T17:57:54.624 DEBUG:boto:Method: HEAD 2024-03-04T17:57:54.624 DEBUG:boto:Path: /ppfahn-29/obj 2024-03-04T17:57:54.624 DEBUG:boto:Data: 2024-03-04T17:57:54.624 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.624 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:54.624 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.624 DEBUG:boto:Params: {} 2024-03-04T17:57:54.624 DEBUG:boto:Token: None 2024-03-04T17:57:54.624 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/obj 2024-03-04T17:57:54.625 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pVvyzEbrEc7VjF1mBnTTYHhIQsw= 2024-03-04T17:57:54.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pVvyzEbrEc7VjF1mBnTTYHhIQsw=', 'Content-Length': '0'} 2024-03-04T17:57:54.627 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000007d275a2cd086e0a0-0065e60ba2-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.627 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-29 zones={test-zone2, test-zone1} 2024-03-04T17:57:54.627 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-29 zones={test-zone3, test-zone1} 2024-03-04T17:57:54.627 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.628 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.628 DEBUG:boto:Method: HEAD 2024-03-04T17:57:54.628 DEBUG:boto:Path: /ppfahn-29/ 2024-03-04T17:57:54.628 DEBUG:boto:Data: 2024-03-04T17:57:54.628 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.628 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:54.628 DEBUG:boto:Port: 8001 2024-03-04T17:57:54.628 DEBUG:boto:Params: {} 2024-03-04T17:57:54.628 DEBUG:boto:Token: None 2024-03-04T17:57:54.628 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/ 2024-03-04T17:57:54.629 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l8nPEu+phulD0el8H1+yWCMtDN0= 2024-03-04T17:57:54.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l8nPEu+phulD0el8H1+yWCMtDN0=', 'Content-Length': '0'} 2024-03-04T17:57:54.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ffa60e485f5312d2-0065e60ba2-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.633 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.633 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.633 DEBUG:boto:Method: HEAD 2024-03-04T17:57:54.633 DEBUG:boto:Path: /ppfahn-29/ 2024-03-04T17:57:54.633 DEBUG:boto:Data: 2024-03-04T17:57:54.634 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.634 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:54.634 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.634 DEBUG:boto:Params: {} 2024-03-04T17:57:54.634 DEBUG:boto:Token: None 2024-03-04T17:57:54.634 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/ 2024-03-04T17:57:54.634 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l8nPEu+phulD0el8H1+yWCMtDN0= 2024-03-04T17:57:54.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l8nPEu+phulD0el8H1+yWCMtDN0=', 'Content-Length': '0'} 2024-03-04T17:57:54.639 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088a44b92122be5e7-0065e60ba2-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.639 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:54.639 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.639 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.639 DEBUG:boto:path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.639 DEBUG:boto:auth_path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.639 DEBUG:boto:Method: GET 2024-03-04T17:57:54.639 DEBUG:boto:Path: /ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.640 DEBUG:boto:Data: 2024-03-04T17:57:54.640 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.640 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:54.640 DEBUG:boto:Port: 8001 2024-03-04T17:57:54.640 DEBUG:boto:Params: {} 2024-03-04T17:57:54.640 DEBUG:boto:Token: None 2024-03-04T17:57:54.640 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/?versions 2024-03-04T17:57:54.640 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ= 2024-03-04T17:57:54.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ=', 'Content-Length': '0'} 2024-03-04T17:57:54.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007d2124b4215f459-0065e60ba2-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.688 DEBUG:boto:b'ppfahn-29999falseobjnulltrue2024-03-04T17:57:21.286Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:54.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:54.688 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:54.688 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.689 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.689 DEBUG:boto:path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.689 DEBUG:boto:auth_path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.689 DEBUG:boto:Method: GET 2024-03-04T17:57:54.689 DEBUG:boto:Path: /ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.689 DEBUG:boto:Data: 2024-03-04T17:57:54.689 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.689 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:54.689 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.689 DEBUG:boto:Params: {} 2024-03-04T17:57:54.689 DEBUG:boto:Token: None 2024-03-04T17:57:54.690 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/?versions 2024-03-04T17:57:54.690 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ= 2024-03-04T17:57:54.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ=', 'Content-Length': '0'} 2024-03-04T17:57:54.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081128d454da974e1-0065e60ba2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.695 DEBUG:boto:b'ppfahn-29999falseobjnulltrue2024-03-04T17:57:21.286Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:54.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:54.695 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.695 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.695 DEBUG:boto:path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.696 DEBUG:boto:auth_path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.696 DEBUG:boto:Method: GET 2024-03-04T17:57:54.696 DEBUG:boto:Path: /ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.696 DEBUG:boto:Data: 2024-03-04T17:57:54.696 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.696 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:54.696 DEBUG:boto:Port: 8001 2024-03-04T17:57:54.696 DEBUG:boto:Params: {} 2024-03-04T17:57:54.696 DEBUG:boto:Token: None 2024-03-04T17:57:54.696 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/?versions 2024-03-04T17:57:54.697 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ= 2024-03-04T17:57:54.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ=', 'Content-Length': '0'} 2024-03-04T17:57:54.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e2796c5799a0cc3-0065e60ba2-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.701 DEBUG:boto:b'ppfahn-29999falseobjnulltrue2024-03-04T17:57:21.286Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:54.702 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.702 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.702 DEBUG:boto:path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.702 DEBUG:boto:auth_path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.702 DEBUG:boto:Method: GET 2024-03-04T17:57:54.702 DEBUG:boto:Path: /ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.702 DEBUG:boto:Data: 2024-03-04T17:57:54.702 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.702 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:54.702 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.702 DEBUG:boto:Params: {} 2024-03-04T17:57:54.703 DEBUG:boto:Token: None 2024-03-04T17:57:54.703 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/?versions 2024-03-04T17:57:54.703 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ= 2024-03-04T17:57:54.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ=', 'Content-Length': '0'} 2024-03-04T17:57:54.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e781eee368d137dd-0065e60ba2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.708 DEBUG:boto:b'ppfahn-29999falseobjnulltrue2024-03-04T17:57:21.286Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:54.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:54.709 DEBUG:boto:path=/ppfahn-29/obj 2024-03-04T17:57:54.709 DEBUG:boto:auth_path=/ppfahn-29/obj 2024-03-04T17:57:54.709 DEBUG:boto:Method: HEAD 2024-03-04T17:57:54.709 DEBUG:boto:Path: /ppfahn-29/obj 2024-03-04T17:57:54.709 DEBUG:boto:Data: 2024-03-04T17:57:54.709 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.709 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:54.709 DEBUG:boto:Port: 8001 2024-03-04T17:57:54.709 DEBUG:boto:Params: {} 2024-03-04T17:57:54.709 DEBUG:boto:Token: None 2024-03-04T17:57:54.710 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/obj 2024-03-04T17:57:54.710 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pVvyzEbrEc7VjF1mBnTTYHhIQsw= 2024-03-04T17:57:54.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pVvyzEbrEc7VjF1mBnTTYHhIQsw=', 'Content-Length': '0'} 2024-03-04T17:57:54.712 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006172958dcda84f99-0065e60ba2-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.712 DEBUG:boto:path=/ppfahn-29/obj 2024-03-04T17:57:54.712 DEBUG:boto:auth_path=/ppfahn-29/obj 2024-03-04T17:57:54.712 DEBUG:boto:Method: HEAD 2024-03-04T17:57:54.712 DEBUG:boto:Path: /ppfahn-29/obj 2024-03-04T17:57:54.713 DEBUG:boto:Data: 2024-03-04T17:57:54.713 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.713 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:54.713 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.713 DEBUG:boto:Params: {} 2024-03-04T17:57:54.713 DEBUG:boto:Token: None 2024-03-04T17:57:54.713 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/obj 2024-03-04T17:57:54.713 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pVvyzEbrEc7VjF1mBnTTYHhIQsw= 2024-03-04T17:57:54.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pVvyzEbrEc7VjF1mBnTTYHhIQsw=', 'Content-Length': '0'} 2024-03-04T17:57:54.715 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000023da7a6c29cd4f4a-0065e60ba2-4349-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.716 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-29 zones={test-zone3, test-zone1} 2024-03-04T17:57:54.716 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ppfahn-29 zones={test-zone3, test-zone2} 2024-03-04T17:57:54.716 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.716 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.716 DEBUG:boto:Method: HEAD 2024-03-04T17:57:54.716 DEBUG:boto:Path: /ppfahn-29/ 2024-03-04T17:57:54.716 DEBUG:boto:Data: 2024-03-04T17:57:54.716 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.716 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:54.716 DEBUG:boto:Port: 8001 2024-03-04T17:57:54.716 DEBUG:boto:Params: {} 2024-03-04T17:57:54.717 DEBUG:boto:Token: None 2024-03-04T17:57:54.717 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/ 2024-03-04T17:57:54.717 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l8nPEu+phulD0el8H1+yWCMtDN0= 2024-03-04T17:57:54.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l8nPEu+phulD0el8H1+yWCMtDN0=', 'Content-Length': '0'} 2024-03-04T17:57:54.721 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000588c2f42d4b653ca-0065e60ba2-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.721 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.722 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.722 DEBUG:boto:Method: HEAD 2024-03-04T17:57:54.722 DEBUG:boto:Path: /ppfahn-29/ 2024-03-04T17:57:54.722 DEBUG:boto:Data: 2024-03-04T17:57:54.722 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.722 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:54.722 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.722 DEBUG:boto:Params: {} 2024-03-04T17:57:54.722 DEBUG:boto:Token: None 2024-03-04T17:57:54.722 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/ 2024-03-04T17:57:54.722 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:l8nPEu+phulD0el8H1+yWCMtDN0= 2024-03-04T17:57:54.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l8nPEu+phulD0el8H1+yWCMtDN0=', 'Content-Length': '0'} 2024-03-04T17:57:54.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000041859cca4fde076-0065e60ba2-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.727 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T17:57:54.727 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.727 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.727 DEBUG:boto:path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.728 DEBUG:boto:auth_path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.728 DEBUG:boto:Method: GET 2024-03-04T17:57:54.728 DEBUG:boto:Path: /ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.728 DEBUG:boto:Data: 2024-03-04T17:57:54.728 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.728 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:54.728 DEBUG:boto:Port: 8001 2024-03-04T17:57:54.728 DEBUG:boto:Params: {} 2024-03-04T17:57:54.728 DEBUG:boto:Token: None 2024-03-04T17:57:54.728 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/?versions 2024-03-04T17:57:54.729 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ= 2024-03-04T17:57:54.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ=', 'Content-Length': '0'} 2024-03-04T17:57:54.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000037303acda22247-0065e60ba2-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.776 DEBUG:boto:b'ppfahn-29999falseobjnulltrue2024-03-04T17:57:21.286Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:54.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:54.776 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T17:57:54.776 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.777 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.777 DEBUG:boto:path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.777 DEBUG:boto:auth_path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.777 DEBUG:boto:Method: GET 2024-03-04T17:57:54.777 DEBUG:boto:Path: /ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.777 DEBUG:boto:Data: 2024-03-04T17:57:54.777 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.777 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:54.777 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.777 DEBUG:boto:Params: {} 2024-03-04T17:57:54.777 DEBUG:boto:Token: None 2024-03-04T17:57:54.778 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/?versions 2024-03-04T17:57:54.778 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ= 2024-03-04T17:57:54.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ=', 'Content-Length': '0'} 2024-03-04T17:57:54.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6784a3bccc4f1b3-0065e60ba2-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.783 DEBUG:boto:b'ppfahn-29999falseobjnulltrue2024-03-04T17:57:21.286Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:54.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T17:57:54.784 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.784 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.784 DEBUG:boto:path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.784 DEBUG:boto:auth_path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.784 DEBUG:boto:Method: GET 2024-03-04T17:57:54.784 DEBUG:boto:Path: /ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.784 DEBUG:boto:Data: 2024-03-04T17:57:54.784 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.784 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:54.784 DEBUG:boto:Port: 8001 2024-03-04T17:57:54.784 DEBUG:boto:Params: {} 2024-03-04T17:57:54.785 DEBUG:boto:Token: None 2024-03-04T17:57:54.785 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/?versions 2024-03-04T17:57:54.785 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ= 2024-03-04T17:57:54.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ=', 'Content-Length': '0'} 2024-03-04T17:57:54.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005def9e44c542f93-0065e60ba2-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.790 DEBUG:boto:b'ppfahn-29999falseobjnulltrue2024-03-04T17:57:21.286Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:54.790 DEBUG:boto:path=/ppfahn-29/ 2024-03-04T17:57:54.790 DEBUG:boto:auth_path=/ppfahn-29/ 2024-03-04T17:57:54.790 DEBUG:boto:path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.790 DEBUG:boto:auth_path=/ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.791 DEBUG:boto:Method: GET 2024-03-04T17:57:54.791 DEBUG:boto:Path: /ppfahn-29/?versions&max-keys=999 2024-03-04T17:57:54.791 DEBUG:boto:Data: 2024-03-04T17:57:54.791 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.791 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:54.791 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.791 DEBUG:boto:Params: {} 2024-03-04T17:57:54.791 DEBUG:boto:Token: None 2024-03-04T17:57:54.791 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/?versions 2024-03-04T17:57:54.791 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ= 2024-03-04T17:57:54.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IBLpFmD0KZH9gXKfgcKaKyDR2SQ=', 'Content-Length': '0'} 2024-03-04T17:57:54.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1f7d6e4049e31c4-0065e60ba2-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.797 DEBUG:boto:b'ppfahn-29999falseobjnulltrue2024-03-04T17:57:21.286Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T17:57:54.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T17:57:54.797 DEBUG:boto:path=/ppfahn-29/obj 2024-03-04T17:57:54.797 DEBUG:boto:auth_path=/ppfahn-29/obj 2024-03-04T17:57:54.797 DEBUG:boto:Method: HEAD 2024-03-04T17:57:54.797 DEBUG:boto:Path: /ppfahn-29/obj 2024-03-04T17:57:54.797 DEBUG:boto:Data: 2024-03-04T17:57:54.797 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.797 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:54.798 DEBUG:boto:Port: 8001 2024-03-04T17:57:54.798 DEBUG:boto:Params: {} 2024-03-04T17:57:54.798 DEBUG:boto:Token: None 2024-03-04T17:57:54.798 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/obj 2024-03-04T17:57:54.798 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pVvyzEbrEc7VjF1mBnTTYHhIQsw= 2024-03-04T17:57:54.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pVvyzEbrEc7VjF1mBnTTYHhIQsw=', 'Content-Length': '0'} 2024-03-04T17:57:54.800 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000838ee148a4978af1-0065e60ba2-4356-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.800 DEBUG:boto:path=/ppfahn-29/obj 2024-03-04T17:57:54.801 DEBUG:boto:auth_path=/ppfahn-29/obj 2024-03-04T17:57:54.801 DEBUG:boto:Method: HEAD 2024-03-04T17:57:54.801 DEBUG:boto:Path: /ppfahn-29/obj 2024-03-04T17:57:54.801 DEBUG:boto:Data: 2024-03-04T17:57:54.801 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.801 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:54.801 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.801 DEBUG:boto:Params: {} 2024-03-04T17:57:54.801 DEBUG:boto:Token: None 2024-03-04T17:57:54.801 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-29/obj 2024-03-04T17:57:54.801 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pVvyzEbrEc7VjF1mBnTTYHhIQsw= 2024-03-04T17:57:54.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pVvyzEbrEc7VjF1mBnTTYHhIQsw=', 'Content-Length': '0'} 2024-03-04T17:57:54.804 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000015c2b970ef2ec5a2-0065e60ba2-4306-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.804 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ppfahn-29 zones={test-zone3, test-zone2} 2024-03-04T17:57:54.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2024-03-04T17:57:54.805 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-30 2024-03-04T17:57:54.805 DEBUG:boto:path=/ppfahn-30/ 2024-03-04T17:57:54.805 DEBUG:boto:auth_path=/ppfahn-30/ 2024-03-04T17:57:54.805 DEBUG:boto:Method: PUT 2024-03-04T17:57:54.805 DEBUG:boto:Path: /ppfahn-30/ 2024-03-04T17:57:54.806 DEBUG:boto:Data: 2024-03-04T17:57:54.806 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.806 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:54.806 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.806 DEBUG:boto:Params: {} 2024-03-04T17:57:54.806 DEBUG:boto:Token: None 2024-03-04T17:57:54.806 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-30/ 2024-03-04T17:57:54.806 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8HV53ZjKuJuaK+lxcM4jhes+DpA= 2024-03-04T17:57:54.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8HV53ZjKuJuaK+lxcM4jhes+DpA=', 'Content-Length': '0'} 2024-03-04T17:57:54.838 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004d40ec620dbbd02a-0065e60ba2-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.838 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-31 2024-03-04T17:57:54.838 DEBUG:boto:path=/ppfahn-31/ 2024-03-04T17:57:54.838 DEBUG:boto:auth_path=/ppfahn-31/ 2024-03-04T17:57:54.838 DEBUG:boto:Method: PUT 2024-03-04T17:57:54.839 DEBUG:boto:Path: /ppfahn-31/ 2024-03-04T17:57:54.839 DEBUG:boto:Data: 2024-03-04T17:57:54.839 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.839 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:54.839 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.839 DEBUG:boto:Params: {} 2024-03-04T17:57:54.839 DEBUG:boto:Token: None 2024-03-04T17:57:54.839 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-31/ 2024-03-04T17:57:54.839 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ErkxYKv83jVkQ1a1oG95eExDayg= 2024-03-04T17:57:54.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ErkxYKv83jVkQ1a1oG95eExDayg=', 'Content-Length': '0'} 2024-03-04T17:57:54.895 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f1f54b2f7664d48d-0065e60ba2-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.895 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-32 2024-03-04T17:57:54.896 DEBUG:boto:path=/ppfahn-32/ 2024-03-04T17:57:54.896 DEBUG:boto:auth_path=/ppfahn-32/ 2024-03-04T17:57:54.896 DEBUG:boto:Method: PUT 2024-03-04T17:57:54.896 DEBUG:boto:Path: /ppfahn-32/ 2024-03-04T17:57:54.896 DEBUG:boto:Data: 2024-03-04T17:57:54.896 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.896 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:54.896 DEBUG:boto:Port: 8001 2024-03-04T17:57:54.896 DEBUG:boto:Params: {} 2024-03-04T17:57:54.896 DEBUG:boto:Token: None 2024-03-04T17:57:54.897 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-32/ 2024-03-04T17:57:54.897 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dSnfjoZOrogR5IWRccfVLcKVC+A= 2024-03-04T17:57:54.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dSnfjoZOrogR5IWRccfVLcKVC+A=', 'Content-Length': '0'} 2024-03-04T17:57:54.957 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e4be6e8d91a21a2b-0065e60ba2-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.957 DEBUG:boto:path=/ppfahn-30/ 2024-03-04T17:57:54.957 DEBUG:boto:auth_path=/ppfahn-30/ 2024-03-04T17:57:54.957 DEBUG:boto:path=/ppfahn-30/?versioning 2024-03-04T17:57:54.957 DEBUG:boto:auth_path=/ppfahn-30/?versioning 2024-03-04T17:57:54.957 DEBUG:boto:Method: PUT 2024-03-04T17:57:54.958 DEBUG:boto:Path: /ppfahn-30/?versioning 2024-03-04T17:57:54.958 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:57:54.958 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.958 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:54.958 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.958 DEBUG:boto:Params: {} 2024-03-04T17:57:54.958 DEBUG:boto:Token: None 2024-03-04T17:57:54.958 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-30/?versioning 2024-03-04T17:57:54.958 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:P6bkC6k7E6hAT07WXVfjxieb9aA= 2024-03-04T17:57:54.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:P6bkC6k7E6hAT07WXVfjxieb9aA=', 'Content-Length': '228'} 2024-03-04T17:57:54.971 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003751fdd9f29aab0e-0065e60ba2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.971 DEBUG:boto:path=/ppfahn-30/ 2024-03-04T17:57:54.971 DEBUG:boto:auth_path=/ppfahn-30/ 2024-03-04T17:57:54.971 DEBUG:boto:path=/ppfahn-30/?versioning 2024-03-04T17:57:54.971 DEBUG:boto:auth_path=/ppfahn-30/?versioning 2024-03-04T17:57:54.971 DEBUG:boto:Method: GET 2024-03-04T17:57:54.971 DEBUG:boto:Path: /ppfahn-30/?versioning 2024-03-04T17:57:54.971 DEBUG:boto:Data: 2024-03-04T17:57:54.971 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.971 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:54.971 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.972 DEBUG:boto:Params: {} 2024-03-04T17:57:54.972 DEBUG:boto:Token: None 2024-03-04T17:57:54.972 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-30/?versioning 2024-03-04T17:57:54.972 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:si+8pCqJKn+eJGsRjGyrDrfmQXc= 2024-03-04T17:57:54.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:si+8pCqJKn+eJGsRjGyrDrfmQXc=', 'Content-Length': '0'} 2024-03-04T17:57:54.974 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e4046ae9ac002004-0065e60ba2-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '192'), ('Date', 'Mon, 04 Mar 2024 17:57:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:54.974 DEBUG:boto:EnabledDisabled 2024-03-04T17:57:54.974 DEBUG:boto:path=/ppfahn-31/ 2024-03-04T17:57:54.974 DEBUG:boto:auth_path=/ppfahn-31/ 2024-03-04T17:57:54.974 DEBUG:boto:path=/ppfahn-31/?versioning 2024-03-04T17:57:54.975 DEBUG:boto:auth_path=/ppfahn-31/?versioning 2024-03-04T17:57:54.975 DEBUG:boto:Method: PUT 2024-03-04T17:57:54.975 DEBUG:boto:Path: /ppfahn-31/?versioning 2024-03-04T17:57:54.975 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:57:54.975 DEBUG:boto:Headers: {} 2024-03-04T17:57:54.975 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:54.975 DEBUG:boto:Port: 8000 2024-03-04T17:57:54.975 DEBUG:boto:Params: {} 2024-03-04T17:57:54.975 DEBUG:boto:Token: None 2024-03-04T17:57:54.975 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:54 GMT /ppfahn-31/?versioning 2024-03-04T17:57:54.975 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ykMEWlb48M9IAF/B2bURWUWJyB0= 2024-03-04T17:57:54.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:54 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ykMEWlb48M9IAF/B2bURWUWJyB0=', 'Content-Length': '228'} 2024-03-04T17:57:55.002 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ada27bce5655347c-0065e60ba2-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.002 DEBUG:boto:path=/ppfahn-31/ 2024-03-04T17:57:55.002 DEBUG:boto:auth_path=/ppfahn-31/ 2024-03-04T17:57:55.003 DEBUG:boto:path=/ppfahn-31/?versioning 2024-03-04T17:57:55.003 DEBUG:boto:auth_path=/ppfahn-31/?versioning 2024-03-04T17:57:55.003 DEBUG:boto:Method: GET 2024-03-04T17:57:55.003 DEBUG:boto:Path: /ppfahn-31/?versioning 2024-03-04T17:57:55.003 DEBUG:boto:Data: 2024-03-04T17:57:55.003 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.003 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:55.003 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.003 DEBUG:boto:Params: {} 2024-03-04T17:57:55.003 DEBUG:boto:Token: None 2024-03-04T17:57:55.003 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-31/?versioning 2024-03-04T17:57:55.003 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TZb3/5qAJAWKg6lr4rortN1UIjo= 2024-03-04T17:57:55.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZb3/5qAJAWKg6lr4rortN1UIjo=', 'Content-Length': '0'} 2024-03-04T17:57:55.005 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000328e826989ee6626-0065e60ba3-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '192'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.005 DEBUG:boto:EnabledDisabled 2024-03-04T17:57:55.005 DEBUG:boto:path=/ppfahn-32/ 2024-03-04T17:57:55.005 DEBUG:boto:auth_path=/ppfahn-32/ 2024-03-04T17:57:55.006 DEBUG:boto:path=/ppfahn-32/?versioning 2024-03-04T17:57:55.006 DEBUG:boto:auth_path=/ppfahn-32/?versioning 2024-03-04T17:57:55.006 DEBUG:boto:Method: PUT 2024-03-04T17:57:55.006 DEBUG:boto:Path: /ppfahn-32/?versioning 2024-03-04T17:57:55.006 DEBUG:boto:Data: Enabled Disabled 2024-03-04T17:57:55.006 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.006 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:55.006 DEBUG:boto:Port: 8001 2024-03-04T17:57:55.006 DEBUG:boto:Params: {} 2024-03-04T17:57:55.006 DEBUG:boto:Token: None 2024-03-04T17:57:55.006 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-32/?versioning 2024-03-04T17:57:55.007 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:voHhO45nHvkPbkbANgkplESUtEQ= 2024-03-04T17:57:55.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:voHhO45nHvkPbkbANgkplESUtEQ=', 'Content-Length': '228'} 2024-03-04T17:57:55.019 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005a7f6ee55e96e243-0065e60ba3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.020 DEBUG:boto:path=/ppfahn-32/ 2024-03-04T17:57:55.020 DEBUG:boto:auth_path=/ppfahn-32/ 2024-03-04T17:57:55.020 DEBUG:boto:path=/ppfahn-32/?versioning 2024-03-04T17:57:55.020 DEBUG:boto:auth_path=/ppfahn-32/?versioning 2024-03-04T17:57:55.020 DEBUG:boto:Method: GET 2024-03-04T17:57:55.020 DEBUG:boto:Path: /ppfahn-32/?versioning 2024-03-04T17:57:55.020 DEBUG:boto:Data: 2024-03-04T17:57:55.020 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.020 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:55.020 DEBUG:boto:Port: 8001 2024-03-04T17:57:55.021 DEBUG:boto:Params: {} 2024-03-04T17:57:55.021 DEBUG:boto:Token: None 2024-03-04T17:57:55.021 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-32/?versioning 2024-03-04T17:57:55.021 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3FrLIkNFLrHDp6nkJ0zZZTy+Yo4= 2024-03-04T17:57:55.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3FrLIkNFLrHDp6nkJ0zZZTy+Yo4=', 'Content-Length': '0'} 2024-03-04T17:57:55.022 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003a7a79ca93ba9350-0065e60ba3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '192'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.064 DEBUG:boto:EnabledDisabled 2024-03-04T17:57:55.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2024-03-04T17:57:55.065 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-33 2024-03-04T17:57:55.065 DEBUG:boto:path=/ppfahn-33/ 2024-03-04T17:57:55.065 DEBUG:boto:auth_path=/ppfahn-33/ 2024-03-04T17:57:55.065 DEBUG:boto:Method: PUT 2024-03-04T17:57:55.065 DEBUG:boto:Path: /ppfahn-33/ 2024-03-04T17:57:55.065 DEBUG:boto:Data: 2024-03-04T17:57:55.065 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.065 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:55.065 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.066 DEBUG:boto:Params: {} 2024-03-04T17:57:55.066 DEBUG:boto:Token: None 2024-03-04T17:57:55.066 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-33/ 2024-03-04T17:57:55.066 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:f8ctKY31n9Fwdva7PmoyHZ8LMdg= 2024-03-04T17:57:55.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:f8ctKY31n9Fwdva7PmoyHZ8LMdg=', 'Content-Length': '0'} 2024-03-04T17:57:55.099 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f23cdc572a5c00c1-0065e60ba3-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.099 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-34 2024-03-04T17:57:55.099 DEBUG:boto:path=/ppfahn-34/ 2024-03-04T17:57:55.099 DEBUG:boto:auth_path=/ppfahn-34/ 2024-03-04T17:57:55.099 DEBUG:boto:Method: PUT 2024-03-04T17:57:55.099 DEBUG:boto:Path: /ppfahn-34/ 2024-03-04T17:57:55.099 DEBUG:boto:Data: 2024-03-04T17:57:55.099 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.099 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:55.099 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.099 DEBUG:boto:Params: {} 2024-03-04T17:57:55.100 DEBUG:boto:Token: None 2024-03-04T17:57:55.100 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-34/ 2024-03-04T17:57:55.100 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0Kyu1DVaGxC76j20SIgvcyO5I1k= 2024-03-04T17:57:55.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0Kyu1DVaGxC76j20SIgvcyO5I1k=', 'Content-Length': '0'} 2024-03-04T17:57:55.162 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ecff3a331ca2d8d-0065e60ba3-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.162 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-35 2024-03-04T17:57:55.162 DEBUG:boto:path=/ppfahn-35/ 2024-03-04T17:57:55.162 DEBUG:boto:auth_path=/ppfahn-35/ 2024-03-04T17:57:55.162 DEBUG:boto:Method: PUT 2024-03-04T17:57:55.162 DEBUG:boto:Path: /ppfahn-35/ 2024-03-04T17:57:55.162 DEBUG:boto:Data: 2024-03-04T17:57:55.162 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.162 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:55.163 DEBUG:boto:Port: 8001 2024-03-04T17:57:55.163 DEBUG:boto:Params: {} 2024-03-04T17:57:55.163 DEBUG:boto:Token: None 2024-03-04T17:57:55.163 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-35/ 2024-03-04T17:57:55.163 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:snQe9O5aW/+7ZaaGa6sU0CSCRFU= 2024-03-04T17:57:55.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:snQe9O5aW/+7ZaaGa6sU0CSCRFU=', 'Content-Length': '0'} 2024-03-04T17:57:55.260 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000029ccdd4a6a75c535-0065e60ba3-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.261 DEBUG:boto:path=/ppfahn-33/ 2024-03-04T17:57:55.261 DEBUG:boto:auth_path=/ppfahn-33/ 2024-03-04T17:57:55.261 DEBUG:boto:path=/ppfahn-33/?acl 2024-03-04T17:57:55.261 DEBUG:boto:auth_path=/ppfahn-33/?acl 2024-03-04T17:57:55.261 DEBUG:boto:Method: GET 2024-03-04T17:57:55.261 DEBUG:boto:Path: /ppfahn-33/?acl 2024-03-04T17:57:55.261 DEBUG:boto:Data: 2024-03-04T17:57:55.261 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.261 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:55.261 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.261 DEBUG:boto:Params: {} 2024-03-04T17:57:55.262 DEBUG:boto:Token: None 2024-03-04T17:57:55.262 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-33/?acl 2024-03-04T17:57:55.262 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lHJS19SNDTmmPYpslzQbiyTx7bs= 2024-03-04T17:57:55.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lHJS19SNDTmmPYpslzQbiyTx7bs=', 'Content-Length': '0'} 2024-03-04T17:57:55.263 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000060c18527ff15a10d-0065e60ba3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '489'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.264 DEBUG:boto:path=/ppfahn-33/ 2024-03-04T17:57:55.264 DEBUG:boto:auth_path=/ppfahn-33/ 2024-03-04T17:57:55.264 DEBUG:boto:path=/ppfahn-33/?acl 2024-03-04T17:57:55.264 DEBUG:boto:auth_path=/ppfahn-33/?acl 2024-03-04T17:57:55.264 DEBUG:boto:Method: PUT 2024-03-04T17:57:55.264 DEBUG:boto:Path: /ppfahn-33/?acl 2024-03-04T17:57:55.264 DEBUG:boto:Data: 2024-03-04T17:57:55.264 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T17:57:55.264 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:55.264 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.265 DEBUG:boto:Params: {} 2024-03-04T17:57:55.265 DEBUG:boto:Token: None 2024-03-04T17:57:55.265 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:55 GMT x-amz-acl:public-read /ppfahn-33/?acl 2024-03-04T17:57:55.265 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:YB/FLbBmPqgNIwdxGQK4cGOTYts= 2024-03-04T17:57:55.265 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:YB/FLbBmPqgNIwdxGQK4cGOTYts=', 'Content-Length': '0'} 2024-03-04T17:57:55.271 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000b0bce783c1163aa-0065e60ba3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.271 DEBUG:boto:path=/ppfahn-33/ 2024-03-04T17:57:55.272 DEBUG:boto:auth_path=/ppfahn-33/ 2024-03-04T17:57:55.272 DEBUG:boto:path=/ppfahn-33/?acl 2024-03-04T17:57:55.272 DEBUG:boto:auth_path=/ppfahn-33/?acl 2024-03-04T17:57:55.272 DEBUG:boto:Method: GET 2024-03-04T17:57:55.272 DEBUG:boto:Path: /ppfahn-33/?acl 2024-03-04T17:57:55.272 DEBUG:boto:Data: 2024-03-04T17:57:55.272 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.272 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:55.272 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.272 DEBUG:boto:Params: {} 2024-03-04T17:57:55.272 DEBUG:boto:Token: None 2024-03-04T17:57:55.273 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-33/?acl 2024-03-04T17:57:55.273 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lHJS19SNDTmmPYpslzQbiyTx7bs= 2024-03-04T17:57:55.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lHJS19SNDTmmPYpslzQbiyTx7bs=', 'Content-Length': '0'} 2024-03-04T17:57:55.274 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005b59ed1dec5648f8-0065e60ba3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '681'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.275 DEBUG:boto:path=/ppfahn-34/ 2024-03-04T17:57:55.275 DEBUG:boto:auth_path=/ppfahn-34/ 2024-03-04T17:57:55.275 DEBUG:boto:path=/ppfahn-34/?acl 2024-03-04T17:57:55.275 DEBUG:boto:auth_path=/ppfahn-34/?acl 2024-03-04T17:57:55.275 DEBUG:boto:Method: GET 2024-03-04T17:57:55.275 DEBUG:boto:Path: /ppfahn-34/?acl 2024-03-04T17:57:55.275 DEBUG:boto:Data: 2024-03-04T17:57:55.275 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.275 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:55.275 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.276 DEBUG:boto:Params: {} 2024-03-04T17:57:55.276 DEBUG:boto:Token: None 2024-03-04T17:57:55.276 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-34/?acl 2024-03-04T17:57:55.276 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ty7Ghcqd0IqMSQULhtv8bJrK640= 2024-03-04T17:57:55.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ty7Ghcqd0IqMSQULhtv8bJrK640=', 'Content-Length': '0'} 2024-03-04T17:57:55.278 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f117cf3c5ff1a66c-0065e60ba3-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '489'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.278 DEBUG:boto:path=/ppfahn-34/ 2024-03-04T17:57:55.278 DEBUG:boto:auth_path=/ppfahn-34/ 2024-03-04T17:57:55.278 DEBUG:boto:path=/ppfahn-34/?acl 2024-03-04T17:57:55.278 DEBUG:boto:auth_path=/ppfahn-34/?acl 2024-03-04T17:57:55.278 DEBUG:boto:Method: PUT 2024-03-04T17:57:55.278 DEBUG:boto:Path: /ppfahn-34/?acl 2024-03-04T17:57:55.279 DEBUG:boto:Data: 2024-03-04T17:57:55.279 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T17:57:55.279 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:55.279 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.279 DEBUG:boto:Params: {} 2024-03-04T17:57:55.279 DEBUG:boto:Token: None 2024-03-04T17:57:55.279 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:55 GMT x-amz-acl:public-read /ppfahn-34/?acl 2024-03-04T17:57:55.279 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:n/XPNLiEZrtY6eb8kUg7J+XkDFU= 2024-03-04T17:57:55.279 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:n/XPNLiEZrtY6eb8kUg7J+XkDFU=', 'Content-Length': '0'} 2024-03-04T17:57:55.294 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009ca3b0240a75c29e-0065e60ba3-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.294 DEBUG:boto:path=/ppfahn-34/ 2024-03-04T17:57:55.294 DEBUG:boto:auth_path=/ppfahn-34/ 2024-03-04T17:57:55.294 DEBUG:boto:path=/ppfahn-34/?acl 2024-03-04T17:57:55.295 DEBUG:boto:auth_path=/ppfahn-34/?acl 2024-03-04T17:57:55.295 DEBUG:boto:Method: GET 2024-03-04T17:57:55.295 DEBUG:boto:Path: /ppfahn-34/?acl 2024-03-04T17:57:55.295 DEBUG:boto:Data: 2024-03-04T17:57:55.295 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.295 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:55.295 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.295 DEBUG:boto:Params: {} 2024-03-04T17:57:55.295 DEBUG:boto:Token: None 2024-03-04T17:57:55.295 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-34/?acl 2024-03-04T17:57:55.295 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ty7Ghcqd0IqMSQULhtv8bJrK640= 2024-03-04T17:57:55.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ty7Ghcqd0IqMSQULhtv8bJrK640=', 'Content-Length': '0'} 2024-03-04T17:57:55.297 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004464cd255a1e8871-0065e60ba3-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '681'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.298 DEBUG:boto:path=/ppfahn-35/ 2024-03-04T17:57:55.298 DEBUG:boto:auth_path=/ppfahn-35/ 2024-03-04T17:57:55.298 DEBUG:boto:path=/ppfahn-35/?acl 2024-03-04T17:57:55.298 DEBUG:boto:auth_path=/ppfahn-35/?acl 2024-03-04T17:57:55.298 DEBUG:boto:Method: GET 2024-03-04T17:57:55.298 DEBUG:boto:Path: /ppfahn-35/?acl 2024-03-04T17:57:55.298 DEBUG:boto:Data: 2024-03-04T17:57:55.298 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.298 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:55.298 DEBUG:boto:Port: 8001 2024-03-04T17:57:55.299 DEBUG:boto:Params: {} 2024-03-04T17:57:55.299 DEBUG:boto:Token: None 2024-03-04T17:57:55.299 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-35/?acl 2024-03-04T17:57:55.299 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZQwh/YGpsac7Nl8HojujENU8Hug= 2024-03-04T17:57:55.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZQwh/YGpsac7Nl8HojujENU8Hug=', 'Content-Length': '0'} 2024-03-04T17:57:55.300 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000006ea8a7604a6b0da-0065e60ba3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '489'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.301 DEBUG:boto:path=/ppfahn-35/ 2024-03-04T17:57:55.301 DEBUG:boto:auth_path=/ppfahn-35/ 2024-03-04T17:57:55.301 DEBUG:boto:path=/ppfahn-35/?acl 2024-03-04T17:57:55.301 DEBUG:boto:auth_path=/ppfahn-35/?acl 2024-03-04T17:57:55.301 DEBUG:boto:Method: PUT 2024-03-04T17:57:55.301 DEBUG:boto:Path: /ppfahn-35/?acl 2024-03-04T17:57:55.301 DEBUG:boto:Data: 2024-03-04T17:57:55.301 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T17:57:55.301 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:55.302 DEBUG:boto:Port: 8001 2024-03-04T17:57:55.302 DEBUG:boto:Params: {} 2024-03-04T17:57:55.302 DEBUG:boto:Token: None 2024-03-04T17:57:55.302 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:55 GMT x-amz-acl:public-read /ppfahn-35/?acl 2024-03-04T17:57:55.302 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:IZmyAP4WkMJSPvxf9eiwXl9DkSM= 2024-03-04T17:57:55.302 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IZmyAP4WkMJSPvxf9eiwXl9DkSM=', 'Content-Length': '0'} 2024-03-04T17:57:55.315 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003942a976a9fd9f6e-0065e60ba3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.315 DEBUG:boto:path=/ppfahn-35/ 2024-03-04T17:57:55.316 DEBUG:boto:auth_path=/ppfahn-35/ 2024-03-04T17:57:55.316 DEBUG:boto:path=/ppfahn-35/?acl 2024-03-04T17:57:55.316 DEBUG:boto:auth_path=/ppfahn-35/?acl 2024-03-04T17:57:55.316 DEBUG:boto:Method: GET 2024-03-04T17:57:55.316 DEBUG:boto:Path: /ppfahn-35/?acl 2024-03-04T17:57:55.316 DEBUG:boto:Data: 2024-03-04T17:57:55.316 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.316 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:55.316 DEBUG:boto:Port: 8001 2024-03-04T17:57:55.316 DEBUG:boto:Params: {} 2024-03-04T17:57:55.316 DEBUG:boto:Token: None 2024-03-04T17:57:55.317 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-35/?acl 2024-03-04T17:57:55.317 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ZQwh/YGpsac7Nl8HojujENU8Hug= 2024-03-04T17:57:55.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZQwh/YGpsac7Nl8HojujENU8Hug=', 'Content-Length': '0'} 2024-03-04T17:57:55.318 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c6bb1e33328e3013-0065e60ba3-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '681'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2024-03-04T17:57:55.361 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-36 2024-03-04T17:57:55.361 DEBUG:boto:path=/ppfahn-36/ 2024-03-04T17:57:55.361 DEBUG:boto:auth_path=/ppfahn-36/ 2024-03-04T17:57:55.361 DEBUG:boto:Method: PUT 2024-03-04T17:57:55.361 DEBUG:boto:Path: /ppfahn-36/ 2024-03-04T17:57:55.361 DEBUG:boto:Data: 2024-03-04T17:57:55.361 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.361 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:55.361 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.361 DEBUG:boto:Params: {} 2024-03-04T17:57:55.362 DEBUG:boto:Token: None 2024-03-04T17:57:55.362 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-36/ 2024-03-04T17:57:55.362 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:57fSj5TBd4TZg3GWy8S31ajQXI4= 2024-03-04T17:57:55.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:57fSj5TBd4TZg3GWy8S31ajQXI4=', 'Content-Length': '0'} 2024-03-04T17:57:55.392 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003bac973211430af5-0065e60ba3-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.392 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-37 2024-03-04T17:57:55.393 DEBUG:boto:path=/ppfahn-37/ 2024-03-04T17:57:55.393 DEBUG:boto:auth_path=/ppfahn-37/ 2024-03-04T17:57:55.393 DEBUG:boto:Method: PUT 2024-03-04T17:57:55.393 DEBUG:boto:Path: /ppfahn-37/ 2024-03-04T17:57:55.393 DEBUG:boto:Data: 2024-03-04T17:57:55.393 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.393 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:55.393 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.393 DEBUG:boto:Params: {} 2024-03-04T17:57:55.393 DEBUG:boto:Token: None 2024-03-04T17:57:55.393 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-37/ 2024-03-04T17:57:55.394 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:udtL7FMbVXzgz/bwbLnle99SQ/w= 2024-03-04T17:57:55.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:udtL7FMbVXzgz/bwbLnle99SQ/w=', 'Content-Length': '0'} 2024-03-04T17:57:55.455 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ada4273eb18b12dd-0065e60ba3-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.456 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-38 2024-03-04T17:57:55.456 DEBUG:boto:path=/ppfahn-38/ 2024-03-04T17:57:55.456 DEBUG:boto:auth_path=/ppfahn-38/ 2024-03-04T17:57:55.456 DEBUG:boto:Method: PUT 2024-03-04T17:57:55.456 DEBUG:boto:Path: /ppfahn-38/ 2024-03-04T17:57:55.456 DEBUG:boto:Data: 2024-03-04T17:57:55.456 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.456 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:55.456 DEBUG:boto:Port: 8001 2024-03-04T17:57:55.456 DEBUG:boto:Params: {} 2024-03-04T17:57:55.456 DEBUG:boto:Token: None 2024-03-04T17:57:55.457 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-38/ 2024-03-04T17:57:55.457 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cTBsOCuHTOL21gvmtebkPyYvi/g= 2024-03-04T17:57:55.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cTBsOCuHTOL21gvmtebkPyYvi/g=', 'Content-Length': '0'} 2024-03-04T17:57:55.515 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ad7ae238075d420-0065e60ba3-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.515 DEBUG:boto:path=/ppfahn-36/ 2024-03-04T17:57:55.515 DEBUG:boto:auth_path=/ppfahn-36/ 2024-03-04T17:57:55.516 DEBUG:boto:path=/ppfahn-36/?cors 2024-03-04T17:57:55.516 DEBUG:boto:auth_path=/ppfahn-36/?cors 2024-03-04T17:57:55.516 DEBUG:boto:Method: PUT 2024-03-04T17:57:55.516 DEBUG:boto:Path: /ppfahn-36/?cors 2024-03-04T17:57:55.516 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-03-04T17:57:55.516 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-03-04T17:57:55.516 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:55.516 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.516 DEBUG:boto:Params: {} 2024-03-04T17:57:55.516 DEBUG:boto:Token: None 2024-03-04T17:57:55.516 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-36/?cors 2024-03-04T17:57:55.517 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:B/Nm8knRdTfn3VDWcbdxadoK/bA= 2024-03-04T17:57:55.517 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:B/Nm8knRdTfn3VDWcbdxadoK/bA=', 'Content-Length': '218'} 2024-03-04T17:57:55.528 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ad337e16ea6af67f-0065e60ba3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.528 DEBUG:boto:path=/ppfahn-36/ 2024-03-04T17:57:55.528 DEBUG:boto:auth_path=/ppfahn-36/ 2024-03-04T17:57:55.528 DEBUG:boto:path=/ppfahn-36/?cors 2024-03-04T17:57:55.528 DEBUG:boto:auth_path=/ppfahn-36/?cors 2024-03-04T17:57:55.528 DEBUG:boto:Method: GET 2024-03-04T17:57:55.529 DEBUG:boto:Path: /ppfahn-36/?cors 2024-03-04T17:57:55.529 DEBUG:boto:Data: 2024-03-04T17:57:55.529 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.529 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:55.529 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.529 DEBUG:boto:Params: {} 2024-03-04T17:57:55.529 DEBUG:boto:Token: None 2024-03-04T17:57:55.529 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-36/?cors 2024-03-04T17:57:55.529 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9YhS6oJfAwYqHGeOjNyYp+4koq8= 2024-03-04T17:57:55.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9YhS6oJfAwYqHGeOjNyYp+4koq8=', 'Content-Length': '0'} 2024-03-04T17:57:55.531 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ba6487a552969fc-0065e60ba3-4349-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '266'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.572 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-03-04T17:57:55.572 DEBUG:boto:path=/ppfahn-37/ 2024-03-04T17:57:55.572 DEBUG:boto:auth_path=/ppfahn-37/ 2024-03-04T17:57:55.572 DEBUG:boto:path=/ppfahn-37/?cors 2024-03-04T17:57:55.573 DEBUG:boto:auth_path=/ppfahn-37/?cors 2024-03-04T17:57:55.573 DEBUG:boto:Method: PUT 2024-03-04T17:57:55.573 DEBUG:boto:Path: /ppfahn-37/?cors 2024-03-04T17:57:55.573 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-03-04T17:57:55.573 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-03-04T17:57:55.573 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:55.573 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.573 DEBUG:boto:Params: {} 2024-03-04T17:57:55.573 DEBUG:boto:Token: None 2024-03-04T17:57:55.573 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Mon, 04 Mar 2024 17:57:55 GMT /ppfahn-37/?cors 2024-03-04T17:57:55.573 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ESrySasItOvqf9kDt+rrsfsAoc4= 2024-03-04T17:57:55.574 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ESrySasItOvqf9kDt+rrsfsAoc4=', 'Content-Length': '218'} 2024-03-04T17:57:56.593 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002d4620df791890d7-0065e60ba3-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:56.593 DEBUG:boto:path=/ppfahn-37/ 2024-03-04T17:57:56.593 DEBUG:boto:auth_path=/ppfahn-37/ 2024-03-04T17:57:56.593 DEBUG:boto:path=/ppfahn-37/?cors 2024-03-04T17:57:56.593 DEBUG:boto:auth_path=/ppfahn-37/?cors 2024-03-04T17:57:56.594 DEBUG:boto:Method: GET 2024-03-04T17:57:56.594 DEBUG:boto:Path: /ppfahn-37/?cors 2024-03-04T17:57:56.594 DEBUG:boto:Data: 2024-03-04T17:57:56.594 DEBUG:boto:Headers: {} 2024-03-04T17:57:56.594 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:56.594 DEBUG:boto:Port: 8000 2024-03-04T17:57:56.594 DEBUG:boto:Params: {} 2024-03-04T17:57:56.594 DEBUG:boto:Token: None 2024-03-04T17:57:56.594 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:56 GMT /ppfahn-37/?cors 2024-03-04T17:57:56.594 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9fLOUGK4kND1S7/wv4i8p88UdMk= 2024-03-04T17:57:56.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:56 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9fLOUGK4kND1S7/wv4i8p88UdMk=', 'Content-Length': '0'} 2024-03-04T17:57:56.596 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000050677517aed4385c-0065e60ba4-4306-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '266'), ('Date', 'Mon, 04 Mar 2024 17:57:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:56.640 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-03-04T17:57:56.640 DEBUG:boto:path=/ppfahn-38/ 2024-03-04T17:57:56.640 DEBUG:boto:auth_path=/ppfahn-38/ 2024-03-04T17:57:56.640 DEBUG:boto:path=/ppfahn-38/?cors 2024-03-04T17:57:56.641 DEBUG:boto:auth_path=/ppfahn-38/?cors 2024-03-04T17:57:56.641 DEBUG:boto:Method: PUT 2024-03-04T17:57:56.641 DEBUG:boto:Path: /ppfahn-38/?cors 2024-03-04T17:57:56.641 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-03-04T17:57:56.641 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-03-04T17:57:56.641 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:56.641 DEBUG:boto:Port: 8001 2024-03-04T17:57:56.641 DEBUG:boto:Params: {} 2024-03-04T17:57:56.641 DEBUG:boto:Token: None 2024-03-04T17:57:56.641 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Mon, 04 Mar 2024 17:57:56 GMT /ppfahn-38/?cors 2024-03-04T17:57:56.642 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:r/f9l1sydPhDMDhX236ww+4cyEM= 2024-03-04T17:57:56.642 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:56 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:r/f9l1sydPhDMDhX236ww+4cyEM=', 'Content-Length': '218'} 2024-03-04T17:57:57.665 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bffcc212d662a49c-0065e60ba4-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:57.665 DEBUG:boto:path=/ppfahn-38/ 2024-03-04T17:57:57.665 DEBUG:boto:auth_path=/ppfahn-38/ 2024-03-04T17:57:57.665 DEBUG:boto:path=/ppfahn-38/?cors 2024-03-04T17:57:57.665 DEBUG:boto:auth_path=/ppfahn-38/?cors 2024-03-04T17:57:57.665 DEBUG:boto:Method: GET 2024-03-04T17:57:57.665 DEBUG:boto:Path: /ppfahn-38/?cors 2024-03-04T17:57:57.665 DEBUG:boto:Data: 2024-03-04T17:57:57.666 DEBUG:boto:Headers: {} 2024-03-04T17:57:57.666 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:57.666 DEBUG:boto:Port: 8001 2024-03-04T17:57:57.666 DEBUG:boto:Params: {} 2024-03-04T17:57:57.666 DEBUG:boto:Token: None 2024-03-04T17:57:57.666 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:57:57 GMT /ppfahn-38/?cors 2024-03-04T17:57:57.666 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:EGflLiaoqLNxJMqFnwGxXtzCs9c= 2024-03-04T17:57:57.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:57 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EGflLiaoqLNxJMqFnwGxXtzCs9c=', 'Content-Length': '0'} 2024-03-04T17:57:57.668 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b9941a15ee2b97e7-0065e60ba5-4356-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '266'), ('Date', 'Mon, 04 Mar 2024 17:57:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:57.668 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-03-04T17:57:57.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2024-03-04T17:57:57.669 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-39 2024-03-04T17:57:57.669 DEBUG:boto:path=/ppfahn-39/ 2024-03-04T17:57:57.669 DEBUG:boto:auth_path=/ppfahn-39/ 2024-03-04T17:57:57.669 DEBUG:boto:Method: PUT 2024-03-04T17:57:57.670 DEBUG:boto:Path: /ppfahn-39/ 2024-03-04T17:57:57.670 DEBUG:boto:Data: 2024-03-04T17:57:57.670 DEBUG:boto:Headers: {} 2024-03-04T17:57:57.670 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T17:57:57.670 DEBUG:boto:Port: 8000 2024-03-04T17:57:57.670 DEBUG:boto:Params: {} 2024-03-04T17:57:57.670 DEBUG:boto:Token: None 2024-03-04T17:57:57.670 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:57 GMT /ppfahn-39/ 2024-03-04T17:57:57.670 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pOLIVhlha78u/MxFGMU3xnGMw6Y= 2024-03-04T17:57:57.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:57 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pOLIVhlha78u/MxFGMU3xnGMw6Y=', 'Content-Length': '0'} 2024-03-04T17:57:57.706 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e4f4a20d5da73bbd-0065e60ba5-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:57.706 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-40 2024-03-04T17:57:57.707 DEBUG:boto:path=/ppfahn-40/ 2024-03-04T17:57:57.707 DEBUG:boto:auth_path=/ppfahn-40/ 2024-03-04T17:57:57.707 DEBUG:boto:Method: PUT 2024-03-04T17:57:57.707 DEBUG:boto:Path: /ppfahn-40/ 2024-03-04T17:57:57.707 DEBUG:boto:Data: 2024-03-04T17:57:57.707 DEBUG:boto:Headers: {} 2024-03-04T17:57:57.707 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T17:57:57.707 DEBUG:boto:Port: 8000 2024-03-04T17:57:57.707 DEBUG:boto:Params: {} 2024-03-04T17:57:57.707 DEBUG:boto:Token: None 2024-03-04T17:57:57.708 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:57 GMT /ppfahn-40/ 2024-03-04T17:57:57.708 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:illUyZQB/5xMr3RxrtGlJJ/Ya6Y= 2024-03-04T17:57:57.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:57 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:illUyZQB/5xMr3RxrtGlJJ/Ya6Y=', 'Content-Length': '0'} 2024-03-04T17:57:57.783 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000090cb67cd70ec89f7-0065e60ba5-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:57.783 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-41 2024-03-04T17:57:57.783 DEBUG:boto:path=/ppfahn-41/ 2024-03-04T17:57:57.783 DEBUG:boto:auth_path=/ppfahn-41/ 2024-03-04T17:57:57.783 DEBUG:boto:Method: PUT 2024-03-04T17:57:57.784 DEBUG:boto:Path: /ppfahn-41/ 2024-03-04T17:57:57.784 DEBUG:boto:Data: 2024-03-04T17:57:57.784 DEBUG:boto:Headers: {} 2024-03-04T17:57:57.784 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T17:57:57.784 DEBUG:boto:Port: 8001 2024-03-04T17:57:57.784 DEBUG:boto:Params: {} 2024-03-04T17:57:57.784 DEBUG:boto:Token: None 2024-03-04T17:57:57.784 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:57 GMT /ppfahn-41/ 2024-03-04T17:57:57.784 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hoP2wBD9hudc1aKEgssHGx7sW1c= 2024-03-04T17:57:57.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:57 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hoP2wBD9hudc1aKEgssHGx7sW1c=', 'Content-Length': '0'} 2024-03-04T17:57:57.892 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a42e7d8d3fb2275a-0065e60ba5-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:57.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:57.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:57.893 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:57.956 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:57.956 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:58.013 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T17:57:58.036 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709575077.825024_336.1","last_update":"2024-03-04T17:57:57.825024Z"}] 2024-03-04T17:57:58.036 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:57:58.037 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:57:58.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:58.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:58.037 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:58.079 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:58.079 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:58.143+0000 7fb798edd980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:57:58.143+0000 7fb798edd980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:58.163 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:58Z"} 2024-03-04T17:57:58.164 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:58Z"} 2024-03-04T17:57:58.164 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T17:57:58.164 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:57:58.164 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T17:57:58.164 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T17:58:03.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:03.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:03.166 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:03.225 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:03.225 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:03.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:03.302+0000 7f7bc75dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:03.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:03.302+0000 7f7bc75dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:03.323 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:03Z"} 2024-03-04T17:58:03.323 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:03Z"} 2024-03-04T17:58:03.323 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T17:58:03.323 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:58:03.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T17:58:03.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T17:58:08.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:08.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:08.325 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:08.369 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:08.369 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:08.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:08.458+0000 7f640dc9e980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:08.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:08.462+0000 7f640dc9e980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:08.481 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:08Z"} 2024-03-04T17:58:08.481 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:08Z"} 2024-03-04T17:58:08.481 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T17:58:08.481 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:58:08.481 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T17:58:08.481 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T17:58:13.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:13.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:13.483 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:13.586+0000 7f6de1720980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:13.586+0000 7f6de1720980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:13.606 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:13Z"} 2024-03-04T17:58:13.606 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:13Z"} 2024-03-04T17:58:13.606 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.861873_275.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.861873Z', 'realm_epoch': 2}}] 2024-03-04T17:58:13.606 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:58:13.607 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.861873_275.1'} 2024-03-04T17:58:13.607 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.861873_275.1 2024-03-04T17:58:18.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:18.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:18.609 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:18.651 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:18.651 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:18.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:18.710+0000 7f1d59043980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:18.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:18.710+0000 7f1d59043980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:18.731 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:18Z"} 2024-03-04T17:58:18.731 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:18Z"} 2024-03-04T17:58:18.731 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T17:58:18.732 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:58:18.732 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T17:58:18.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T17:58:23.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:23.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:23.733 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:23.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:23.834+0000 7fe89b4c8980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:23.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:23.834+0000 7fe89b4c8980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:23.868 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:23Z"} 2024-03-04T17:58:23.868 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:23Z"} 2024-03-04T17:58:23.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T17:58:23.869 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:58:23.869 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T17:58:23.869 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T17:58:28.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:28.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:28.871 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:28.914 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:28.914 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:28.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:28.974+0000 7fe70021f980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:28.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:28.974+0000 7fe70021f980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:28.995 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:28Z"} 2024-03-04T17:58:28.995 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:28Z"} 2024-03-04T17:58:28.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T17:58:28.995 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:58:28.996 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T17:58:28.996 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T17:58:33.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:33.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:33.997 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:34.040 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:34.040 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:34.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:34.098+0000 7fa1fe158980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:34.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:34.098+0000 7fa1fe158980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:34.120 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:34Z"} 2024-03-04T17:58:34.120 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:34Z"} 2024-03-04T17:58:34.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T17:58:34.120 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:58:34.120 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T17:58:34.121 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T17:58:39.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:39.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:39.122 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:39.173 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:39.173 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:39.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:39.230+0000 7f30fde7d980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:39.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:39.234+0000 7f30fde7d980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:39.253 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:39Z"} 2024-03-04T17:58:39.253 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:39Z"} 2024-03-04T17:58:39.254 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T17:58:39.254 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:58:39.254 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T17:58:39.254 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T17:58:44.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:44.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:44.256 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:44.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:44.349+0000 7ff7cfad2980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:44.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:44.353+0000 7ff7cfad2980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:44.373 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:44Z"} 2024-03-04T17:58:44.374 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:44Z"} 2024-03-04T17:58:44.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T17:58:44.374 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:58:44.374 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T17:58:44.374 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T17:58:49.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:49.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:49.376 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:49.418 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:49.418 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:49.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:49.473+0000 7fb9324a9980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:49.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:49.473+0000 7fb9324a9980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:49.496 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:49Z"} 2024-03-04T17:58:49.496 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:49Z"} 2024-03-04T17:58:49.496 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T17:58:49.496 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:58:49.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T17:58:49.496 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T17:58:54.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:54.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:54.497 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:54.542 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:54.542 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:54.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:54.597+0000 7fd9b4b3e980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:54.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:54.597+0000 7fd9b4b3e980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:54.621 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:54Z"} 2024-03-04T17:58:54.621 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:54Z"} 2024-03-04T17:58:54.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T17:58:54.621 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:58:54.622 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T17:58:54.622 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T17:58:59.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:59.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:59.623 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:59.666 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:59.666 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:59.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:59.725+0000 7ff5027c6980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:59.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:58:59.725+0000 7ff5027c6980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:59.746 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:59Z"} 2024-03-04T17:58:59.746 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:59Z"} 2024-03-04T17:58:59.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T17:58:59.746 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:58:59.746 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T17:58:59.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T17:59:04.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:04.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:04.748 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:04.787 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:04.787 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:04.869+0000 7f4dc672b980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:04.869+0000 7f4dc672b980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:04.898 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:04Z"} 2024-03-04T17:59:04.898 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:04Z"} 2024-03-04T17:59:04.898 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T17:59:04.898 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:59:04.899 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T17:59:04.899 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T17:59:09.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:09.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:09.900 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:09.946 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:09.946 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:10.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:10.001+0000 7f43bc79b980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:10.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:10.005+0000 7f43bc79b980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:10.025 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:10Z"} 2024-03-04T17:59:10.026 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:10Z"} 2024-03-04T17:59:10.026 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.984537_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.984537Z', 'realm_epoch': 2}}] 2024-03-04T17:59:10.026 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:59:10.026 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.984537_285.1'} 2024-03-04T17:59:10.026 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.984537_285.1 2024-03-04T17:59:15.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:15.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:15.028 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:15.071 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:15.071 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:15.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:15.129+0000 7f4fd3fbd980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:15.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:15.129+0000 7f4fd3fbd980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:15.152 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:15Z"} 2024-03-04T17:59:15.152 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:15Z"} 2024-03-04T17:59:15.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.984537_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.984537Z', 'realm_epoch': 2}}] 2024-03-04T17:59:15.152 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:59:15.152 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.984537_285.1'} 2024-03-04T17:59:15.153 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.984537_285.1 2024-03-04T17:59:20.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:20.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:20.155 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:20.199 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:20.199 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:20.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:20.268+0000 7f220e7cf980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:20.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:20.268+0000 7f220e7cf980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:20.290 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:20Z"} 2024-03-04T17:59:20.290 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:20Z"} 2024-03-04T17:59:20.290 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.984537_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.984537Z', 'realm_epoch': 2}}] 2024-03-04T17:59:20.290 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:59:20.291 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.984537_285.1'} 2024-03-04T17:59:20.291 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.984537_285.1 2024-03-04T17:59:25.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:25.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:25.292 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:25.335 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:25.335 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:25.392+0000 7f8be327a980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:25.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:25.396+0000 7f8be327a980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:25.417 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:25Z"} 2024-03-04T17:59:25.417 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:25Z"} 2024-03-04T17:59:25.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T17:59:25.418 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:59:25.418 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T17:59:25.418 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T17:59:30.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:30.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:30.419 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:30.463 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:30.463 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:30.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:30.520+0000 7f3b2b833980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:30.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:30.520+0000 7f3b2b833980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:30.546 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:30Z"} 2024-03-04T17:59:30.546 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:30Z"} 2024-03-04T17:59:30.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T17:59:30.547 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:59:30.547 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T17:59:30.547 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T17:59:35.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:35.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:35.548 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:35.591 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:35.591 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:35.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:35.652+0000 7f03cc4a1980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:35.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:35.652+0000 7f03cc4a1980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:35.674 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:35Z"} 2024-03-04T17:59:35.675 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:35Z"} 2024-03-04T17:59:35.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T17:59:35.675 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:59:35.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T17:59:35.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T17:59:40.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:40.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:40.676 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:40.718 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:40.718 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:40.772+0000 7f078f2e5980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:40.772+0000 7f078f2e5980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:40.798 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:40Z"} 2024-03-04T17:59:40.798 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:40Z"} 2024-03-04T17:59:40.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T17:59:40.798 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:59:40.798 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T17:59:40.798 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T17:59:45.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:45.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:45.800 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:45.842 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:45.842 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:45.900+0000 7f39433b0980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:45.900+0000 7f39433b0980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:45.924 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:45Z"} 2024-03-04T17:59:45.924 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:45Z"} 2024-03-04T17:59:45.925 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T17:59:45.925 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:59:45.925 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T17:59:45.925 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T17:59:50.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:50.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:50.926 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:51.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:51.023+0000 7fa501a0e980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:51.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:51.023+0000 7fa501a0e980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:51.048 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:51Z"} 2024-03-04T17:59:51.049 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:51Z"} 2024-03-04T17:59:51.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T17:59:51.049 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:59:51.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T17:59:51.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T17:59:56.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:56.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:56.051 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:56.090 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:56.091 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:56.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:56.143+0000 7f495086d980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:56.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T17:59:56.143+0000 7f495086d980 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:56.171 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:56Z"} 2024-03-04T17:59:56.171 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:56Z"} 2024-03-04T17:59:56.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T17:59:56.172 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T17:59:56.172 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T17:59:56.172 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:00:01.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:01.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:01.173 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:01.213 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:01.213 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:01.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:01.267+0000 7f17dc209980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:01.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:01.267+0000 7f17dc209980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:01.292 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:01Z"} 2024-03-04T18:00:01.292 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:01Z"} 2024-03-04T18:00:01.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T18:00:01.293 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:00:01.293 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T18:00:01.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T18:00:06.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:06.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:06.294 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:06.345 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:06.345 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:06.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:06.399+0000 7f35b7ed8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:06.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:06.399+0000 7f35b7ed8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:06.423 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:06Z"} 2024-03-04T18:00:06.424 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:06Z"} 2024-03-04T18:00:06.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:00:06.424 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:00:06.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:00:06.424 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:00:11.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:11.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:11.426 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:11.468 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:11.468 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:11.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:11.523+0000 7faed07d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:11.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:11.523+0000 7faed07d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:11.547 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:11Z"} 2024-03-04T18:00:11.547 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:11Z"} 2024-03-04T18:00:11.548 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:00:11.548 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:00:11.548 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:00:11.548 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:00:16.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:16.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:16.549 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:16.592 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:16.592 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:16.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:16.651+0000 7fce752d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:16.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:16.651+0000 7fce752d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:16.671 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:16Z"} 2024-03-04T18:00:16.671 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:16Z"} 2024-03-04T18:00:16.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:00:16.671 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:00:16.672 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:00:16.672 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575075.271482_301.1 2024-03-04T18:00:21.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:21.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:21.673 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:21.715 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:21.715 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:21.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:21.771+0000 7f3789c3c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:21.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:21.775+0000 7f3789c3c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:21.794 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:21Z"} 2024-03-04T18:00:21.795 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:21Z"} 2024-03-04T18:00:21.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:00:21.795 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:00:21.795 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:00:21.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T18:00:26.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:26.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:26.796 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:26.898+0000 7f41d2cab980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:26.898+0000 7f41d2cab980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:26.919 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:26Z"} 2024-03-04T18:00:26.919 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:26Z"} 2024-03-04T18:00:26.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:00:26.920 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:00:26.920 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:00:26.920 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:00:31.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:31.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:31.921 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:31.963 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:31.964 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:32.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:32.018+0000 7fb259a3c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:32.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:32.022+0000 7fb259a3c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:32.043 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:32Z"} 2024-03-04T18:00:32.043 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:32Z"} 2024-03-04T18:00:32.043 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:00:32.043 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:00:32.043 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:00:32.044 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575075.271482_301.1 2024-03-04T18:00:37.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:37.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:37.045 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:37.086 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:37.086 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:37.142+0000 7f3e0dcc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:37.142+0000 7f3e0dcc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:37.166 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.019898_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.019898Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:37Z"} 2024-03-04T18:00:37.166 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.019898_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.019898Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:37Z"} 2024-03-04T18:00:37.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.019898_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.019898Z', 'realm_epoch': 2}}] 2024-03-04T18:00:37.167 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:00:37.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.019898_288.1'} 2024-03-04T18:00:37.167 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575075.019898_288.1 2024-03-04T18:00:42.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:42.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:42.169 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:42.213 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:42.213 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:42.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:42.274+0000 7f1d6e290980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:42.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:42.274+0000 7f1d6e290980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:42.297 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:42Z"} 2024-03-04T18:00:42.297 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:42Z"} 2024-03-04T18:00:42.297 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:00:42.297 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:00:42.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:00:42.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T18:00:47.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:47.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:47.299 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:47.343 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:47.343 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:47.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:47.398+0000 7fe0e2880980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:47.398+0000 7fe0e2880980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:47.421 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:47Z"} 2024-03-04T18:00:47.421 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:47Z"} 2024-03-04T18:00:47.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:00:47.421 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:00:47.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:00:47.421 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T18:00:52.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:52.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:52.423 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:52.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:52.562+0000 7fc8e90e1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:52.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:52.562+0000 7fc8e90e1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:52.585 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:52Z"} 2024-03-04T18:00:52.586 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:52Z"} 2024-03-04T18:00:52.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:00:52.586 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:00:52.587 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:00:52.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T18:00:57.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:57.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:57.588 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:57.690+0000 7f9d932a6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:57.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:00:57.690+0000 7f9d932a6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:57.712 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:57Z"} 2024-03-04T18:00:57.712 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:57Z"} 2024-03-04T18:00:57.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:00:57.713 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:00:57.713 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:00:57.713 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:01:02.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:02.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:02.714 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:02.757 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:02.757 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:02.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:02.813+0000 7f90b465e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:02.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:02.813+0000 7f90b465e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:02.837 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:02Z"} 2024-03-04T18:01:02.837 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:02Z"} 2024-03-04T18:01:02.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T18:01:02.838 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:01:02.838 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T18:01:02.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T18:01:07.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:07.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:07.839 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:07.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:07.937+0000 7f0c191a6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:07.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:07.937+0000 7f0c191a6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:07.962 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:07Z"} 2024-03-04T18:01:07.963 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:07Z"} 2024-03-04T18:01:07.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:01:07.963 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:01:07.963 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:01:07.963 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T18:01:12.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:12.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:12.965 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:13.007 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:13.008 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:13.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:13.065+0000 7fe97ac82980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:13.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:13.065+0000 7fe97ac82980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:13.089 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:13Z"} 2024-03-04T18:01:13.089 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:13Z"} 2024-03-04T18:01:13.089 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:01:13.089 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:01:13.089 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:01:13.090 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T18:01:18.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:18.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:18.091 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:18.136 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:18.136 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:18.193+0000 7f87ac9ae980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:18.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:18.193+0000 7f87ac9ae980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:18.217 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:18Z"} 2024-03-04T18:01:18.217 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:18Z"} 2024-03-04T18:01:18.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:01:18.218 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:01:18.218 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:01:18.218 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T18:01:23.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:23.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:23.220 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:23.263 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:23.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:23.317+0000 7f9d587cf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:23.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:23.317+0000 7f9d587cf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:23.342 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:23Z"} 2024-03-04T18:01:23.342 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:23Z"} 2024-03-04T18:01:23.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:01:23.343 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:01:23.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:01:23.343 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:01:28.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:28.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:28.344 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:28.385 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:28.385 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:28.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:28.441+0000 7f1ee3bfd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:28.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:28.441+0000 7f1ee3bfd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:28.465 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:28Z"} 2024-03-04T18:01:28.466 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:28Z"} 2024-03-04T18:01:28.466 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:01:28.466 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:01:28.466 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:01:28.466 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:01:33.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:33.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:33.468 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:33.511 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:33.511 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:33.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:33.565+0000 7f2ef7fdc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:33.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:33.565+0000 7f2ef7fdc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:33.590 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.424470_313.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.424470Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:33Z"} 2024-03-04T18:01:33.590 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.424470_313.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.424470Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:33Z"} 2024-03-04T18:01:33.591 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.424470_313.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.424470Z', 'realm_epoch': 2}}] 2024-03-04T18:01:33.591 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:01:33.591 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.424470_313.1'} 2024-03-04T18:01:33.591 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575075.424470_313.1 2024-03-04T18:01:38.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:38.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:38.592 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:38.635 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:38.635 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:38.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:38.688+0000 7fccc0ec9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:38.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:38.688+0000 7fccc0ec9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:38.714 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:38Z"} 2024-03-04T18:01:38.714 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:38Z"} 2024-03-04T18:01:38.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:01:38.715 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:01:38.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:01:38.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T18:01:43.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:43.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:43.716 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:43.759 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:43.760 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:43.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:43.812+0000 7f5f8c63a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:43.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:43.812+0000 7f5f8c63a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:43.839 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:43Z"} 2024-03-04T18:01:43.839 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:43Z"} 2024-03-04T18:01:43.839 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:01:43.840 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:01:43.840 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:01:43.840 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:01:48.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:48.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:48.841 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:48.884 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:48.884 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:48.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:48.936+0000 7f95511a8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:48.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:48.940+0000 7f95511a8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:48.963 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:48Z"} 2024-03-04T18:01:48.964 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:48Z"} 2024-03-04T18:01:48.964 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:01:48.964 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:01:48.964 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:01:48.964 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T18:01:53.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:53.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:53.966 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:54.009 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:54.009 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:54.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:54.064+0000 7fc270bea980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:54.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:54.064+0000 7fc270bea980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:54.089 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:54Z"} 2024-03-04T18:01:54.089 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:54Z"} 2024-03-04T18:01:54.089 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:01:54.089 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:01:54.090 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:01:54.090 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T18:01:59.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:59.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:59.091 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:59.133 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:59.133 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:59.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:59.188+0000 7f592d269980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:59.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:01:59.188+0000 7f592d269980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:59.213 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:59Z"} 2024-03-04T18:01:59.213 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:59Z"} 2024-03-04T18:01:59.213 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T18:01:59.213 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:01:59.214 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T18:01:59.214 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.937123_282.1 2024-03-04T18:02:04.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:04.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:04.215 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:04.257 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:04.258 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:04.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:04.316+0000 7f66623de980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:04.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:04.316+0000 7f66623de980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:04.342 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:04Z"} 2024-03-04T18:02:04.342 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:04Z"} 2024-03-04T18:02:04.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:02:04.342 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:02:04.342 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:02:04.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:02:09.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:09.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:09.344 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:09.387 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:09.388 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:09.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:09.440+0000 7ff43f4fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:09.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:09.444+0000 7ff43f4fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:09.467 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:09Z"} 2024-03-04T18:02:09.467 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:09Z"} 2024-03-04T18:02:09.467 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:02:09.468 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:02:09.468 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:02:09.468 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T18:02:14.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:14.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:14.470 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:14.527 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:14.527 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:14.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:14.620+0000 7f7a48ef8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:14.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:14.620+0000 7f7a48ef8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:14.645 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:14Z"} 2024-03-04T18:02:14.645 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:14Z"} 2024-03-04T18:02:14.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:02:14.646 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:02:14.646 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:02:14.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T18:02:19.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:19.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:19.648 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:19.691 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:19.692 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:19.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:19.747+0000 7f7952ab1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:19.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:19.747+0000 7f7952ab1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:19.773 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:19Z"} 2024-03-04T18:02:19.773 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:19Z"} 2024-03-04T18:02:19.773 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T18:02:19.773 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:02:19.774 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T18:02:19.774 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.937123_282.1 2024-03-04T18:02:24.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:24.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:24.776 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:24.820 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:24.820 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:24.875+0000 7fd819909980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:24.875+0000 7fd819909980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:24.903 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:24Z"} 2024-03-04T18:02:24.903 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:24Z"} 2024-03-04T18:02:24.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:02:24.903 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:02:24.903 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:02:24.904 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:02:29.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:29.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:29.906 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:29.951 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:29.951 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:30.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:30.003+0000 7f98aeaa0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:30.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:30.007+0000 7f98aeaa0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:30.031 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:30Z"} 2024-03-04T18:02:30.032 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:30Z"} 2024-03-04T18:02:30.032 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:02:30.032 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:02:30.032 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:02:30.032 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:02:35.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:35.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:35.034 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:35.078 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:35.078 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:35.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:35.131+0000 7f188d3fc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:35.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:35.131+0000 7f188d3fc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:35.158 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:35Z"} 2024-03-04T18:02:35.158 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:35Z"} 2024-03-04T18:02:35.158 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:02:35.158 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:02:35.158 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:02:35.159 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T18:02:36.210 INFO:tasks.rgw.c2.client.0.smithi153.stdout:2024-03-04T18:02:36.207+0000 7fa53962a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:02:40.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:40.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:40.160 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:40.205 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:40.206 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:40.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:40.259+0000 7fee128d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:40.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:40.259+0000 7fee128d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:40.288 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:40Z"} 2024-03-04T18:02:40.288 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:40Z"} 2024-03-04T18:02:40.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:02:40.288 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:02:40.288 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:02:40.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:02:45.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:45.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:45.289 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:45.332 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:45.332 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:45.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:45.387+0000 7f7eb87ec980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:45.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:45.387+0000 7f7eb87ec980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:45.412 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:45Z"} 2024-03-04T18:02:45.412 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:45Z"} 2024-03-04T18:02:45.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:02:45.413 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:02:45.413 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:02:45.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:02:50.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:50.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:50.415 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:50.457 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:50.458 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:50.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:50.511+0000 7f7ed19cd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:50.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:50.511+0000 7f7ed19cd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:50.538 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:50Z"} 2024-03-04T18:02:50.538 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:50Z"} 2024-03-04T18:02:50.538 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.984537_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.984537Z', 'realm_epoch': 2}}] 2024-03-04T18:02:50.539 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:02:50.539 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.984537_285.1'} 2024-03-04T18:02:50.539 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.984537_285.1 2024-03-04T18:02:55.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:55.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:55.541 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:55.584 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:55.585 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:55.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:55.639+0000 7fccfc8fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:55.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:02:55.643+0000 7fccfc8fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:55.665 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:55Z"} 2024-03-04T18:02:55.665 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:55Z"} 2024-03-04T18:02:55.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.984537_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.984537Z', 'realm_epoch': 2}}] 2024-03-04T18:02:55.666 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:02:55.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.984537_285.1'} 2024-03-04T18:02:55.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.984537_285.1 2024-03-04T18:03:00.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:00.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:00.668 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:00.710 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:00.710 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:00.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:00.767+0000 7f4136719980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:00.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:00.771+0000 7f4136719980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:00.793 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:00Z"} 2024-03-04T18:03:00.793 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:00Z"} 2024-03-04T18:03:00.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:03:00.793 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T18:03:00.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:03:00.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T18:03:05.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2024-03-04T18:03:05.797 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-42 2024-03-04T18:03:05.797 DEBUG:boto:path=/ppfahn-42/ 2024-03-04T18:03:05.797 DEBUG:boto:auth_path=/ppfahn-42/ 2024-03-04T18:03:05.798 DEBUG:boto:Method: PUT 2024-03-04T18:03:05.798 DEBUG:boto:Path: /ppfahn-42/ 2024-03-04T18:03:05.798 DEBUG:boto:Data: 2024-03-04T18:03:05.798 DEBUG:boto:Headers: {} 2024-03-04T18:03:05.798 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:03:05.798 DEBUG:boto:Port: 8000 2024-03-04T18:03:05.798 DEBUG:boto:Params: {} 2024-03-04T18:03:05.799 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:03:05.799 DEBUG:boto:Token: None 2024-03-04T18:03:05.799 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:03:05 GMT /ppfahn-42/ 2024-03-04T18:03:05.799 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tU1VwWnzM0JT5PGbQgquKhABkX0= 2024-03-04T18:03:05.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:03:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tU1VwWnzM0JT5PGbQgquKhABkX0=', 'Content-Length': '0'} 2024-03-04T18:03:05.838 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086fe4b65a61911d9-0065e60cd9-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:03:05.838 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-43 2024-03-04T18:03:05.838 DEBUG:boto:path=/ppfahn-43/ 2024-03-04T18:03:05.839 DEBUG:boto:auth_path=/ppfahn-43/ 2024-03-04T18:03:05.839 DEBUG:boto:Method: PUT 2024-03-04T18:03:05.839 DEBUG:boto:Path: /ppfahn-43/ 2024-03-04T18:03:05.839 DEBUG:boto:Data: 2024-03-04T18:03:05.839 DEBUG:boto:Headers: {} 2024-03-04T18:03:05.839 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:03:05.839 DEBUG:boto:Port: 8000 2024-03-04T18:03:05.839 DEBUG:boto:Params: {} 2024-03-04T18:03:05.840 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:03:05.840 DEBUG:boto:Token: None 2024-03-04T18:03:05.840 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:03:05 GMT /ppfahn-43/ 2024-03-04T18:03:05.840 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:xvEzjdVNPdXqcIFOTddK6DPfETQ= 2024-03-04T18:03:05.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:03:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xvEzjdVNPdXqcIFOTddK6DPfETQ=', 'Content-Length': '0'} 2024-03-04T18:03:05.910 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fbd19d050df4c6e1-0065e60cd9-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:03:05.911 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-44 2024-03-04T18:03:05.911 DEBUG:boto:path=/ppfahn-44/ 2024-03-04T18:03:05.911 DEBUG:boto:auth_path=/ppfahn-44/ 2024-03-04T18:03:05.911 DEBUG:boto:Method: PUT 2024-03-04T18:03:05.911 DEBUG:boto:Path: /ppfahn-44/ 2024-03-04T18:03:05.911 DEBUG:boto:Data: 2024-03-04T18:03:05.911 DEBUG:boto:Headers: {} 2024-03-04T18:03:05.911 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:03:05.911 DEBUG:boto:Port: 8001 2024-03-04T18:03:05.911 DEBUG:boto:Params: {} 2024-03-04T18:03:05.912 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T18:03:05.912 DEBUG:boto:Token: None 2024-03-04T18:03:05.912 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:03:05 GMT /ppfahn-44/ 2024-03-04T18:03:05.912 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FCyRPDq0sFMq2EdDx79FKUHKEjI= 2024-03-04T18:03:05.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:03:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FCyRPDq0sFMq2EdDx79FKUHKEjI=', 'Content-Length': '0'} 2024-03-04T18:03:06.066 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000035f7572cab1bf4a9-0065e60cd9-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:03:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:03:06.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:06.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:06.066 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:06.099 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:06.099 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:06.150 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T18:03:06.176 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709575386.010513_362.1","last_update":"2024-03-04T18:03:06.010513Z"}] 2024-03-04T18:03:06.176 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:03:06.176 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:03:06.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:06.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:06.177 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:06.228 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:06.229 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:06.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:06.290+0000 7f0e97fd5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:06.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:06.290+0000 7f0e97fd5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:06.314 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:06Z"} 2024-03-04T18:03:06.315 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:06Z"} 2024-03-04T18:03:06.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T18:03:06.315 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:03:06.315 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T18:03:06.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.937123_282.1 2024-03-04T18:03:11.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:11.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:11.317 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:11.360 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:11.360 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:11.426+0000 7f3db5652980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:11.426+0000 7f3db5652980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:11.452 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.132048_294.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.132048Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:11Z"} 2024-03-04T18:03:11.452 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.132048_294.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.132048Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:11Z"} 2024-03-04T18:03:11.452 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.132048_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.132048Z', 'realm_epoch': 2}}] 2024-03-04T18:03:11.453 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:03:11.453 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.132048_294.1'} 2024-03-04T18:03:11.453 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.132048_294.1 2024-03-04T18:03:16.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:16.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:16.454 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:16.498 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:16.498 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:16.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:16.566+0000 7f1cfd5bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:16.566+0000 7f1cfd5bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:16.589 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:16Z"} 2024-03-04T18:03:16.590 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:16Z"} 2024-03-04T18:03:16.590 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:03:16.590 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:03:16.590 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:03:16.590 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:03:21.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:21.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:21.591 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:21.635 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:21.635 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:21.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:21.690+0000 7f872aae3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:21.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:21.690+0000 7f872aae3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:21.716 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.132048_294.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.132048Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:21Z"} 2024-03-04T18:03:21.716 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.132048_294.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.132048Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:21Z"} 2024-03-04T18:03:21.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.132048_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.132048Z', 'realm_epoch': 2}}] 2024-03-04T18:03:21.717 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:03:21.717 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.132048_294.1'} 2024-03-04T18:03:21.717 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.132048_294.1 2024-03-04T18:03:26.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:26.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:26.718 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:26.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:26.818+0000 7f96db7de980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:26.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:26.818+0000 7f96db7de980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:26.842 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:26Z"} 2024-03-04T18:03:26.842 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:26Z"} 2024-03-04T18:03:26.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T18:03:26.842 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:03:26.842 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T18:03:26.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.937123_282.1 2024-03-04T18:03:31.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:31.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:31.844 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:31.887 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:31.887 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:31.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:31.946+0000 7fc3c0f2d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:31.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:31.946+0000 7fc3c0f2d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:31.972 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:31Z"} 2024-03-04T18:03:31.972 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:31Z"} 2024-03-04T18:03:31.972 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T18:03:31.972 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:03:31.972 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T18:03:31.972 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.930202_281.1 2024-03-04T18:03:36.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:36.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:36.973 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:37.015 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:37.015 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:37.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:37.070+0000 7f1e180d9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:37.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:37.074+0000 7f1e180d9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:37.097 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:37Z"} 2024-03-04T18:03:37.098 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:37Z"} 2024-03-04T18:03:37.098 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T18:03:37.098 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:03:37.098 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T18:03:37.098 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.937123_282.1 2024-03-04T18:03:42.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:42.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:42.100 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:42.152 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:42.152 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:42.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:42.218+0000 7fda27909980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:42.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:42.218+0000 7fda27909980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:42.243 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:42Z"} 2024-03-04T18:03:42.243 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:42Z"} 2024-03-04T18:03:42.243 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:03:42.244 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:03:42.244 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:03:42.244 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T18:03:47.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:47.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:47.245 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:47.288 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:47.288 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:47.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:47.346+0000 7f5ff9024980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:47.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:47.346+0000 7f5ff9024980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:47.369 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:47Z"} 2024-03-04T18:03:47.369 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:47Z"} 2024-03-04T18:03:47.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:03:47.369 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:03:47.369 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:03:47.370 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T18:03:52.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:52.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:52.371 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:52.413 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:52.413 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:52.469+0000 7f11891ee980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:52.469+0000 7f11891ee980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:52.493 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:52Z"} 2024-03-04T18:03:52.493 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:52Z"} 2024-03-04T18:03:52.493 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:03:52.493 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:03:52.493 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:03:52.494 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T18:03:57.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:57.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:57.495 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:57.593+0000 7f07d8961980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:03:57.593+0000 7f07d8961980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:57.618 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:57Z"} 2024-03-04T18:03:57.619 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:57Z"} 2024-03-04T18:03:57.619 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:03:57.619 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:03:57.619 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:03:57.619 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T18:04:02.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:02.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:02.621 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:02.663 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:02.663 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:02.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:02.717+0000 7f20f0e26980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:02.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:02.721+0000 7f20f0e26980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:02.744 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:02Z"} 2024-03-04T18:04:02.744 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:02Z"} 2024-03-04T18:04:02.745 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:04:02.745 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:04:02.745 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:04:02.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:04:07.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:07.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:07.746 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:07.788 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:07.789 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:07.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:07.865+0000 7fd0a1ecf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:07.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:07.869+0000 7fd0a1ecf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:07.904 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:07Z"} 2024-03-04T18:04:07.904 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:07Z"} 2024-03-04T18:04:07.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:04:07.907 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:04:07.907 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:04:07.907 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:04:12.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:12.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:12.908 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:12.951 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:12.951 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:13.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:13.005+0000 7f29be0d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:13.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:13.005+0000 7f29be0d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:13.031 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:13Z"} 2024-03-04T18:04:13.031 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:13Z"} 2024-03-04T18:04:13.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T18:04:13.031 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:04:13.031 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T18:04:13.031 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.937123_282.1 2024-03-04T18:04:18.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:18.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:18.032 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:18.080 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:18.080 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:18.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:18.137+0000 7f6d5cae4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:18.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:18.137+0000 7f6d5cae4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:18.163 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:18Z"} 2024-03-04T18:04:18.164 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:18Z"} 2024-03-04T18:04:18.164 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T18:04:18.164 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:04:18.164 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T18:04:18.164 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.937123_282.1 2024-03-04T18:04:23.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:23.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:23.166 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:23.265+0000 7fa531d8a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:23.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:23.265+0000 7fa531d8a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:23.292 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:23Z"} 2024-03-04T18:04:23.292 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:23Z"} 2024-03-04T18:04:23.292 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:04:23.293 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:04:23.293 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:04:23.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:04:28.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:28.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:28.294 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:28.335 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:28.336 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:28.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:28.389+0000 7f8ddaf90980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:28.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:28.389+0000 7f8ddaf90980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:28.416 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:28Z"} 2024-03-04T18:04:28.416 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:28Z"} 2024-03-04T18:04:28.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.861873_275.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.861873Z', 'realm_epoch': 2}}] 2024-03-04T18:04:28.417 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:04:28.417 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.861873_275.1'} 2024-03-04T18:04:28.417 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.861873_275.1 2024-03-04T18:04:33.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:33.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:33.418 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:33.460 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:33.460 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:33.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:33.517+0000 7fa47f75a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:33.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:33.517+0000 7fa47f75a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:33.542 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:33Z"} 2024-03-04T18:04:33.542 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:33Z"} 2024-03-04T18:04:33.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:04:33.543 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:04:33.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:04:33.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:04:38.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:38.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:38.544 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:38.586 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:38.587 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:38.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:38.641+0000 7fed7b362980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:38.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:38.641+0000 7fed7b362980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:38.668 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:38Z"} 2024-03-04T18:04:38.668 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:38Z"} 2024-03-04T18:04:38.668 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:04:38.668 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:04:38.668 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:04:38.668 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:04:43.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:43.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:43.670 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:43.713 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:43.713 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:43.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:43.768+0000 7f7c2ae7b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:43.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:43.768+0000 7f7c2ae7b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:43.794 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:43Z"} 2024-03-04T18:04:43.794 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:43Z"} 2024-03-04T18:04:43.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:04:43.795 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:04:43.795 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:04:43.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:04:48.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:48.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:48.797 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:48.840 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:48.840 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:48.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:48.896+0000 7f93f04dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:48.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:48.896+0000 7f93f04dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:48.923 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:48Z"} 2024-03-04T18:04:48.923 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:48Z"} 2024-03-04T18:04:48.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:04:48.924 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:04:48.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:04:48.924 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.288119_303.1 2024-03-04T18:04:53.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:53.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:53.926 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:53.969 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:53.969 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:54.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:54.024+0000 7f8653971980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:54.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:54.024+0000 7f8653971980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:54.091 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:54Z"} 2024-03-04T18:04:54.092 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:54Z"} 2024-03-04T18:04:54.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:04:54.092 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:04:54.092 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:04:54.092 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T18:04:59.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:59.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:59.093 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:59.137 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:59.137 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:59.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:59.192+0000 7f64b0f5b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:59.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:04:59.192+0000 7f64b0f5b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:59.216 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:59Z"} 2024-03-04T18:04:59.217 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:59Z"} 2024-03-04T18:04:59.217 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T18:04:59.217 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:04:59.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T18:04:59.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.930202_281.1 2024-03-04T18:05:04.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:04.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:04.219 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:04.261 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:04.261 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:04.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:04.320+0000 7ff55ce32980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:04.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:04.320+0000 7ff55ce32980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:04.346 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:04Z"} 2024-03-04T18:05:04.346 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:04Z"} 2024-03-04T18:05:04.346 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:05:04.347 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:05:04.347 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:05:04.347 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:05:09.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:09.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:09.348 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:09.391 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:09.392 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:09.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:09.444+0000 7f8a21902980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:09.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:09.448+0000 7f8a21902980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:09.472 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:09Z"} 2024-03-04T18:05:09.473 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:09Z"} 2024-03-04T18:05:09.473 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:05:09.473 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:05:09.473 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:05:09.473 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:05:14.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:14.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:14.475 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:14.515 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:14.515 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:14.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:14.568+0000 7f44c2a08980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:14.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:14.568+0000 7f44c2a08980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:14.595 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:14Z"} 2024-03-04T18:05:14.595 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:14Z"} 2024-03-04T18:05:14.595 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:05:14.595 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:05:14.595 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:05:14.595 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.271482_301.1 2024-03-04T18:05:19.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:19.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:19.597 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:19.639 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:19.640 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:19.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:19.692+0000 7f4260ea6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:19.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:19.692+0000 7f4260ea6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:19.719 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:19Z"} 2024-03-04T18:05:19.720 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:19Z"} 2024-03-04T18:05:19.720 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:05:19.720 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:05:19.720 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:05:19.720 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.271482_301.1 2024-03-04T18:05:24.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:24.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:24.722 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:24.763 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:24.763 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:24.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:24.816+0000 7f4a8c315980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:24.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:24.816+0000 7f4a8c315980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:24.844 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:24Z"} 2024-03-04T18:05:24.844 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:24Z"} 2024-03-04T18:05:24.845 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T18:05:24.845 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:05:24.845 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T18:05:24.845 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.930202_281.1 2024-03-04T18:05:29.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:29.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:29.847 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:29.890 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:29.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:29.944+0000 7fe942232980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:29.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:29.948+0000 7fe942232980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:29.972 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:29Z"} 2024-03-04T18:05:29.973 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:29Z"} 2024-03-04T18:05:29.973 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:05:29.973 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:05:29.973 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:05:29.973 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:05:34.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:34.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:34.974 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:35.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:35.072+0000 7f0243cc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:35.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:35.072+0000 7f0243cc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:35.099 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:35Z"} 2024-03-04T18:05:35.099 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:35Z"} 2024-03-04T18:05:35.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:05:35.099 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:05:35.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:05:35.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:05:40.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:40.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:40.101 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:40.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:40.199+0000 7ff5cfa82980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:40.199+0000 7ff5cfa82980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:40.227 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:40Z"} 2024-03-04T18:05:40.227 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:40Z"} 2024-03-04T18:05:40.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:05:40.227 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:05:40.227 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:05:40.228 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:05:45.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:45.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:45.228 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:45.271 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:45.271 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:45.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:45.323+0000 7fb71872a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:45.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:45.323+0000 7fb71872a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:45.351 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:45Z"} 2024-03-04T18:05:45.351 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:45Z"} 2024-03-04T18:05:45.351 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:05:45.352 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:05:45.352 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:05:45.352 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:05:50.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:50.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:50.353 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:50.395 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:50.396 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:50.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:50.447+0000 7f15b9bfb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:50.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:50.447+0000 7f15b9bfb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:50.475 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:50Z"} 2024-03-04T18:05:50.475 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:50Z"} 2024-03-04T18:05:50.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:05:50.476 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:05:50.476 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:05:50.476 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T18:05:55.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:55.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:55.477 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:55.520 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:55.520 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:55.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:55.575+0000 7f4c92d00980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:55.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:05:55.583+0000 7f4c92d00980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:55.615 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:55Z"} 2024-03-04T18:05:55.615 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:55Z"} 2024-03-04T18:05:55.615 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T18:05:55.615 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:05:55.616 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T18:05:55.616 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.930202_281.1 2024-03-04T18:06:00.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:00.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:00.617 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:00.659 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:00.659 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:00.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:00.715+0000 7f5de1b70980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:00.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:00.715+0000 7f5de1b70980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:00.746 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:00Z"} 2024-03-04T18:06:00.746 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:00Z"} 2024-03-04T18:06:00.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:06:00.746 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:06:00.746 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:06:00.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:06:05.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:05.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:05.748 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:05.791 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:05.791 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:05.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:05.843+0000 7fbecaf25980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:05.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:05.847+0000 7fbecaf25980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:05.873 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:05Z"} 2024-03-04T18:06:05.873 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:05Z"} 2024-03-04T18:06:05.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:06:05.873 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:06:05.874 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:06:05.874 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T18:06:10.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:10.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:10.875 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:10.917 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:10.917 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:10.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:10.975+0000 7fa56e18b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:10.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:10.975+0000 7fa56e18b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:11.000 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:10Z"} 2024-03-04T18:06:11.001 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:10Z"} 2024-03-04T18:06:11.001 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:06:11.001 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:06:11.001 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:06:11.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.288119_303.1 2024-03-04T18:06:16.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:16.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:16.003 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:16.047 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:16.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:16.102+0000 7f77a5427980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:16.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:16.102+0000 7f77a5427980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:16.132 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:16Z"} 2024-03-04T18:06:16.132 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:16Z"} 2024-03-04T18:06:16.132 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:06:16.132 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:06:16.133 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:06:16.133 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:06:21.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:21.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:21.134 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:21.177 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:21.177 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:21.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:21.234+0000 7feddf602980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:21.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:21.234+0000 7feddf602980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:21.259 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:21Z"} 2024-03-04T18:06:21.259 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:21Z"} 2024-03-04T18:06:21.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:06:21.259 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:06:21.260 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:06:21.260 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:06:26.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:26.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:26.262 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:26.315 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:26.315 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:26.382+0000 7fc35ae40980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:26.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:26.382+0000 7fc35ae40980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:26.408 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:26Z"} 2024-03-04T18:06:26.408 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:26Z"} 2024-03-04T18:06:26.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:06:26.408 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:06:26.408 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:06:26.409 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:06:31.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:31.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:31.410 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:31.452 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:31.453 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:31.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:31.506+0000 7f20e6d10980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:31.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:31.510+0000 7f20e6d10980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:31.533 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:31Z"} 2024-03-04T18:06:31.533 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:31Z"} 2024-03-04T18:06:31.533 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:06:31.534 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:06:31.534 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:06:31.534 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:06:36.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:36.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:36.536 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:36.578 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:36.579 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:36.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:36.638+0000 7efdd3821980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:36.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:36.638+0000 7efdd3821980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:36.663 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:36Z"} 2024-03-04T18:06:36.663 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:36Z"} 2024-03-04T18:06:36.664 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.861873_275.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.861873Z', 'realm_epoch': 2}}] 2024-03-04T18:06:36.664 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:06:36.664 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.861873_275.1'} 2024-03-04T18:06:36.664 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.861873_275.1 2024-03-04T18:06:41.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:41.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:41.666 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:41.708 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:41.708 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:41.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:41.766+0000 7fe9fe1a8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:41.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:41.766+0000 7fe9fe1a8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:41.792 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:41Z"} 2024-03-04T18:06:41.792 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:41Z"} 2024-03-04T18:06:41.792 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:06:41.793 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:06:41.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:06:41.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:06:46.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:46.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:46.794 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:46.844 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:46.844 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:46.914+0000 7ff549a3e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:46.914+0000 7ff549a3e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:46.944 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:46Z"} 2024-03-04T18:06:46.944 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:46Z"} 2024-03-04T18:06:46.944 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:06:46.944 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:06:46.944 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:06:46.944 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:06:51.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:51.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:51.946 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:51.990 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:51.990 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:52.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:52.050+0000 7ffbe6fe8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:52.050+0000 7ffbe6fe8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:52.078 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:52Z"} 2024-03-04T18:06:52.078 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:52Z"} 2024-03-04T18:06:52.078 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:06:52.078 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:06:52.079 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:06:52.079 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:06:57.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:57.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:57.080 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:57.124 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:57.124 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:57.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:57.177+0000 7f2db4fea980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:06:57.181+0000 7f2db4fea980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:57.205 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:57Z"} 2024-03-04T18:06:57.206 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:57Z"} 2024-03-04T18:06:57.206 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:06:57.206 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:06:57.206 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:06:57.206 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:07:02.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:02.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:02.208 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:02.250 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:02.250 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:02.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:02.305+0000 7fc9bf976980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:02.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:02.309+0000 7fc9bf976980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:02.335 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.081961_289.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.081961Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:02Z"} 2024-03-04T18:07:02.335 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.081961_289.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.081961Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:02Z"} 2024-03-04T18:07:02.335 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.081961_289.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.081961Z', 'realm_epoch': 2}}] 2024-03-04T18:07:02.335 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:07:02.336 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.081961_289.1'} 2024-03-04T18:07:02.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.081961_289.1 2024-03-04T18:07:07.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:07.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:07.337 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:07.389 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:07.389 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:07.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:07.449+0000 7fbf361af980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:07.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:07.449+0000 7fbf361af980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:07.476 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:07Z"} 2024-03-04T18:07:07.476 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:07Z"} 2024-03-04T18:07:07.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:07:07.476 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:07:07.477 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:07:07.477 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T18:07:12.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:12.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:12.478 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:12.521 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:12.521 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:12.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:12.577+0000 7f0c1130c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:12.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:12.577+0000 7f0c1130c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:12.603 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:12Z"} 2024-03-04T18:07:12.603 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:12Z"} 2024-03-04T18:07:12.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:07:12.604 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:07:12.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:07:12.604 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T18:07:17.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:17.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:17.605 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:17.651 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:17.651 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:17.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:17.709+0000 7f85b10ce980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:17.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:17.709+0000 7f85b10ce980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:17.737 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.088577_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.088577Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:17Z"} 2024-03-04T18:07:17.737 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.088577_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.088577Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:17Z"} 2024-03-04T18:07:17.737 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.088577_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.088577Z', 'realm_epoch': 2}}] 2024-03-04T18:07:17.737 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:07:17.738 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.088577_290.1'} 2024-03-04T18:07:17.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.088577_290.1 2024-03-04T18:07:22.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:22.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:22.739 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:22.783 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:22.783 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:22.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:22.837+0000 7f5971d60980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:22.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:22.837+0000 7f5971d60980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:22.864 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.088577_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.088577Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:22Z"} 2024-03-04T18:07:22.865 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.088577_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.088577Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:22Z"} 2024-03-04T18:07:22.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.088577_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.088577Z', 'realm_epoch': 2}}] 2024-03-04T18:07:22.865 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:07:22.865 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.088577_290.1'} 2024-03-04T18:07:22.865 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.088577_290.1 2024-03-04T18:07:27.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:27.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:27.867 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:27.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:27.957+0000 7fe6ac772980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:27.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:27.957+0000 7fe6ac772980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:27.985 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:27Z"} 2024-03-04T18:07:27.985 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:27Z"} 2024-03-04T18:07:27.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:07:27.986 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:07:27.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:07:27.986 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:07:32.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:32.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:32.987 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:33.031 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:33.031 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:33.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:33.085+0000 7f01d218c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:33.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:33.085+0000 7f01d218c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:33.112 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:33Z"} 2024-03-04T18:07:33.113 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:33Z"} 2024-03-04T18:07:33.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:07:33.113 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:07:33.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:07:33.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T18:07:36.771 INFO:tasks.rgw.c2.client.0.smithi153.stdout:2024-03-04T18:07:36.769+0000 7fa53962a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:07:38.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:38.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:38.115 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:38.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:38.216+0000 7f93b7b1a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:38.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:38.216+0000 7f93b7b1a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:38.246 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.014763_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.014763Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:38Z"} 2024-03-04T18:07:38.246 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.014763_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.014763Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:38Z"} 2024-03-04T18:07:38.246 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.014763_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.014763Z', 'realm_epoch': 2}}] 2024-03-04T18:07:38.246 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:07:38.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.014763_287.1'} 2024-03-04T18:07:38.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.014763_287.1 2024-03-04T18:07:43.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:43.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:43.249 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:43.292 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:43.292 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:43.344+0000 7f733cc40980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:43.348+0000 7f733cc40980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:43.374 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:43Z"} 2024-03-04T18:07:43.374 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:43Z"} 2024-03-04T18:07:43.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:07:43.374 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:07:43.374 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:07:43.375 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:07:48.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:48.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:48.377 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:48.421 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:48.421 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:48.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:48.472+0000 7f3d67cef980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:48.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:48.476+0000 7f3d67cef980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:48.502 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:48Z"} 2024-03-04T18:07:48.503 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:48Z"} 2024-03-04T18:07:48.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.861873_275.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.861873Z', 'realm_epoch': 2}}] 2024-03-04T18:07:48.503 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:07:48.503 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.861873_275.1'} 2024-03-04T18:07:48.503 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.861873_275.1 2024-03-04T18:07:53.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:53.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:53.504 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:53.547 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:53.547 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:53.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:53.600+0000 7f78d66d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:53.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:53.600+0000 7f78d66d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:53.629 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:53Z"} 2024-03-04T18:07:53.629 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:53Z"} 2024-03-04T18:07:53.629 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:07:53.629 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:07:53.629 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:07:53.630 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:07:58.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:58.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:58.631 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:58.671 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:58.671 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:58.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:58.728+0000 7f8912488980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:58.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:07:58.732+0000 7f8912488980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:58.758 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:58Z"} 2024-03-04T18:07:58.758 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:58Z"} 2024-03-04T18:07:58.759 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:07:58.759 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:07:58.759 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:07:58.759 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T18:08:03.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:03.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:03.761 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:03.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:03.856+0000 7f2a0c393980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:03.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:03.860+0000 7f2a0c393980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:03.886 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:03Z"} 2024-03-04T18:08:03.886 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:03Z"} 2024-03-04T18:08:03.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.984537_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.984537Z', 'realm_epoch': 2}}] 2024-03-04T18:08:03.887 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:08:03.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.984537_285.1'} 2024-03-04T18:08:03.887 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.984537_285.1 2024-03-04T18:08:08.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:08.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:08.888 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:08.939 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:08.939 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:08.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:08.992+0000 7f793548a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:08.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:08.992+0000 7f793548a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:09.020 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.141267_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.141267Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:08Z"} 2024-03-04T18:08:09.020 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.141267_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.141267Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:08Z"} 2024-03-04T18:08:09.020 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.141267_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.141267Z', 'realm_epoch': 2}}] 2024-03-04T18:08:09.020 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T18:08:09.021 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.141267_296.1'} 2024-03-04T18:08:09.021 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.141267_296.1 2024-03-04T18:08:14.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2024-03-04T18:08:14.024 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-45 2024-03-04T18:08:14.024 DEBUG:boto:path=/ppfahn-45/ 2024-03-04T18:08:14.025 DEBUG:boto:auth_path=/ppfahn-45/ 2024-03-04T18:08:14.025 DEBUG:boto:Method: PUT 2024-03-04T18:08:14.025 DEBUG:boto:Path: /ppfahn-45/ 2024-03-04T18:08:14.025 DEBUG:boto:Data: 2024-03-04T18:08:14.025 DEBUG:boto:Headers: {} 2024-03-04T18:08:14.025 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:08:14.025 DEBUG:boto:Port: 8000 2024-03-04T18:08:14.025 DEBUG:boto:Params: {} 2024-03-04T18:08:14.026 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:08:14.026 DEBUG:boto:Token: None 2024-03-04T18:08:14.026 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:08:14 GMT /ppfahn-45/ 2024-03-04T18:08:14.026 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:suSXJ/VJKMFY+GtJ+WlgMhEpxjI= 2024-03-04T18:08:14.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:suSXJ/VJKMFY+GtJ+WlgMhEpxjI=', 'Content-Length': '0'} 2024-03-04T18:08:14.069 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fcf4ea21f29ee96e-0065e60e0e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:08:14.069 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-46 2024-03-04T18:08:14.069 DEBUG:boto:path=/ppfahn-46/ 2024-03-04T18:08:14.069 DEBUG:boto:auth_path=/ppfahn-46/ 2024-03-04T18:08:14.069 DEBUG:boto:Method: PUT 2024-03-04T18:08:14.069 DEBUG:boto:Path: /ppfahn-46/ 2024-03-04T18:08:14.069 DEBUG:boto:Data: 2024-03-04T18:08:14.069 DEBUG:boto:Headers: {} 2024-03-04T18:08:14.070 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:08:14.070 DEBUG:boto:Port: 8000 2024-03-04T18:08:14.070 DEBUG:boto:Params: {} 2024-03-04T18:08:14.070 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:08:14.070 DEBUG:boto:Token: None 2024-03-04T18:08:14.070 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:08:14 GMT /ppfahn-46/ 2024-03-04T18:08:14.070 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FbmbJuALSYhgLTcEmvtG7mvM7fY= 2024-03-04T18:08:14.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FbmbJuALSYhgLTcEmvtG7mvM7fY=', 'Content-Length': '0'} 2024-03-04T18:08:14.149 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c5937fc317071284-0065e60e0e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:08:14.149 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-47 2024-03-04T18:08:14.149 DEBUG:boto:path=/ppfahn-47/ 2024-03-04T18:08:14.150 DEBUG:boto:auth_path=/ppfahn-47/ 2024-03-04T18:08:14.150 DEBUG:boto:Method: PUT 2024-03-04T18:08:14.150 DEBUG:boto:Path: /ppfahn-47/ 2024-03-04T18:08:14.150 DEBUG:boto:Data: 2024-03-04T18:08:14.150 DEBUG:boto:Headers: {} 2024-03-04T18:08:14.150 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:08:14.150 DEBUG:boto:Port: 8001 2024-03-04T18:08:14.150 DEBUG:boto:Params: {} 2024-03-04T18:08:14.150 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T18:08:14.151 DEBUG:boto:Token: None 2024-03-04T18:08:14.151 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:08:14 GMT /ppfahn-47/ 2024-03-04T18:08:14.151 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XEjhPCgV+B80W3he7f9eeXVZVtU= 2024-03-04T18:08:14.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XEjhPCgV+B80W3he7f9eeXVZVtU=', 'Content-Length': '0'} 2024-03-04T18:08:14.319 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000659ded98b03d680e-0065e60e0e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:08:14.319 DEBUG:boto:path=/ppfahn-45/ 2024-03-04T18:08:14.319 DEBUG:boto:auth_path=/ppfahn-45/ 2024-03-04T18:08:14.319 DEBUG:boto:Method: HEAD 2024-03-04T18:08:14.319 DEBUG:boto:Path: /ppfahn-45/ 2024-03-04T18:08:14.319 DEBUG:boto:Data: 2024-03-04T18:08:14.319 DEBUG:boto:Headers: {} 2024-03-04T18:08:14.320 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:08:14.320 DEBUG:boto:Port: 8000 2024-03-04T18:08:14.320 DEBUG:boto:Params: {} 2024-03-04T18:08:14.320 DEBUG:boto:Token: None 2024-03-04T18:08:14.320 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:08:14 GMT /ppfahn-45/ 2024-03-04T18:08:14.320 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XaJ8BxCu51i63Enp7PsthffGI2A= 2024-03-04T18:08:14.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XaJ8BxCu51i63Enp7PsthffGI2A=', 'Content-Length': '0'} 2024-03-04T18:08:14.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b263d024a4802d9-0065e60e0e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:08:14.324 DEBUG:boto:path=/ppfahn-45/key 2024-03-04T18:08:14.325 DEBUG:boto:auth_path=/ppfahn-45/key 2024-03-04T18:08:14.325 DEBUG:boto:Method: PUT 2024-03-04T18:08:14.325 DEBUG:boto:Path: /ppfahn-45/key 2024-03-04T18:08:14.325 DEBUG:boto:Data: 2024-03-04T18:08:14.325 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-03-04T18:08:14.325 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:08:14.325 DEBUG:boto:Port: 8000 2024-03-04T18:08:14.325 DEBUG:boto:Params: {} 2024-03-04T18:08:14.325 DEBUG:boto:Token: None 2024-03-04T18:08:14.325 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Mon, 04 Mar 2024 18:08:14 GMT /ppfahn-45/key 2024-03-04T18:08:14.326 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Qv3fj3rGqiE+lvX2L856dIni0Ec= 2024-03-04T18:08:14.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Qv3fj3rGqiE+lvX2L856dIni0Ec='} 2024-03-04T18:08:14.349 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072efd20f71d2651e-0065e60e0e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:08:14.349 DEBUG:boto:path=/ppfahn-46/ 2024-03-04T18:08:14.349 DEBUG:boto:auth_path=/ppfahn-46/ 2024-03-04T18:08:14.349 DEBUG:boto:Method: HEAD 2024-03-04T18:08:14.349 DEBUG:boto:Path: /ppfahn-46/ 2024-03-04T18:08:14.349 DEBUG:boto:Data: 2024-03-04T18:08:14.350 DEBUG:boto:Headers: {} 2024-03-04T18:08:14.350 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:08:14.350 DEBUG:boto:Port: 8000 2024-03-04T18:08:14.350 DEBUG:boto:Params: {} 2024-03-04T18:08:14.350 DEBUG:boto:Token: None 2024-03-04T18:08:14.350 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:08:14 GMT /ppfahn-46/ 2024-03-04T18:08:14.350 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FxD4ssHhqUiQT2doSGQNgnXkELo= 2024-03-04T18:08:14.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FxD4ssHhqUiQT2doSGQNgnXkELo=', 'Content-Length': '0'} 2024-03-04T18:08:14.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000261a348c6c77c07e-0065e60e0e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:08:14.356 DEBUG:boto:path=/ppfahn-46/key 2024-03-04T18:08:14.356 DEBUG:boto:auth_path=/ppfahn-46/key 2024-03-04T18:08:14.356 DEBUG:boto:Method: PUT 2024-03-04T18:08:14.356 DEBUG:boto:Path: /ppfahn-46/key 2024-03-04T18:08:14.356 DEBUG:boto:Data: 2024-03-04T18:08:14.356 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-03-04T18:08:14.356 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:08:14.356 DEBUG:boto:Port: 8000 2024-03-04T18:08:14.356 DEBUG:boto:Params: {} 2024-03-04T18:08:14.356 DEBUG:boto:Token: None 2024-03-04T18:08:14.357 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Mon, 04 Mar 2024 18:08:14 GMT /ppfahn-46/key 2024-03-04T18:08:14.357 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ex3lukmqPCx5dG5ZNyNgXh9fb5I= 2024-03-04T18:08:14.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ex3lukmqPCx5dG5ZNyNgXh9fb5I='} 2024-03-04T18:08:14.373 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e96d1a884e4426b-0065e60e0e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:08:14.373 DEBUG:boto:path=/ppfahn-47/ 2024-03-04T18:08:14.373 DEBUG:boto:auth_path=/ppfahn-47/ 2024-03-04T18:08:14.373 DEBUG:boto:Method: HEAD 2024-03-04T18:08:14.373 DEBUG:boto:Path: /ppfahn-47/ 2024-03-04T18:08:14.373 DEBUG:boto:Data: 2024-03-04T18:08:14.374 DEBUG:boto:Headers: {} 2024-03-04T18:08:14.374 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:08:14.374 DEBUG:boto:Port: 8001 2024-03-04T18:08:14.374 DEBUG:boto:Params: {} 2024-03-04T18:08:14.374 DEBUG:boto:Token: None 2024-03-04T18:08:14.374 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:08:14 GMT /ppfahn-47/ 2024-03-04T18:08:14.374 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VRlljhMkpkcRxkVkZXnOGLF/alE= 2024-03-04T18:08:14.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VRlljhMkpkcRxkVkZXnOGLF/alE=', 'Content-Length': '0'} 2024-03-04T18:08:14.377 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063dce749e837266b-0065e60e0e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:08:14.377 DEBUG:boto:path=/ppfahn-47/key 2024-03-04T18:08:14.377 DEBUG:boto:auth_path=/ppfahn-47/key 2024-03-04T18:08:14.377 DEBUG:boto:Method: PUT 2024-03-04T18:08:14.378 DEBUG:boto:Path: /ppfahn-47/key 2024-03-04T18:08:14.378 DEBUG:boto:Data: 2024-03-04T18:08:14.378 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-03-04T18:08:14.378 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:08:14.378 DEBUG:boto:Port: 8001 2024-03-04T18:08:14.378 DEBUG:boto:Params: {} 2024-03-04T18:08:14.378 DEBUG:boto:Token: None 2024-03-04T18:08:14.378 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Mon, 04 Mar 2024 18:08:14 GMT /ppfahn-47/key 2024-03-04T18:08:14.378 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hBSNuy382yAt/lM9W3iz8nwMb5c= 2024-03-04T18:08:14.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hBSNuy382yAt/lM9W3iz8nwMb5c='} 2024-03-04T18:08:14.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016c9eaa4e9709bb8-0065e60e0e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:08:14.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:14.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:14.390 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:14.423 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:14.423 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:14.490 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T18:08:14.520 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709575694.259906_393.1","last_update":"2024-03-04T18:08:14.259906Z"}] 2024-03-04T18:08:14.520 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:08:14.520 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:08:14.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:14.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:14.521 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:14.572 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:14.572 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:14.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:14.656+0000 7fadc9c0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:14.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:14.656+0000 7fadc9c0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:14.684 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:14Z"} 2024-03-04T18:08:14.684 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:14Z"} 2024-03-04T18:08:14.684 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:08:14.684 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:08:14.685 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:08:14.685 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:08:16.785 INFO:tasks.rgw.c2.client.0.smithi153.stdout:2024-03-04T18:08:16.780+0000 7fa54263c640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T18:08:19.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:19.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:19.686 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:19.729 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:19.729 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:19.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:19.812+0000 7f435591c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:19.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:19.812+0000 7f435591c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:19.840 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:19Z"} 2024-03-04T18:08:19.840 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:19Z"} 2024-03-04T18:08:19.841 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T18:08:19.841 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:08:19.841 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T18:08:19.841 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.937123_282.1 2024-03-04T18:08:24.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:24.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:24.842 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:24.854 INFO:tasks.rgw.c2.client.0.smithi153.stdout:2024-03-04T18:08:24.851+0000 7fa541e3b640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T18:08:24.890 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:24.890 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:24.947+0000 7f37e6260980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:24.947+0000 7f37e6260980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:24.976 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:24Z"} 2024-03-04T18:08:24.976 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:24Z"} 2024-03-04T18:08:24.976 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:08:24.976 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:08:24.976 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:08:24.976 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:08:29.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:29.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:29.978 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:30.019 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:30.019 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:30.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:30.071+0000 7f2bd1c88980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:30.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:30.071+0000 7f2bd1c88980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:30.102 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:30Z"} 2024-03-04T18:08:30.102 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:30Z"} 2024-03-04T18:08:30.102 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:08:30.102 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:08:30.102 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:08:30.102 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:08:35.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:35.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:35.104 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:35.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:35.199+0000 7f12182de980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:35.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:35.203+0000 7f12182de980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:35.230 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:35Z"} 2024-03-04T18:08:35.230 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:35Z"} 2024-03-04T18:08:35.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T18:08:35.230 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:08:35.230 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T18:08:35.230 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.937123_282.1 2024-03-04T18:08:40.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:40.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:40.232 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:40.276 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:40.276 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:40.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:40.331+0000 7f4d1021c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:40.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:40.331+0000 7f4d1021c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:40.360 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:40Z"} 2024-03-04T18:08:40.361 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:40Z"} 2024-03-04T18:08:40.361 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:08:40.361 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:08:40.361 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:08:40.361 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:08:45.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:45.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:45.363 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:45.406 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:45.406 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:45.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:45.459+0000 7f71c8a66980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:45.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:45.459+0000 7f71c8a66980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:45.489 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:45Z"} 2024-03-04T18:08:45.490 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:45Z"} 2024-03-04T18:08:45.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:08:45.490 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:08:45.490 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:08:45.490 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:08:50.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:50.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:50.492 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:50.535 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:50.536 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:50.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:50.591+0000 7ff7f8511980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:50.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:50.591+0000 7ff7f8511980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:50.620 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:50Z"} 2024-03-04T18:08:50.620 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:50Z"} 2024-03-04T18:08:50.620 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:08:50.620 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:08:50.620 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:08:50.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:08:55.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:55.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:55.623 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:55.665 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:55.665 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:55.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:55.715+0000 7f5a33be4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:55.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:08:55.719+0000 7f5a33be4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:55.747 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:55Z"} 2024-03-04T18:08:55.747 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:55Z"} 2024-03-04T18:08:55.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:08:55.747 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:08:55.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:08:55.748 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:09:00.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:00.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:00.749 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:00.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:00.847+0000 7fe4f351e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:00.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:00.847+0000 7fe4f351e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:00.873 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:00Z"} 2024-03-04T18:09:00.873 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:00Z"} 2024-03-04T18:09:00.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:09:00.873 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:09:00.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:09:00.874 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:09:05.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:05.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:05.875 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:05.918 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:05.918 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:05.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:05.975+0000 7f71bb39e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:05.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:05.975+0000 7f71bb39e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:06.001 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:05Z"} 2024-03-04T18:09:06.001 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:05Z"} 2024-03-04T18:09:06.001 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:09:06.001 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:09:06.001 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:09:06.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:09:11.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:11.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:11.003 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:11.045 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:11.045 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:11.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:11.102+0000 7fbe796ef980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:11.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:11.102+0000 7fbe796ef980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:11.128 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:11Z"} 2024-03-04T18:09:11.128 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:11Z"} 2024-03-04T18:09:11.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:09:11.129 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:09:11.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:09:11.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:09:16.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:16.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:16.130 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:16.173 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:16.173 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:16.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:16.230+0000 7f534205f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:16.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:16.234+0000 7f534205f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:16.260 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:16Z"} 2024-03-04T18:09:16.260 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:16Z"} 2024-03-04T18:09:16.260 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T18:09:16.261 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:09:16.261 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T18:09:16.261 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.937123_282.1 2024-03-04T18:09:21.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:21.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:21.262 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:21.305 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:21.305 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:21.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:21.358+0000 7fd0f249d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:21.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:21.362+0000 7fd0f249d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:21.386 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:21Z"} 2024-03-04T18:09:21.386 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:21Z"} 2024-03-04T18:09:21.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.388355_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.388355Z', 'realm_epoch': 2}}] 2024-03-04T18:09:21.387 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:09:21.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.388355_309.1'} 2024-03-04T18:09:21.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575075.388355_309.1 2024-03-04T18:09:26.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:26.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:26.389 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:26.431 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:26.431 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:26.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:26.486+0000 7fc8ebf3a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:26.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:26.486+0000 7fc8ebf3a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:26.514 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:26Z"} 2024-03-04T18:09:26.514 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:26Z"} 2024-03-04T18:09:26.514 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:09:26.515 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:09:26.515 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:09:26.515 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:09:31.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:31.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:31.516 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:31.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:31.614+0000 7f37afb83980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:31.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:31.614+0000 7f37afb83980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:31.642 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:31Z"} 2024-03-04T18:09:31.642 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:31Z"} 2024-03-04T18:09:31.642 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:09:31.643 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:09:31.643 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:09:31.643 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:09:36.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:36.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:36.644 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:36.688 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:36.688 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:36.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:36.742+0000 7f1a81922980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:36.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:36.742+0000 7f1a81922980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:36.769 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:36Z"} 2024-03-04T18:09:36.769 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:36Z"} 2024-03-04T18:09:36.769 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:09:36.769 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:09:36.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:09:36.770 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:09:41.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:41.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:41.770 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:41.814 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:41.815 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:41.870+0000 7fe559db8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:41.870+0000 7fe559db8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:41.895 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:41Z"} 2024-03-04T18:09:41.896 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:41Z"} 2024-03-04T18:09:41.896 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:09:41.896 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:09:41.896 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:09:41.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:09:46.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:46.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:46.899 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:46.942 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:46.942 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:46.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:46.998+0000 7fcb223f5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:46.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:46.998+0000 7fcb223f5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:47.023 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.014763_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.014763Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:47Z"} 2024-03-04T18:09:47.024 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.014763_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.014763Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:47Z"} 2024-03-04T18:09:47.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.014763_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.014763Z', 'realm_epoch': 2}}] 2024-03-04T18:09:47.024 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:09:47.024 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.014763_287.1'} 2024-03-04T18:09:47.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575075.014763_287.1 2024-03-04T18:09:52.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:52.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:52.026 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:52.069 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:52.069 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:52.126+0000 7fa9dc957980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:52.126+0000 7fa9dc957980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:52Z"} 2024-03-04T18:09:52.154 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:52Z"} 2024-03-04T18:09:52.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:09:52.155 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:09:52.155 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:09:52.155 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:09:57.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:57.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:57.157 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:57.199 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:57.199 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:57.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:57.254+0000 7f30c4aa7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:57.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:09:57.258+0000 7f30c4aa7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:57.282 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:57Z"} 2024-03-04T18:09:57.282 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:57Z"} 2024-03-04T18:09:57.283 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:09:57.283 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:09:57.283 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:09:57.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:10:02.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:02.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:02.284 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:02.324 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:02.324 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:02.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:02.381+0000 7fa50023c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:02.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:02.381+0000 7fa50023c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:02.409 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.973154_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.973154Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:02Z"} 2024-03-04T18:10:02.410 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.973154_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.973154Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:02Z"} 2024-03-04T18:10:02.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.973154_284.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.973154Z', 'realm_epoch': 2}}] 2024-03-04T18:10:02.410 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:10:02.410 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.973154_284.1'} 2024-03-04T18:10:02.410 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.973154_284.1 2024-03-04T18:10:07.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:07.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:07.412 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:07.454 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:07.455 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:07.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:07.509+0000 7f2136e62980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:07.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:07.509+0000 7f2136e62980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:07.537 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.973154_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.973154Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:07Z"} 2024-03-04T18:10:07.537 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.973154_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.973154Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:07Z"} 2024-03-04T18:10:07.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.973154_284.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.973154Z', 'realm_epoch': 2}}] 2024-03-04T18:10:07.538 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:10:07.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.973154_284.1'} 2024-03-04T18:10:07.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.973154_284.1 2024-03-04T18:10:12.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:12.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:12.539 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:12.584 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:12.584 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:12.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:12.637+0000 7faed5d6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:12.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:12.637+0000 7faed5d6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:12.666 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:12Z"} 2024-03-04T18:10:12.666 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:12Z"} 2024-03-04T18:10:12.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:10:12.667 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:10:12.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:10:12.667 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:10:17.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:17.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:17.669 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:17.711 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:17.711 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:17.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:17.765+0000 7f6a06ca5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:17.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:17.765+0000 7f6a06ca5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:17.792 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:17Z"} 2024-03-04T18:10:17.792 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:17Z"} 2024-03-04T18:10:17.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:10:17.793 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:10:17.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:10:17.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:10:22.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:22.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:22.794 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:22.839 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:22.839 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:22.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:22.893+0000 7fb7c4184980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:22.893+0000 7fb7c4184980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:22.921 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:22Z"} 2024-03-04T18:10:22.921 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:22Z"} 2024-03-04T18:10:22.921 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:10:22.921 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:10:22.921 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:10:22.922 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:10:27.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:27.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:27.924 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:27.966 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:27.966 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:28.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:28.021+0000 7f69e7ff2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:28.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:28.021+0000 7f69e7ff2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:28.048 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:28Z"} 2024-03-04T18:10:28.048 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:28Z"} 2024-03-04T18:10:28.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:10:28.049 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:10:28.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:10:28.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:10:33.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:33.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:33.051 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:33.093 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:33.093 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:33.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:33.149+0000 7f6833b7f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:33.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:33.149+0000 7f6833b7f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:33.176 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:33Z"} 2024-03-04T18:10:33.177 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:33Z"} 2024-03-04T18:10:33.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:10:33.177 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:10:33.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:10:33.177 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:10:38.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:38.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:38.178 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:38.223 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:38.223 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:38.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:38.277+0000 7f1b6b560980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:38.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:38.277+0000 7f1b6b560980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:38.303 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:38Z"} 2024-03-04T18:10:38.303 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:38Z"} 2024-03-04T18:10:38.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T18:10:38.304 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:10:38.304 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T18:10:38.304 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.937123_282.1 2024-03-04T18:10:43.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:43.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:43.305 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:43.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:43.401+0000 7fea97301980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:43.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:43.401+0000 7fea97301980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:43.430 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:43Z"} 2024-03-04T18:10:43.430 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:43Z"} 2024-03-04T18:10:43.431 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:10:43.431 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:10:43.431 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:10:43.431 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:10:48.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:48.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:48.433 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:48.476 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:48.476 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:48.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:48.529+0000 7f4ee293d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:48.529+0000 7f4ee293d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:48.559 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:48Z"} 2024-03-04T18:10:48.559 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:48Z"} 2024-03-04T18:10:48.559 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:10:48.559 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:10:48.559 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:10:48.560 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:10:53.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:53.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:53.561 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:53.611 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:53.611 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:53.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:53.672+0000 7fd4e8192980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:53.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:53.672+0000 7fd4e8192980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:53.701 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:53Z"} 2024-03-04T18:10:53.702 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:53Z"} 2024-03-04T18:10:53.702 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T18:10:53.702 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:10:53.702 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T18:10:53.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.937123_282.1 2024-03-04T18:10:58.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:58.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:58.703 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:58.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:58.800+0000 7ff8e005b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:58.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:10:58.800+0000 7ff8e005b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:58.829 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:58Z"} 2024-03-04T18:10:58.829 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:58Z"} 2024-03-04T18:10:58.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:10:58.830 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:10:58.830 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:10:58.830 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:11:03.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:03.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:03.831 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:03.874 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:03.874 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:03.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:03.928+0000 7f196296a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:03.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:03.928+0000 7f196296a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:03.957 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:03Z"} 2024-03-04T18:11:03.958 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:03Z"} 2024-03-04T18:11:03.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:11:03.958 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:11:03.958 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:11:03.958 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:11:08.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:08.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:08.959 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:09.002 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:09.002 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:09.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:09.056+0000 7ff5892cc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:09.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:09.056+0000 7ff5892cc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:09.085 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:09Z"} 2024-03-04T18:11:09.085 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:09Z"} 2024-03-04T18:11:09.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:11:09.086 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:11:09.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:11:09.086 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:11:14.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:14.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:14.088 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:14.130 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:14.130 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:14.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:14.184+0000 7fb4404cf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:14.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:14.188+0000 7fb4404cf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:14.216 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:14Z"} 2024-03-04T18:11:14.216 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:14Z"} 2024-03-04T18:11:14.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T18:11:14.216 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:11:14.216 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T18:11:14.216 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.937123_282.1 2024-03-04T18:11:19.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:19.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:19.218 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:19.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:19.312+0000 7f7779413980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:19.312+0000 7f7779413980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:19.342 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:19Z"} 2024-03-04T18:11:19.342 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:19Z"} 2024-03-04T18:11:19.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:11:19.342 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:11:19.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:11:19.343 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:11:24.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:24.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:24.344 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:24.387 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:24.387 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:24.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:24.444+0000 7f2b6d780980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:24.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:24.444+0000 7f2b6d780980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:24.472 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:24Z"} 2024-03-04T18:11:24.472 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:24Z"} 2024-03-04T18:11:24.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.861873_275.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.861873Z', 'realm_epoch': 2}}] 2024-03-04T18:11:24.472 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:11:24.472 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.861873_275.1'} 2024-03-04T18:11:24.473 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.861873_275.1 2024-03-04T18:11:29.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:29.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:29.474 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:29.516 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:29.516 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:29.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:29.568+0000 7f14272b7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:29.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:29.572+0000 7f14272b7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:29.599 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:29Z"} 2024-03-04T18:11:29.599 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:29Z"} 2024-03-04T18:11:29.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T18:11:29.600 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:11:29.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T18:11:29.600 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.930202_281.1 2024-03-04T18:11:34.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:34.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:34.601 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:34.642 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:34.643 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:34.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:34.696+0000 7ffad7826980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:34.696+0000 7ffad7826980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:34.726 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:34Z"} 2024-03-04T18:11:34.726 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:34Z"} 2024-03-04T18:11:34.726 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:11:34.726 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:11:34.727 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:11:34.727 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:11:39.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:39.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:39.729 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:39.771 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:39.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:39.824+0000 7fb6f6421980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:39.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:39.824+0000 7fb6f6421980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:39.852 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:39Z"} 2024-03-04T18:11:39.853 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:39Z"} 2024-03-04T18:11:39.853 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:11:39.853 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:11:39.853 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:11:39.853 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:11:44.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:44.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:44.855 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:44.907 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:44.908 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:44.964+0000 7f5d41e27980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:44.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:44.964+0000 7f5d41e27980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:44.995 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:44Z"} 2024-03-04T18:11:44.995 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:44Z"} 2024-03-04T18:11:44.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:11:44.995 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:11:44.995 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:11:44.995 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:11:49.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:49.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:49.996 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:50.040 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:50.040 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:50.091+0000 7fa4750b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:50.095+0000 7fa4750b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:50.123 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:50Z"} 2024-03-04T18:11:50.123 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:50Z"} 2024-03-04T18:11:50.123 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T18:11:50.123 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:11:50.123 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T18:11:50.123 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.930202_281.1 2024-03-04T18:11:55.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:55.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:55.124 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:55.168 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:55.168 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:55.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:55.219+0000 7ff48bd75980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:55.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:11:55.219+0000 7ff48bd75980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:55.250 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:55Z"} 2024-03-04T18:11:55.251 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:55Z"} 2024-03-04T18:11:55.251 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:11:55.251 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:11:55.251 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:11:55.251 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:12:00.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:00.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:00.252 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:00.297 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:00.298 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:00.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:00.347+0000 7ff5d419d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:00.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:00.351+0000 7ff5d419d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:00.379 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:00Z"} 2024-03-04T18:12:00.379 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:00Z"} 2024-03-04T18:12:00.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T18:12:00.380 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:12:00.380 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T18:12:00.380 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.930202_281.1 2024-03-04T18:12:05.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:05.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:05.381 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:05.424 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:05.424 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:05.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:05.475+0000 7f5e91167980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:05.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:05.475+0000 7f5e91167980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:05.506 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:05Z"} 2024-03-04T18:12:05.506 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:05Z"} 2024-03-04T18:12:05.506 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:12:05.506 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:12:05.507 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:12:05.507 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:12:10.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:10.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:10.508 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:10.550 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:10.550 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:10.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:10.603+0000 7ff8d1a77980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:10.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:10.603+0000 7ff8d1a77980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:10.633 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:10Z"} 2024-03-04T18:12:10.633 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:10Z"} 2024-03-04T18:12:10.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:12:10.634 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:12:10.634 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:12:10.634 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:12:15.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:15.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:15.636 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:15.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:15.731+0000 7fb741047980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:15.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:15.731+0000 7fb741047980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:15.761 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:15Z"} 2024-03-04T18:12:15.762 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:15Z"} 2024-03-04T18:12:15.762 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:12:15.762 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:12:15.762 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:12:15.762 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:12:20.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:20.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:20.764 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:20.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:20.863+0000 7fea7aa35980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:20.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:20.867+0000 7fea7aa35980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:20.895 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:20Z"} 2024-03-04T18:12:20.895 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:20Z"} 2024-03-04T18:12:20.895 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:12:20.895 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:12:20.895 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:12:20.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:12:25.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:25.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:25.897 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:25.941 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:25.941 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:25.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:25.995+0000 7f16fd211980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:26.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:25.995+0000 7f16fd211980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:26.024 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:26Z"} 2024-03-04T18:12:26.024 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:26Z"} 2024-03-04T18:12:26.025 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:12:26.025 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:12:26.025 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:12:26.025 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:12:31.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:31.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:31.026 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:31.068 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:31.068 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:31.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:31.123+0000 7f42e07e4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:31.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:31.127+0000 7f42e07e4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:31.151 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:31Z"} 2024-03-04T18:12:31.151 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:31Z"} 2024-03-04T18:12:31.151 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:12:31.152 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:12:31.152 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:12:31.152 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:12:36.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:36.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:36.154 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:36.200 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:36.200 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:36.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:36.263+0000 7fbcb68fa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:36.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:36.263+0000 7fbcb68fa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:36.290 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:36Z"} 2024-03-04T18:12:36.290 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:36Z"} 2024-03-04T18:12:36.290 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:12:36.290 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:12:36.291 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:12:36.291 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575075.125374_293.1 2024-03-04T18:12:41.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:41.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:41.292 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:41.333 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:41.333 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:41.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:41.391+0000 7f711f02c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:41.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:41.391+0000 7f711f02c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:41.417 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:41Z"} 2024-03-04T18:12:41.417 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:41Z"} 2024-03-04T18:12:41.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:12:41.417 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:12:41.417 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:12:41.418 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575075.271482_301.1 2024-03-04T18:12:44.584 INFO:tasks.rgw.c2.client.0.smithi153.stdout:2024-03-04T18:12:44.582+0000 7fa53962a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:12:46.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:46.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:46.419 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:46.462 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:46.518+0000 7f3bf34b3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:46.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:46.518+0000 7f3bf34b3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:46.545 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:46Z"} 2024-03-04T18:12:46.545 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:46Z"} 2024-03-04T18:12:46.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:12:46.546 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:12:46.546 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:12:46.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:12:51.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:51.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:51.547 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:51.590 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:51.590 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:51.646+0000 7f3d76c62980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:51.646+0000 7f3d76c62980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:51.673 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:51Z"} 2024-03-04T18:12:51.673 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:51Z"} 2024-03-04T18:12:51.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:12:51.673 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:12:51.673 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:12:51.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:12:56.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:56.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:56.675 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:56.774+0000 7f491cf8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:12:56.774+0000 7f491cf8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:56Z"} 2024-03-04T18:12:56.803 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:56Z"} 2024-03-04T18:12:56.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:12:56.804 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:12:56.804 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:12:56.804 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:13:01.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:01.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:01.805 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:01.847 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:01.847 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:01.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:13:01.902+0000 7fa49161a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:01.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:13:01.902+0000 7fa49161a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:01.930 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:01Z"} 2024-03-04T18:13:01.930 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:01Z"} 2024-03-04T18:13:01.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:13:01.931 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:13:01.931 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:13:01.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:13:06.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:06.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:06.932 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:06.974 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:06.974 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:07.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:13:07.030+0000 7fe6b08ac980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:13:07.034+0000 7fe6b08ac980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:07.059 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:07Z"} 2024-03-04T18:13:07.060 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:07Z"} 2024-03-04T18:13:07.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:13:07.060 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:13:07.060 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:13:07.060 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:13:12.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:12.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:12.062 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:12.104 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:12.105 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:12.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:13:12.166+0000 7fc0bf4a3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:12.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:13:12.166+0000 7fc0bf4a3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:12.192 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:12Z"} 2024-03-04T18:13:12.192 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:12Z"} 2024-03-04T18:13:12.192 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T18:13:12.192 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:13:12.193 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T18:13:12.193 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T18:13:17.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:17.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:17.194 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:17.238 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:17.238 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:17.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:13:17.330+0000 7f5374665980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:17.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:13:17.330+0000 7f5374665980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:17.355 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:17Z"} 2024-03-04T18:13:17.355 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:17Z"} 2024-03-04T18:13:17.355 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T18:13:17.356 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T18:13:17.356 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T18:13:17.356 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T18:13:22.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2024-03-04T18:13:22.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:13:22.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:13:22.361 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T18:13:22.402 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:22.402 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:22.452 INFO:teuthology.orchestra.run.smithi153.stdout:{"id":"185c722f-6c40-48e4-8855-bf605b5498c7","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"1THAB257QDW2GNAH","secret_key":"QoUA0kEYOw3e9ZPsidmlofuWIy510GDX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","notif_pool":"test-zone2.rgw.log:notif"} 2024-03-04T18:13:22.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:13:22.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:13:22.453 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T18:13:22.531 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:22.531 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:22.601 INFO:teuthology.orchestra.run.smithi153.stderr:Sending period to new master zone d96636ac-054d-40e3-90f3-e3912da784b2 2024-03-04T18:13:22.664 INFO:teuthology.orchestra.run.smithi153.stdout:{"id":"f6a09d88-afdd-40f4-a159-dda085c373df","epoch":4,"predecessor_uuid":"aa7942d8-051d-40ca-b86f-359c9c0301cc","sync_status":[],"period_map":{"id":"f6a09d88-afdd-40f4-a159-dda085c373df","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","zones":[{"id":"185c722f-6c40-48e4-8855-bf605b5498c7","name":"test-zone2","endpoints":["http://smithi153.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"c146b41a-1ed1-469f-afe6-2dddc3577925","name":"test-zone3","endpoints":["http://smithi136.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"d96636ac-054d-40e3-90f3-e3912da784b2","name":"test-zone1","endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"185c722f-6c40-48e4-8855-bf605b5498c7","val":1962731440},{"key":"c146b41a-1ed1-469f-afe6-2dddc3577925","val":3710957901},{"key":"d96636ac-054d-40e3-90f3-e3912da784b2","val":2093114680}]},"master_zonegroup":"test-zonegroup","master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","realm_epoch":2} 2024-03-04T18:13:22.665 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2024-03-04T18:14:22.666 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-48 2024-03-04T18:14:22.666 DEBUG:boto:path=/ppfahn-48/ 2024-03-04T18:14:22.666 DEBUG:boto:auth_path=/ppfahn-48/ 2024-03-04T18:14:22.667 DEBUG:boto:Method: PUT 2024-03-04T18:14:22.667 DEBUG:boto:Path: /ppfahn-48/ 2024-03-04T18:14:22.667 DEBUG:boto:Data: 2024-03-04T18:14:22.667 DEBUG:boto:Headers: {} 2024-03-04T18:14:22.667 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:14:22.667 DEBUG:boto:Port: 8000 2024-03-04T18:14:22.667 DEBUG:boto:Params: {} 2024-03-04T18:14:22.667 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:14:22.667 DEBUG:boto:Token: None 2024-03-04T18:14:22.668 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:14:22 GMT /ppfahn-48/ 2024-03-04T18:14:22.668 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eGGUzqY5elGWUKQZm6Zd1kL2BKQ= 2024-03-04T18:14:22.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:14:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eGGUzqY5elGWUKQZm6Zd1kL2BKQ=', 'Content-Length': '0'} 2024-03-04T18:14:22.714 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fccba63d48f35bb7-0065e60f7e-5302-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:14:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:14:22.715 DEBUG:boto:path=/ppfahn-48/testredirect 2024-03-04T18:14:22.715 DEBUG:boto:auth_path=/ppfahn-48/testredirect 2024-03-04T18:14:22.715 DEBUG:boto:Method: PUT 2024-03-04T18:14:22.716 DEBUG:boto:Path: /ppfahn-48/testredirect 2024-03-04T18:14:22.716 DEBUG:boto:Data: 2024-03-04T18:14:22.716 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-03-04T18:14:22.716 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:14:22.716 DEBUG:boto:Port: 8000 2024-03-04T18:14:22.717 DEBUG:boto:Params: {} 2024-03-04T18:14:22.717 DEBUG:boto:Token: None 2024-03-04T18:14:22.717 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Mon, 04 Mar 2024 18:14:22 GMT /ppfahn-48/testredirect 2024-03-04T18:14:22.717 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oe9XnGFVLFRLI+s4jED2Q2uxQyk= 2024-03-04T18:14:22.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:14:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oe9XnGFVLFRLI+s4jED2Q2uxQyk='} 2024-03-04T18:14:22.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a204acfd39fd5a3-0065e60f7e-5302-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:14:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:14:22.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:22.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:22.740 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:22.780 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:22.783 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:22.843 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T18:14:22.871 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709576062.712249_431.1","last_update":"2024-03-04T18:14:22.712249Z"}] 2024-03-04T18:14:22.872 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:14:22.872 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:14:22.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:22.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:22.873 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:22.981+0000 7fc75bbea980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:22.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:22.985+0000 7fc75bbea980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:23.011 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:22Z"} 2024-03-04T18:14:23.011 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:22Z"} 2024-03-04T18:14:23.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T18:14:23.011 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:14:23.011 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T18:14:23.011 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T18:14:28.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:28.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:28.013 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:28.057 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:28.057 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:28.113+0000 7f99b03af980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:28.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:28.113+0000 7f99b03af980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:28.141 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:28Z"} 2024-03-04T18:14:28.141 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:28Z"} 2024-03-04T18:14:28.141 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:14:28.141 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:14:28.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:14:28.142 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:14:33.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:33.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:33.143 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:33.186 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:33.186 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:33.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:33.241+0000 7f8ce1aa9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:33.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:33.241+0000 7f8ce1aa9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:33.269 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:33Z"} 2024-03-04T18:14:33.269 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:33Z"} 2024-03-04T18:14:33.269 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T18:14:33.269 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:14:33.269 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T18:14:33.269 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T18:14:38.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:38.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:38.271 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:38.314 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:38.315 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:38.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:38.373+0000 7f518ac4c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:38.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:38.377+0000 7f518ac4c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:38.405 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:38Z"} 2024-03-04T18:14:38.405 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:38Z"} 2024-03-04T18:14:38.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:14:38.405 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:14:38.405 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:14:38.405 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:14:43.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:43.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:43.407 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:43.450 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:43.450 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:43.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:43.504+0000 7f434df46980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:43.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:43.504+0000 7f434df46980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:43.534 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:43Z"} 2024-03-04T18:14:43.534 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:43Z"} 2024-03-04T18:14:43.534 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T18:14:43.534 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:14:43.534 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T18:14:43.534 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T18:14:48.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:48.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:48.536 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:48.580 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:48.580 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:48.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:48.636+0000 7f4b6f078980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:48.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:48.636+0000 7f4b6f078980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:48.665 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:48Z"} 2024-03-04T18:14:48.665 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:48Z"} 2024-03-04T18:14:48.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T18:14:48.665 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:14:48.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T18:14:48.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.380518_307.1 2024-03-04T18:14:53.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:53.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:53.667 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:53.710 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:53.710 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:53.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:53.764+0000 7fee696d2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:53.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:53.764+0000 7fee696d2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:53.792 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:53Z"} 2024-03-04T18:14:53.792 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:53Z"} 2024-03-04T18:14:53.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T18:14:53.793 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:14:53.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T18:14:53.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.380518_307.1 2024-03-04T18:14:54.020 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:14:54.019+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:14:54.121 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:14:54.119+0000 7f5c190f6640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:14:54.163 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:14:54.163+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:14:54.165 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:14:54.163+0000 7f5c29116640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T18:14:58.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:58.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:58.794 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:58.836 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:58.836 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:58.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:58.888+0000 7fa58dd0f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:58.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:14:58.892+0000 7fa58dd0f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:58.918 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:58Z"} 2024-03-04T18:14:58.918 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:58Z"} 2024-03-04T18:14:58.918 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:14:58.919 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:14:58.919 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:14:58.919 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:15:03.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:03.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:03.920 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:03.962 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:03.963 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:04.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:04.016+0000 7f0ebeded980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:04.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:04.016+0000 7f0ebeded980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:04.044 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:04Z"} 2024-03-04T18:15:04.045 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:04Z"} 2024-03-04T18:15:04.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:15:04.045 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:15:04.045 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:15:04.045 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:15:09.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:09.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:09.046 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:09.089 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:09.089 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:09.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:09.144+0000 7f0205354980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:09.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:09.144+0000 7f0205354980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:09.173 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.417189_311.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.417189Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:09Z"} 2024-03-04T18:15:09.173 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.417189_311.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.417189Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:09Z"} 2024-03-04T18:15:09.173 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.417189_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.417189Z', 'realm_epoch': 2}}] 2024-03-04T18:15:09.174 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:15:09.174 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.417189_311.1'} 2024-03-04T18:15:09.174 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.417189_311.1 2024-03-04T18:15:14.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:14.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:14.175 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:14.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:14.272+0000 7f8b934ec980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:14.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:14.276+0000 7f8b934ec980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:14.307 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.837105_354.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.837105Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:14Z"} 2024-03-04T18:15:14.307 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.837105_354.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.837105Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:14Z"} 2024-03-04T18:15:14.307 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575385.837105_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:03:05.837105Z', 'realm_epoch': 2}}] 2024-03-04T18:15:14.307 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:15:14.307 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575385.837105_354.1'} 2024-03-04T18:15:14.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575385.837105_354.1 2024-03-04T18:15:19.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:19.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:19.309 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:19.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:19.404+0000 7f43df131980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:19.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:19.408+0000 7f43df131980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:19.435 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:19Z"} 2024-03-04T18:15:19.435 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:19Z"} 2024-03-04T18:15:19.435 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:15:19.435 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:15:19.435 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:15:19.435 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:15:24.140 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:15:24.139+0000 7fae3197f640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:15:24.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:24.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:24.437 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:24.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:24.532+0000 7fcd93736980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:24.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:24.532+0000 7fcd93736980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:24.559 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:24Z"} 2024-03-04T18:15:24.560 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:24Z"} 2024-03-04T18:15:24.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:15:24.560 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:15:24.560 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:15:24.560 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:15:29.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:29.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:29.562 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:29.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:29.660+0000 7fc328164980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:29.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:29.660+0000 7fc328164980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:29.690 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:29Z"} 2024-03-04T18:15:29.691 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:29Z"} 2024-03-04T18:15:29.691 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T18:15:29.691 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:15:29.691 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T18:15:29.691 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T18:15:34.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:34.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:34.692 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:34.735 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:34.735 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:34.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:34.787+0000 7f612edfd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:34.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:34.787+0000 7f612edfd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:34.818 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:34Z"} 2024-03-04T18:15:34.818 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:34Z"} 2024-03-04T18:15:34.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:15:34.818 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:15:34.819 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:15:34.819 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:15:39.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:39.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:39.820 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:39.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:39.919+0000 7fc900758980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:39.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:39.919+0000 7fc900758980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:39.949 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:39Z"} 2024-03-04T18:15:39.949 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:39Z"} 2024-03-04T18:15:39.949 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:15:39.950 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:15:39.950 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:15:39.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:15:44.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:44.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:44.951 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:44.993 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:44.993 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:45.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:45.043+0000 7f2a8b950980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:45.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:45.047+0000 7f2a8b950980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:45.075 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:45Z"} 2024-03-04T18:15:45.075 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:45Z"} 2024-03-04T18:15:45.076 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:15:45.076 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:15:45.076 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:15:45.076 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:15:50.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:50.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:50.078 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:50.120 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:50.120 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:50.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:50.179+0000 7f2306e61980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:50.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:50.183+0000 7f2306e61980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:50.232 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:50Z"} 2024-03-04T18:15:50.232 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:50Z"} 2024-03-04T18:15:50.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:15:50.233 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:15:50.233 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:15:50.233 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:15:55.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:55.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:55.234 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:55.279 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:55.279 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:55.331+0000 7f62ff4b4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:15:55.331+0000 7f62ff4b4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:55.363 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:55Z"} 2024-03-04T18:15:55.363 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:55Z"} 2024-03-04T18:15:55.363 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:15:55.363 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:15:55.363 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:15:55.363 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:16:00.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:00.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:00.365 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:00.407 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:00.408 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:00.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:00.459+0000 7f35ac25f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:00.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:00.463+0000 7f35ac25f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:00.491 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:00Z"} 2024-03-04T18:16:00.491 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:00Z"} 2024-03-04T18:16:00.492 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T18:16:00.492 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:16:00.492 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T18:16:00.492 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T18:16:05.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:05.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:05.494 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:05.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:05.595+0000 7f01efabd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:05.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:05.599+0000 7f01efabd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:05.626 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:05Z"} 2024-03-04T18:16:05.626 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:05Z"} 2024-03-04T18:16:05.627 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:16:05.627 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:16:05.627 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:16:05.627 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:16:10.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:10.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:10.628 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:10.670 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:10.671 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:10.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:10.727+0000 7f20a7dc0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:10.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:10.731+0000 7f20a7dc0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:10.760 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:10Z"} 2024-03-04T18:16:10.760 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:10Z"} 2024-03-04T18:16:10.760 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:16:10.760 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:16:10.760 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:16:10.760 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:16:15.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:15.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:15.762 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:15.806 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:15.806 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:15.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:15.863+0000 7f9e2db7b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:15.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:15.863+0000 7f9e2db7b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:15.890 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:15Z"} 2024-03-04T18:16:15.890 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:15Z"} 2024-03-04T18:16:15.891 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:16:15.891 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:16:15.891 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:16:15.891 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:16:20.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:20.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:20.893 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:20.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:20.999+0000 7f756c47f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:20.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:20.999+0000 7f756c47f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:21.024 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:21Z"} 2024-03-04T18:16:21.025 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:21Z"} 2024-03-04T18:16:21.025 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:16:21.025 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:16:21.025 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:16:21.025 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:16:26.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:26.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:26.027 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:26.070 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:26.071 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:26.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:26.127+0000 7fa8c0b26980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:26.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:26.127+0000 7fa8c0b26980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:26.155 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:26Z"} 2024-03-04T18:16:26.155 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:26Z"} 2024-03-04T18:16:26.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:16:26.155 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:16:26.156 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:16:26.156 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:16:31.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:31.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:31.157 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:31.199 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:31.200 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:31.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:31.258+0000 7f8dcf7e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:31.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:31.258+0000 7f8dcf7e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:31.284 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:31Z"} 2024-03-04T18:16:31.284 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:31Z"} 2024-03-04T18:16:31.284 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T18:16:31.285 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:16:31.285 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T18:16:31.285 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T18:16:36.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:36.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:36.287 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:36.332 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:36.333 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:36.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:36.398+0000 7f1914182980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:36.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:36.402+0000 7f1914182980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:36.427 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:36Z"} 2024-03-04T18:16:36.427 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:36Z"} 2024-03-04T18:16:36.427 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:16:36.427 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:16:36.427 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:16:36.427 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:16:41.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:41.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:41.429 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:41.471 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:41.472 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:41.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:41.526+0000 7facefb90980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:41.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:41.530+0000 7facefb90980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:41.554 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:41Z"} 2024-03-04T18:16:41.555 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:41Z"} 2024-03-04T18:16:41.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:16:41.555 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:16:41.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:16:41.555 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:16:46.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:46.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:46.557 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:46.600 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:46.600 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:46.658+0000 7f2d5c88d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:46.658+0000 7f2d5c88d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:46.683 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:46Z"} 2024-03-04T18:16:46.684 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:46Z"} 2024-03-04T18:16:46.684 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:16:46.684 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:16:46.684 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:16:46.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:16:51.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:51.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:51.686 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:51.736 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:51.737 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:51.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:51.790+0000 7fb13caa3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:51.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:51.794+0000 7fb13caa3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:51.819 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:51Z"} 2024-03-04T18:16:51.820 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:51Z"} 2024-03-04T18:16:51.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:16:51.820 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:16:51.820 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:16:51.820 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:16:56.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:56.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:56.822 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:56.864 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:56.922+0000 7f61b1306980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:16:56.922+0000 7f61b1306980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:56.948 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:56Z"} 2024-03-04T18:16:56.948 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:56Z"} 2024-03-04T18:16:56.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:16:56.949 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:16:56.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:16:56.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:17:01.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:01.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:01.950 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:01.991 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:01.991 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:02.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:02.046+0000 7f40f20dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:02.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:02.046+0000 7f40f20dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:02.074 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:02Z"} 2024-03-04T18:17:02.074 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:02Z"} 2024-03-04T18:17:02.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:17:02.074 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:17:02.074 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:17:02.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:17:07.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:07.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:07.076 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:07.118 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:07.118 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:07.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:07.174+0000 7f570aa8e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:07.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:07.174+0000 7f570aa8e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:07.214 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:07Z"} 2024-03-04T18:17:07.214 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:07Z"} 2024-03-04T18:17:07.215 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:17:07.215 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:17:07.215 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:17:07.215 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:17:12.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:12.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:12.216 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:12.261 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:12.261 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:12.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:12.322+0000 7f9359bc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:12.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:12.322+0000 7f9359bc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:12.349 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:12Z"} 2024-03-04T18:17:12.350 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:12Z"} 2024-03-04T18:17:12.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:17:12.350 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:17:12.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:17:12.350 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:17:17.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:17.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:17.351 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:17.396 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:17.397 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:17.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:17.454+0000 7f5e2cda7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:17.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:17.454+0000 7f5e2cda7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:17.481 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:17Z"} 2024-03-04T18:17:17.482 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:17Z"} 2024-03-04T18:17:17.482 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.495723_318.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.495723Z', 'realm_epoch': 2}}] 2024-03-04T18:17:17.482 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:17:17.482 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.495723_318.1'} 2024-03-04T18:17:17.482 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.495723_318.1 2024-03-04T18:17:22.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:22.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:22.484 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:22.526 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:22.526 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:22.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:22.581+0000 7ff6af76d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:22.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:22.585+0000 7ff6af76d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:22.612 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:22Z"} 2024-03-04T18:17:22.612 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:22Z"} 2024-03-04T18:17:22.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:17:22.612 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:17:22.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:17:22.613 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:17:27.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:27.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:27.614 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:27.666 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:27.666 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:27.733+0000 7fc9d3e8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:27.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:27.733+0000 7fc9d3e8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:27.762 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:27Z"} 2024-03-04T18:17:27.762 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:27Z"} 2024-03-04T18:17:27.762 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:17:27.763 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:17:27.763 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:17:27.763 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:17:32.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:32.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:32.764 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:32.806 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:32.806 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:32.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:32.861+0000 7f4cafd4d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:32.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:32.865+0000 7f4cafd4d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:32.891 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:32Z"} 2024-03-04T18:17:32.891 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:32Z"} 2024-03-04T18:17:32.891 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:17:32.891 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:17:32.892 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:17:32.892 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:17:37.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:37.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:37.893 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:37.935 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:37.935 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:37.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:37.993+0000 7f4f24a9d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:37.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:37.993+0000 7f4f24a9d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:38.022 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:37Z"} 2024-03-04T18:17:38.022 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:37Z"} 2024-03-04T18:17:38.022 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:17:38.023 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:17:38.023 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:17:38.023 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:17:43.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:43.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:43.024 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:43.069 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:43.069 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:43.125+0000 7fdc78614980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:43.125+0000 7fdc78614980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:43.155 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:43Z"} 2024-03-04T18:17:43.156 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:43Z"} 2024-03-04T18:17:43.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T18:17:43.156 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:17:43.156 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T18:17:43.156 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.380518_307.1 2024-03-04T18:17:48.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:48.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:48.158 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:48.257+0000 7f223d44c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:48.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:48.257+0000 7f223d44c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:48.285 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.480321_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.480321Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:48Z"} 2024-03-04T18:17:48.286 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.480321_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.480321Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:48Z"} 2024-03-04T18:17:48.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.480321_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.480321Z', 'realm_epoch': 2}}] 2024-03-04T18:17:48.286 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:17:48.286 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.480321_315.1'} 2024-03-04T18:17:48.286 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.480321_315.1 2024-03-04T18:17:53.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:53.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:53.287 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:53.330 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:53.330 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:53.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:53.385+0000 7fa1a56a0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:53.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:53.385+0000 7fa1a56a0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:53.413 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:53Z"} 2024-03-04T18:17:53.413 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:53Z"} 2024-03-04T18:17:53.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T18:17:53.414 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:17:53.414 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T18:17:53.414 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.380518_307.1 2024-03-04T18:17:58.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:58.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:58.416 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:58.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:58.513+0000 7f9e99093980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:58.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:17:58.517+0000 7f9e99093980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:58.543 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.417189_311.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.417189Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:58Z"} 2024-03-04T18:17:58.543 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.417189_311.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.417189Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:58Z"} 2024-03-04T18:17:58.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.417189_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.417189Z', 'realm_epoch': 2}}] 2024-03-04T18:17:58.544 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:17:58.544 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.417189_311.1'} 2024-03-04T18:17:58.545 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.417189_311.1 2024-03-04T18:18:03.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:03.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:03.546 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:03.589 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:03.590 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:03.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:03.645+0000 7f2d9fc28980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:03.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:03.645+0000 7f2d9fc28980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:03.673 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:03Z"} 2024-03-04T18:18:03.674 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:03Z"} 2024-03-04T18:18:03.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:18:03.674 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:18:03.674 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:18:03.674 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:18:08.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:08.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:08.676 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:08.720 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:08.720 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:08.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:08.777+0000 7f3b41f4d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:08.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:08.777+0000 7f3b41f4d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:08.806 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:08Z"} 2024-03-04T18:18:08.807 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:08Z"} 2024-03-04T18:18:08.807 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:18:08.807 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:18:08.807 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:18:08.807 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:18:13.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:13.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:13.808 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:13.851 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:13.851 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:13.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:13.909+0000 7f5d54a94980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:13.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:13.909+0000 7f5d54a94980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:13.937 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:13Z"} 2024-03-04T18:18:13.937 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:13Z"} 2024-03-04T18:18:13.937 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:18:13.937 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:18:13.937 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:18:13.937 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:18:18.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:18.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:18.939 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:18.982 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:18.982 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:19.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:19.036+0000 7fa4dd524980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:19.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:19.036+0000 7fa4dd524980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:19.067 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:19Z"} 2024-03-04T18:18:19.067 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:19Z"} 2024-03-04T18:18:19.067 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:18:19.067 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:18:19.068 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:18:19.068 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:18:24.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:24.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:24.069 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:24.111 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:24.111 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:24.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:24.172+0000 7f32dedee980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:24.172+0000 7f32dedee980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:24.207 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:24Z"} 2024-03-04T18:18:24.207 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:24Z"} 2024-03-04T18:18:24.208 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:18:24.208 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:18:24.208 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:18:24.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:18:29.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:29.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:29.209 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:29.251 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:29.252 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:29.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:29.308+0000 7f373f0f7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:29.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:29.308+0000 7f373f0f7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:29.339 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:29Z"} 2024-03-04T18:18:29.339 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:29Z"} 2024-03-04T18:18:29.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:18:29.339 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:18:29.339 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:18:29.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:18:34.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:34.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:34.341 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:34.381 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:34.381 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:34.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:34.436+0000 7fa2be5fb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:34.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:34.436+0000 7fa2be5fb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:34.466 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:34Z"} 2024-03-04T18:18:34.466 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:34Z"} 2024-03-04T18:18:34.466 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:18:34.466 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:18:34.467 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:18:34.467 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:18:39.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:39.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:39.468 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:39.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:39.564+0000 7f1077fab980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:39.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:39.568+0000 7f1077fab980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:39.610 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:39Z"} 2024-03-04T18:18:39.610 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:39Z"} 2024-03-04T18:18:39.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:18:39.611 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:18:39.611 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:18:39.611 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:18:44.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:44.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:44.612 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:44.654 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:44.654 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:44.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:44.708+0000 7f0a1b60e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:44.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:44.712+0000 7f0a1b60e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:44.740 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:44Z"} 2024-03-04T18:18:44.740 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:44Z"} 2024-03-04T18:18:44.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:18:44.741 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:18:44.741 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:18:44.741 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:18:49.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:49.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:49.742 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:49.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:49.868+0000 7f7dcbc4b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:49.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:49.868+0000 7f7dcbc4b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:49.897 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:49Z"} 2024-03-04T18:18:49.897 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:49Z"} 2024-03-04T18:18:49.898 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:18:49.898 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:18:49.898 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:18:49.898 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:18:54.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:54.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:54.899 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:54.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:54.996+0000 7f47a5703980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:55.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:18:54.996+0000 7f47a5703980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:55.027 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:55Z"} 2024-03-04T18:18:55.027 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:55Z"} 2024-03-04T18:18:55.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:18:55.027 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:18:55.027 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:18:55.027 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:19:00.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:00.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:00.029 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:00.073 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:00.073 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:00.140+0000 7f57ab06e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:00.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:00.140+0000 7f57ab06e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:00.166 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:00Z"} 2024-03-04T18:19:00.167 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:00Z"} 2024-03-04T18:19:00.167 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:19:00.167 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:19:00.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:19:00.167 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:19:05.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:05.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:05.169 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:05.212 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:05.212 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:05.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:05.268+0000 7f3e343c6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:05.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:05.268+0000 7f3e343c6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:05.296 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:05Z"} 2024-03-04T18:19:05.296 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:05Z"} 2024-03-04T18:19:05.297 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:19:05.297 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:19:05.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:19:05.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:19:10.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:10.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:10.298 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:10.341 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:10.341 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:10.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:10.403+0000 7fc0e8649980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:10.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:10.403+0000 7fc0e8649980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:10.439 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:10Z"} 2024-03-04T18:19:10.440 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:10Z"} 2024-03-04T18:19:10.440 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:19:10.440 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:19:10.440 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:19:10.440 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:19:15.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:15.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:15.442 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:15.485 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:15.486 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:15.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:15.539+0000 7f3f4a1ad980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:15.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:15.543+0000 7f3f4a1ad980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:15.571 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:15Z"} 2024-03-04T18:19:15.571 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:15Z"} 2024-03-04T18:19:15.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:19:15.572 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:19:15.572 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:19:15.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:19:20.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:20.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:20.574 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:20.617 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:20.617 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:20.671+0000 7fddb27bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:20.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:20.671+0000 7fddb27bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:20.702 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:20Z"} 2024-03-04T18:19:20.702 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:20Z"} 2024-03-04T18:19:20.702 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:19:20.702 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:19:20.702 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:19:20.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:19:25.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:25.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:25.704 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:25.748 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:25.749 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:25.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:25.803+0000 7ff13f5d4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:25.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:25.803+0000 7ff13f5d4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:25.832 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:25Z"} 2024-03-04T18:19:25.832 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:25Z"} 2024-03-04T18:19:25.832 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:19:25.832 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:19:25.833 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:19:25.833 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:19:30.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2024-03-04T18:19:30.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:30.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:30.837 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:30.879 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:30.879 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:30.947 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T18:19:30.972 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709576062.712249_431.1","last_update":"2024-03-04T18:14:22.712249Z"}] 2024-03-04T18:19:30.972 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:19:30.972 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:19:30.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:30.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:30.972 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:31.016 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:31.016 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:31.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:31.071+0000 7f32e4fdf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:31.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:31.071+0000 7f32e4fdf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:31.101 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:31Z"} 2024-03-04T18:19:31.101 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:31Z"} 2024-03-04T18:19:31.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:19:31.101 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:19:31.102 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:19:31.102 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:19:36.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:36.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:36.103 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:36.146 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:36.146 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:36.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:36.203+0000 7ff91ac11980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:36.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:36.203+0000 7ff91ac11980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:36.235 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:36Z"} 2024-03-04T18:19:36.235 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:36Z"} 2024-03-04T18:19:36.235 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:19:36.236 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:19:36.236 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:19:36.236 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:19:41.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:41.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:41.237 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:41.280 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:41.280 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:41.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:41.335+0000 7fc40d495980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:41.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:41.335+0000 7fc40d495980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:41.365 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:41Z"} 2024-03-04T18:19:41.365 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:41Z"} 2024-03-04T18:19:41.365 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:19:41.365 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:19:41.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:19:41.365 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:19:46.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:46.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:46.367 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:46.404 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:46.404 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:46.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:46.459+0000 7fd828820980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:46.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:46.459+0000 7fd828820980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:46.491 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:46Z"} 2024-03-04T18:19:46.491 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:46Z"} 2024-03-04T18:19:46.491 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:19:46.491 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:19:46.492 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:19:46.492 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:19:51.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:51.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:51.493 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:51.536 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:51.536 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:51.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:51.595+0000 7f0070924980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:51.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:51.595+0000 7f0070924980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:51.620 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:51Z"} 2024-03-04T18:19:51.620 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:51Z"} 2024-03-04T18:19:51.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:19:51.621 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:19:51.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:19:51.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:19:54.031 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:19:54.033+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:19:54.171 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:19:54.173+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:19:54.178 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:19:54.181+0000 7f5c190f6640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:19:56.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:56.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:56.622 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:56.661 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:56.662 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:56.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:56.723+0000 7f1745879980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:56.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:19:56.723+0000 7f1745879980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:56.748 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:56Z"} 2024-03-04T18:19:56.748 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:56Z"} 2024-03-04T18:19:56.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:19:56.749 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:19:56.749 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:19:56.749 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:20:01.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:01.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:01.750 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:01.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:01.851+0000 7f119966d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:01.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:01.851+0000 7f119966d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:01.878 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:01Z"} 2024-03-04T18:20:01.878 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:01Z"} 2024-03-04T18:20:01.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:20:01.878 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:20:01.878 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:20:01.879 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:20:06.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:06.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:06.880 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:06.923 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:06.923 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:06.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:06.978+0000 7fb700f1b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:06.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:06.982+0000 7fb700f1b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:07.007 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:06Z"} 2024-03-04T18:20:07.007 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:06Z"} 2024-03-04T18:20:07.007 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:20:07.007 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:20:07.007 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:20:07.008 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:20:12.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:12.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:12.009 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:12.052 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:12.052 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:12.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:12.110+0000 7f85e6c5d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:12.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:12.110+0000 7f85e6c5d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:12.141 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:12Z"} 2024-03-04T18:20:12.141 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:12Z"} 2024-03-04T18:20:12.141 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:20:12.142 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:20:12.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:20:12.142 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:20:17.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:17.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:17.144 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:17.186 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:17.186 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:17.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:17.242+0000 7faecbb66980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:17.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:17.246+0000 7faecbb66980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:17.271 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:17Z"} 2024-03-04T18:20:17.271 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:17Z"} 2024-03-04T18:20:17.271 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T18:20:17.271 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:20:17.271 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T18:20:17.271 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T18:20:22.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:22.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:22.273 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:22.315 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:22.316 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:22.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:22.374+0000 7f15cddc4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:22.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:22.374+0000 7f15cddc4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:22.401 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:22Z"} 2024-03-04T18:20:22.401 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:22Z"} 2024-03-04T18:20:22.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:20:22.402 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:20:22.402 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:20:22.402 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:20:24.161 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:20:24.164+0000 7fae3197f640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:20:27.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:27.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:27.403 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:27.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:27.502+0000 7f3660310980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:27.506+0000 7f3660310980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:27.531 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:27Z"} 2024-03-04T18:20:27.531 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:27Z"} 2024-03-04T18:20:27.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:20:27.531 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:20:27.532 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:20:27.532 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:20:32.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:32.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:32.533 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:32.576 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:32.576 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:32.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:32.630+0000 7f3c30ceb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:32.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:32.634+0000 7f3c30ceb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:32.663 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:32Z"} 2024-03-04T18:20:32.663 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:32Z"} 2024-03-04T18:20:32.663 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:20:32.663 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:20:32.664 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:20:32.664 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:20:37.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:37.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:37.665 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:37.708 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:37.708 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:37.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:37.766+0000 7f554c248980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:37.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:37.766+0000 7f554c248980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:37.792 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:37Z"} 2024-03-04T18:20:37.792 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:37Z"} 2024-03-04T18:20:37.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:20:37.793 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:20:37.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:20:37.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:20:42.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:42.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:42.794 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:42.837 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:42.837 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:42.894+0000 7fba8e06a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:42.898+0000 7fba8e06a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:42.924 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:42Z"} 2024-03-04T18:20:42.924 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:42Z"} 2024-03-04T18:20:42.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:20:42.925 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:20:42.925 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:20:42.925 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:20:47.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:47.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:47.926 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:47.974 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:47.975 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:48.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:48.030+0000 7f938ba84980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:48.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:48.034+0000 7f938ba84980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:48.058 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:48Z"} 2024-03-04T18:20:48.058 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:48Z"} 2024-03-04T18:20:48.059 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:20:48.059 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:20:48.059 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:20:48.059 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:20:53.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:53.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:53.061 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:53.105 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:53.105 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:53.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:53.166+0000 7f39f191e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:53.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:53.166+0000 7f39f191e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:53.192 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:53Z"} 2024-03-04T18:20:53.192 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:53Z"} 2024-03-04T18:20:53.192 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:20:53.193 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:20:53.193 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:20:53.193 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:20:58.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:58.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:58.195 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:58.252 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:58.252 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:58.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:58.313+0000 7f20b08ab980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:58.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:20:58.313+0000 7f20b08ab980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:58.341 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:58Z"} 2024-03-04T18:20:58.341 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:58Z"} 2024-03-04T18:20:58.341 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:20:58.341 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:20:58.341 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:20:58.341 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:21:03.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:03.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:03.342 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:03.386 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:03.386 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:03.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:03.445+0000 7f283c79b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:03.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:03.445+0000 7f283c79b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:03.472 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:03Z"} 2024-03-04T18:21:03.472 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:03Z"} 2024-03-04T18:21:03.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:21:03.473 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:21:03.473 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:21:03.473 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:21:08.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:08.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:08.475 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:08.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:08.573+0000 7f618005f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:08.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:08.577+0000 7f618005f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:08.602 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:08Z"} 2024-03-04T18:21:08.602 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:08Z"} 2024-03-04T18:21:08.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T18:21:08.603 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:21:08.603 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T18:21:08.603 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.380518_307.1 2024-03-04T18:21:13.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:13.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:13.605 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:13.647 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:13.647 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:13.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:13.701+0000 7fc3b6b0f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:13.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:13.705+0000 7fc3b6b0f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:13.732 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:13Z"} 2024-03-04T18:21:13.733 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:13Z"} 2024-03-04T18:21:13.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:21:13.733 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:21:13.733 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:21:13.733 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:21:18.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:18.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:18.735 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:18.777 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:18.777 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:18.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:18.833+0000 7fca02968980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:18.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:18.837+0000 7fca02968980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:18.863 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:18Z"} 2024-03-04T18:21:18.864 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:18Z"} 2024-03-04T18:21:18.864 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:21:18.864 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:21:18.864 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:21:18.864 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:21:23.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:23.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:23.866 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:23.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:23.965+0000 7f4f60cb2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:23.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:23.965+0000 7f4f60cb2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:23.993 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:23Z"} 2024-03-04T18:21:23.993 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:23Z"} 2024-03-04T18:21:23.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:21:23.994 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:21:23.994 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:21:23.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:21:28.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:28.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:28.995 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:29.038 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:29.038 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:29.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:29.105+0000 7f2672746980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:29.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:29.105+0000 7f2672746980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:29.135 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:29Z"} 2024-03-04T18:21:29.135 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:29Z"} 2024-03-04T18:21:29.135 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:21:29.135 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:21:29.135 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:21:29.136 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:21:34.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:34.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:34.137 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:34.188 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:34.188 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:34.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:34.257+0000 7f45a90eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:34.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:34.257+0000 7f45a90eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:34.292 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:34Z"} 2024-03-04T18:21:34.292 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:34Z"} 2024-03-04T18:21:34.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:21:34.293 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:21:34.293 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:21:34.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:21:39.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:39.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:39.294 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:39.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:39.397+0000 7f3f2c45c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:39.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:39.401+0000 7f3f2c45c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:39.432 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:39Z"} 2024-03-04T18:21:39.433 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:39Z"} 2024-03-04T18:21:39.433 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:21:39.433 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:21:39.433 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:21:39.433 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:21:44.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:44.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:44.436 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:44.479 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:44.479 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:44.537+0000 7eff90032980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:44.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:44.537+0000 7eff90032980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:44.567 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:44Z"} 2024-03-04T18:21:44.568 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:44Z"} 2024-03-04T18:21:44.568 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:21:44.569 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:21:44.569 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:21:44.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:21:49.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:49.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:49.571 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:49.614 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:49.614 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:49.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:49.669+0000 7fdda0a1b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:49.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:49.669+0000 7fdda0a1b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:49.699 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:49Z"} 2024-03-04T18:21:49.699 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:49Z"} 2024-03-04T18:21:49.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:21:49.699 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:21:49.700 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:21:49.700 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:21:54.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:54.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:54.701 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:54.744 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:54.745 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:54.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:54.816+0000 7fb955c32980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:54.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:54.816+0000 7fb955c32980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:54.849 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:54Z"} 2024-03-04T18:21:54.849 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:54Z"} 2024-03-04T18:21:54.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:21:54.850 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:21:54.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:21:54.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:21:59.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:59.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:59.851 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:59.894 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:59.894 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:59.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:59.952+0000 7f39f73be980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:59.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:21:59.952+0000 7f39f73be980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:59.981 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:59Z"} 2024-03-04T18:21:59.981 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:59Z"} 2024-03-04T18:21:59.981 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:21:59.981 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:21:59.981 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:21:59.982 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:22:04.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:04.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:04.983 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:05.024 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:05.024 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:05.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:05.088+0000 7fb98c5ea980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:05.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:05.088+0000 7fb98c5ea980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:05.118 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:05Z"} 2024-03-04T18:22:05.118 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:05Z"} 2024-03-04T18:22:05.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:22:05.118 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:22:05.118 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:22:05.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:22:10.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:10.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:10.120 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:10.164 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:10.165 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:10.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:10.244+0000 7f7bd1e48980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:10.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:10.244+0000 7f7bd1e48980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:10.276 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:10Z"} 2024-03-04T18:22:10.277 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:10Z"} 2024-03-04T18:22:10.277 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:22:10.277 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:22:10.277 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:22:10.277 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:22:15.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:15.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:15.279 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:15.321 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:15.321 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:15.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:15.380+0000 7f5fe2d39980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:15.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:15.384+0000 7f5fe2d39980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:15.420 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:15Z"} 2024-03-04T18:22:15.420 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:15Z"} 2024-03-04T18:22:15.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:22:15.421 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:22:15.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:22:15.421 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:22:20.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:20.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:20.422 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:20.461 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:20.462 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:20.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:20.520+0000 7f286453b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:20.524+0000 7f286453b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:20.551 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:20Z"} 2024-03-04T18:22:20.551 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:20Z"} 2024-03-04T18:22:20.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:22:20.552 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:22:20.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:22:20.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:22:25.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:25.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:25.553 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:25.596 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:25.596 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:25.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:25.656+0000 7f3305d4b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:25.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:25.656+0000 7f3305d4b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:25.687 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:25Z"} 2024-03-04T18:22:25.687 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:25Z"} 2024-03-04T18:22:25.687 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575076.582500_321.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:56.582500Z', 'realm_epoch': 2}}] 2024-03-04T18:22:25.687 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:22:25.688 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575076.582500_321.1'} 2024-03-04T18:22:25.688 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575076.582500_321.1 2024-03-04T18:22:30.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:30.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:30.689 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:30.731 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:30.732 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:30.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:30.804+0000 7fdcc9bb2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:30.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:30.804+0000 7fdcc9bb2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:30.834 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:30Z"} 2024-03-04T18:22:30.835 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:30Z"} 2024-03-04T18:22:30.835 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.523626_319.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.523626Z', 'realm_epoch': 2}}] 2024-03-04T18:22:30.835 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:22:30.835 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.523626_319.1'} 2024-03-04T18:22:30.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.523626_319.1 2024-03-04T18:22:35.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:35.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:35.837 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:35.879 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:35.880 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:35.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:35.936+0000 7fb85887e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:35.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:35.936+0000 7fb85887e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:35.966 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:35Z"} 2024-03-04T18:22:35.967 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:35Z"} 2024-03-04T18:22:35.967 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:22:35.967 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:22:35.967 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:22:35.967 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:22:40.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:40.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:40.968 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:41.010 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:41.010 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:41.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:41.088+0000 7f75b2333980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:41.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:41.088+0000 7f75b2333980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:41.121 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:41Z"} 2024-03-04T18:22:41.121 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:41Z"} 2024-03-04T18:22:41.122 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:22:41.122 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:22:41.122 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:22:41.122 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:22:46.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:46.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:46.124 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:46.170 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:46.171 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:46.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:46.248+0000 7f2d037ec980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:46.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:46.252+0000 7f2d037ec980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:46.287 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:46Z"} 2024-03-04T18:22:46.287 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:46Z"} 2024-03-04T18:22:46.287 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:22:46.288 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:22:46.288 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:22:46.288 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.310269_305.1 2024-03-04T18:22:51.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:51.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:51.290 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:51.343 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:51.343 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:51.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:51.399+0000 7fb4a1b01980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:51.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:51.399+0000 7fb4a1b01980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:51.429 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:51Z"} 2024-03-04T18:22:51.430 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:51Z"} 2024-03-04T18:22:51.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:22:51.430 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:22:51.430 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:22:51.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:22:56.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:56.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:56.431 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:56.474 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:56.474 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:56.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:56.535+0000 7f338a0a0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:56.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:22:56.535+0000 7f338a0a0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:56.570 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:56Z"} 2024-03-04T18:22:56.570 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:56Z"} 2024-03-04T18:22:56.570 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:22:56.570 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:22:56.571 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:22:56.571 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:23:01.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:01.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:01.573 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:01.615 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:01.615 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:01.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:01.683+0000 7fa2aac42980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:01.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:01.687+0000 7fa2aac42980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:01.715 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:01Z"} 2024-03-04T18:23:01.715 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:01Z"} 2024-03-04T18:23:01.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:23:01.716 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:23:01.716 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:23:01.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:23:06.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:06.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:06.718 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:06.762 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:06.762 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:06.831+0000 7f83a203a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:06.831+0000 7f83a203a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:06.871 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:06Z"} 2024-03-04T18:23:06.871 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:06Z"} 2024-03-04T18:23:06.871 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:23:06.872 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:23:06.872 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:23:06.872 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:23:11.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:11.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:11.873 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:11.917 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:11.917 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:11.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:11.983+0000 7f35edc51980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:11.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:11.983+0000 7f35edc51980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:12.024 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:11Z"} 2024-03-04T18:23:12.024 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:11Z"} 2024-03-04T18:23:12.025 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.495723_318.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.495723Z', 'realm_epoch': 2}}] 2024-03-04T18:23:12.025 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:23:12.025 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.495723_318.1'} 2024-03-04T18:23:12.025 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.495723_318.1 2024-03-04T18:23:17.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:17.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:17.027 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:17.075 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:17.075 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:17.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:17.135+0000 7f3824fc1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:17.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:17.135+0000 7f3824fc1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:17.167 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:17Z"} 2024-03-04T18:23:17.167 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:17Z"} 2024-03-04T18:23:17.167 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:23:17.167 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:23:17.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:23:17.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:23:22.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:22.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:22.169 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:22.211 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:22.211 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:22.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:22.283+0000 7f9077926980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:22.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:22.283+0000 7f9077926980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:22.319 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:22Z"} 2024-03-04T18:23:22.319 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:22Z"} 2024-03-04T18:23:22.319 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575076.582500_321.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:56.582500Z', 'realm_epoch': 2}}] 2024-03-04T18:23:22.319 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:23:22.319 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575076.582500_321.1'} 2024-03-04T18:23:22.319 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575076.582500_321.1 2024-03-04T18:23:27.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:27.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:27.321 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:27.368 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:27.368 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:27.443+0000 7f2444e6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:27.447+0000 7f2444e6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:27.482 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:27Z"} 2024-03-04T18:23:27.482 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:27Z"} 2024-03-04T18:23:27.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:23:27.483 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:23:27.483 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:23:27.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:23:32.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:32.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:32.485 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:32.527 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:32.527 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:32.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:32.587+0000 7f527c8be980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:32.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:32.587+0000 7f527c8be980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:32.615 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:32Z"} 2024-03-04T18:23:32.615 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:32Z"} 2024-03-04T18:23:32.615 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:23:32.616 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:23:32.616 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:23:32.616 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:23:37.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:37.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:37.618 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:37.660 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:37.661 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:37.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:37.735+0000 7f55f3245980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:37.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:37.735+0000 7f55f3245980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:37.764 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:37Z"} 2024-03-04T18:23:37.764 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:37Z"} 2024-03-04T18:23:37.765 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:23:37.765 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:23:37.765 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:23:37.765 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:23:42.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:42.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:42.766 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:42.810 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:42.810 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:42.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:42.875+0000 7fe0e7f5b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:42.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:42.875+0000 7fe0e7f5b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:42.900 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:42Z"} 2024-03-04T18:23:42.901 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:42Z"} 2024-03-04T18:23:42.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:23:42.901 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:23:42.901 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:23:42.901 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:23:47.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:47.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:47.902 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:48.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:48.006+0000 7fcf68d7c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:48.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:48.006+0000 7fcf68d7c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:48.035 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:48Z"} 2024-03-04T18:23:48.035 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:48Z"} 2024-03-04T18:23:48.035 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:23:48.035 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:23:48.035 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:23:48.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:23:53.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:53.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:53.036 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:53.080 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:53.080 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:53.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:53.158+0000 7f8b63e20980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:53.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:53.158+0000 7f8b63e20980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:53.184 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:53Z"} 2024-03-04T18:23:53.185 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:53Z"} 2024-03-04T18:23:53.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:23:53.185 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:23:53.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:23:53.185 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:23:58.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:58.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:58.186 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:58.230 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:58.231 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:58.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:58.290+0000 7fd188ddf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:58.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:23:58.290+0000 7fd188ddf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:58.318 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:58Z"} 2024-03-04T18:23:58.318 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:58Z"} 2024-03-04T18:23:58.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:23:58.318 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:23:58.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:23:58.319 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:24:03.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:03.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:03.320 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:03.367 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:03.367 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:03.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:03.430+0000 7f39770c3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:03.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:03.430+0000 7f39770c3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:03.462 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:03Z"} 2024-03-04T18:24:03.463 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:03Z"} 2024-03-04T18:24:03.463 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:24:03.463 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:24:03.463 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:24:03.463 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:24:08.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:08.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:08.465 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:08.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:08.570+0000 7faa35589980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:08.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:08.570+0000 7faa35589980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:08.597 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:08Z"} 2024-03-04T18:24:08.597 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:08Z"} 2024-03-04T18:24:08.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:24:08.598 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:24:08.598 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:24:08.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:24:13.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:13.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:13.600 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:13.647 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:13.647 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:13.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:13.742+0000 7f3d93a64980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:13.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:13.746+0000 7f3d93a64980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:13.774 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:13Z"} 2024-03-04T18:24:13.774 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:13Z"} 2024-03-04T18:24:13.774 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:24:13.775 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:24:13.775 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:24:13.775 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:24:18.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:18.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:18.776 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:18.820 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:18.821 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:18.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:18.882+0000 7f269a7b2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:18.886+0000 7f269a7b2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:18.919 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:18Z"} 2024-03-04T18:24:18.920 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:18Z"} 2024-03-04T18:24:18.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:24:18.920 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:24:18.920 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:24:18.920 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:24:23.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:23.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:23.922 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:23.964 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:23.964 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:24.034+0000 7f798835f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:24.038+0000 7f798835f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:24.065 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:24Z"} 2024-03-04T18:24:24.065 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:24Z"} 2024-03-04T18:24:24.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:24:24.065 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:24:24.065 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:24:24.065 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T18:24:29.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:29.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:29.067 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:29.109 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:29.110 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:29.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:29.178+0000 7fbcf266f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:29.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:29.182+0000 7fbcf266f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:29.219 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:29Z"} 2024-03-04T18:24:29.219 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:29Z"} 2024-03-04T18:24:29.219 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:24:29.219 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:24:29.219 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:24:29.219 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T18:24:34.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:34.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:34.221 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:34.264 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:34.264 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:34.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:34.342+0000 7f05aebc9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:34.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:34.342+0000 7f05aebc9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:34.378 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:34Z"} 2024-03-04T18:24:34.378 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:34Z"} 2024-03-04T18:24:34.378 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T18:24:34.378 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T18:24:34.378 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T18:24:34.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T18:24:39.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2024-03-04T18:24:39.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:39.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:39.383 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T18:24:40.904 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:40.905 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:40.998 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"4c07ecec-0182-419c-9591-1fc948a61db9","name":"remove_zg","api_name":"remove_zg","is_master":false,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":[]}],"default_placement":"default-placement","realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","sync_policy":{"groups":[]},"enabled_features":["resharding"]} 2024-03-04T18:24:40.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:40.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:40.999 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T18:24:41.087 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:41.087 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:41.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T18:24:41.138+0000 7f1b73e3a980 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2024-03-04T18:24:41.151 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"478b7783-d9c3-4698-ad0a-5c719bc03759","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","notif_pool":"remove.rgw.log:notif"} 2024-03-04T18:24:41.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:41.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:41.152 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T18:24:41.184 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:41.184 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:41.293 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"f6a09d88-afdd-40f4-a159-dda085c373df","epoch":5,"predecessor_uuid":"aa7942d8-051d-40ca-b86f-359c9c0301cc","sync_status":[],"period_map":{"id":"f6a09d88-afdd-40f4-a159-dda085c373df","zonegroups":[{"id":"4c07ecec-0182-419c-9591-1fc948a61db9","name":"remove_zg","api_name":"remove_zg","is_master":false,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"478b7783-d9c3-4698-ad0a-5c719bc03759","zones":[{"id":"478b7783-d9c3-4698-ad0a-5c719bc03759","name":"remove","endpoints":[],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","zones":[{"id":"185c722f-6c40-48e4-8855-bf605b5498c7","name":"test-zone2","endpoints":["http://smithi153.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"c146b41a-1ed1-469f-afe6-2dddc3577925","name":"test-zone3","endpoints":["http://smithi136.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"d96636ac-054d-40e3-90f3-e3912da784b2","name":"test-zone1","endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"185c722f-6c40-48e4-8855-bf605b5498c7","val":1962731440},{"key":"478b7783-d9c3-4698-ad0a-5c719bc03759","val":549157452},{"key":"c146b41a-1ed1-469f-afe6-2dddc3577925","val":3710957901},{"key":"d96636ac-054d-40e3-90f3-e3912da784b2","val":2093114680}]},"master_zonegroup":"test-zonegroup","master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","realm_epoch":2} 2024-03-04T18:24:41.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:41.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:41.294 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T18:24:41.367 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:41.367 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:41.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T18:24:41.410+0000 7f279640b980 0 NOTICE: removed master_zone of zonegroup remove_zg 2024-03-04T18:24:41.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T18:24:41.410+0000 7f279640b980 0 Removed zone from zonegroup remove_zg 2024-03-04T18:24:41.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:41.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:41.427 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T18:24:41.503 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:41.503 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:41.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-04T18:24:41.566+0000 7fa809b63980 0 ERROR: zonegroup remove_zg has a non existent master zone 2024-03-04T18:24:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T18:24:41.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:41.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:41.582 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T18:24:41.664 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:41.664 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:41.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:41.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T18:24:41.771 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T18:24:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:42.126 INFO:teuthology.orchestra.run.smithi136.stdout:{"id":"f6a09d88-afdd-40f4-a159-dda085c373df","epoch":6,"predecessor_uuid":"aa7942d8-051d-40ca-b86f-359c9c0301cc","sync_status":[],"period_map":{"id":"f6a09d88-afdd-40f4-a159-dda085c373df","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","zones":[{"id":"185c722f-6c40-48e4-8855-bf605b5498c7","name":"test-zone2","endpoints":["http://smithi153.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"c146b41a-1ed1-469f-afe6-2dddc3577925","name":"test-zone3","endpoints":["http://smithi136.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"d96636ac-054d-40e3-90f3-e3912da784b2","name":"test-zone1","endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"185c722f-6c40-48e4-8855-bf605b5498c7","val":1962731440},{"key":"c146b41a-1ed1-469f-afe6-2dddc3577925","val":3710957901},{"key":"d96636ac-054d-40e3-90f3-e3912da784b2","val":2093114680}]},"master_zonegroup":"test-zonegroup","master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","realm_epoch":2} 2024-03-04T18:24:42.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2024-03-04T18:24:42.127 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-49 2024-03-04T18:24:42.127 DEBUG:boto:path=/ppfahn-49/ 2024-03-04T18:24:42.127 DEBUG:boto:auth_path=/ppfahn-49/ 2024-03-04T18:24:42.128 DEBUG:boto:Method: PUT 2024-03-04T18:24:42.128 DEBUG:boto:Path: /ppfahn-49/ 2024-03-04T18:24:42.128 DEBUG:boto:Data: 2024-03-04T18:24:42.128 DEBUG:boto:Headers: {} 2024-03-04T18:24:42.128 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:24:42.128 DEBUG:boto:Port: 8000 2024-03-04T18:24:42.128 DEBUG:boto:Params: {} 2024-03-04T18:24:42.129 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:24:42.129 DEBUG:boto:Token: None 2024-03-04T18:24:42.129 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:24:42 GMT /ppfahn-49/ 2024-03-04T18:24:42.129 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:M8kPUM1fE43phFYTZR0toxTrBJc= 2024-03-04T18:24:42.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:M8kPUM1fE43phFYTZR0toxTrBJc=', 'Content-Length': '0'} 2024-03-04T18:24:42.946 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a7ef654d8c0caa59-0065e611ea-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:42.947 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-50 2024-03-04T18:24:42.947 DEBUG:boto:path=/ppfahn-50/ 2024-03-04T18:24:42.947 DEBUG:boto:auth_path=/ppfahn-50/ 2024-03-04T18:24:42.947 DEBUG:boto:Method: PUT 2024-03-04T18:24:42.947 DEBUG:boto:Path: /ppfahn-50/ 2024-03-04T18:24:42.947 DEBUG:boto:Data: 2024-03-04T18:24:42.947 DEBUG:boto:Headers: {} 2024-03-04T18:24:42.948 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:24:42.948 DEBUG:boto:Port: 8000 2024-03-04T18:24:42.948 DEBUG:boto:Params: {} 2024-03-04T18:24:42.948 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:24:42.948 DEBUG:boto:Token: None 2024-03-04T18:24:42.948 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:24:42 GMT /ppfahn-50/ 2024-03-04T18:24:42.948 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hc9M0SD9UCZWt7OUYXjg+Vyf0K8= 2024-03-04T18:24:42.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:42 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hc9M0SD9UCZWt7OUYXjg+Vyf0K8=', 'Content-Length': '0'} 2024-03-04T18:24:43.024 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000094869ed7d234d654-0065e611ea-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:43.024 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-51 2024-03-04T18:24:43.024 DEBUG:boto:path=/ppfahn-51/ 2024-03-04T18:24:43.024 DEBUG:boto:auth_path=/ppfahn-51/ 2024-03-04T18:24:43.024 DEBUG:boto:Method: PUT 2024-03-04T18:24:43.025 DEBUG:boto:Path: /ppfahn-51/ 2024-03-04T18:24:43.025 DEBUG:boto:Data: 2024-03-04T18:24:43.025 DEBUG:boto:Headers: {} 2024-03-04T18:24:43.025 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:24:43.025 DEBUG:boto:Port: 8001 2024-03-04T18:24:43.025 DEBUG:boto:Params: {} 2024-03-04T18:24:43.025 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T18:24:43.025 DEBUG:boto:Token: None 2024-03-04T18:24:43.025 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:24:43 GMT /ppfahn-51/ 2024-03-04T18:24:43.026 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:yfYFx52PF8TS+EDe7T/4Z2JybwY= 2024-03-04T18:24:43.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yfYFx52PF8TS+EDe7T/4Z2JybwY=', 'Content-Length': '0'} 2024-03-04T18:24:43.227 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000076cb969b04ec9be6-0065e611eb-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:43.227 DEBUG:boto:path=/ppfahn-49/ 2024-03-04T18:24:43.228 DEBUG:boto:auth_path=/ppfahn-49/ 2024-03-04T18:24:43.228 DEBUG:boto:path=/ppfahn-49/?website 2024-03-04T18:24:43.228 DEBUG:boto:auth_path=/ppfahn-49/?website 2024-03-04T18:24:43.228 DEBUG:boto:Method: PUT 2024-03-04T18:24:43.228 DEBUG:boto:Path: /ppfahn-49/?website 2024-03-04T18:24:43.228 DEBUG:boto:Data: index.htmlerror.html 2024-03-04T18:24:43.228 DEBUG:boto:Headers: {} 2024-03-04T18:24:43.228 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:24:43.228 DEBUG:boto:Port: 8000 2024-03-04T18:24:43.228 DEBUG:boto:Params: {} 2024-03-04T18:24:43.229 DEBUG:boto:Token: None 2024-03-04T18:24:43.229 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:24:43 GMT /ppfahn-49/?website 2024-03-04T18:24:43.229 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:DKQPgJmKLuXVbjnw4KlDMzDdBDY= 2024-03-04T18:24:43.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:DKQPgJmKLuXVbjnw4KlDMzDdBDY=', 'Content-Length': '241'} 2024-03-04T18:24:43.230 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000071b1a410444f9b22-0065e611eb-5376-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:43.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2024-03-04T18:24:43.231 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-52 2024-03-04T18:24:43.232 DEBUG:boto:path=/ppfahn-52/ 2024-03-04T18:24:43.232 DEBUG:boto:auth_path=/ppfahn-52/ 2024-03-04T18:24:43.232 DEBUG:boto:Method: PUT 2024-03-04T18:24:43.232 DEBUG:boto:Path: /ppfahn-52/ 2024-03-04T18:24:43.232 DEBUG:boto:Data: 2024-03-04T18:24:43.232 DEBUG:boto:Headers: {} 2024-03-04T18:24:43.232 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:24:43.232 DEBUG:boto:Port: 8000 2024-03-04T18:24:43.232 DEBUG:boto:Params: {} 2024-03-04T18:24:43.233 DEBUG:boto:Token: None 2024-03-04T18:24:43.233 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:24:43 GMT /ppfahn-52/ 2024-03-04T18:24:43.233 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:LcnZs1NuSlP/S3AGpVLlxRIgQao= 2024-03-04T18:24:43.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LcnZs1NuSlP/S3AGpVLlxRIgQao=', 'Content-Length': '0'} 2024-03-04T18:24:43.281 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008bf365f40189afa0-0065e611eb-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:43.282 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-53 2024-03-04T18:24:43.282 DEBUG:boto:path=/ppfahn-53/ 2024-03-04T18:24:43.282 DEBUG:boto:auth_path=/ppfahn-53/ 2024-03-04T18:24:43.282 DEBUG:boto:Method: PUT 2024-03-04T18:24:43.282 DEBUG:boto:Path: /ppfahn-53/ 2024-03-04T18:24:43.282 DEBUG:boto:Data: 2024-03-04T18:24:43.282 DEBUG:boto:Headers: {} 2024-03-04T18:24:43.282 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:24:43.282 DEBUG:boto:Port: 8000 2024-03-04T18:24:43.282 DEBUG:boto:Params: {} 2024-03-04T18:24:43.283 DEBUG:boto:Token: None 2024-03-04T18:24:43.283 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:24:43 GMT /ppfahn-53/ 2024-03-04T18:24:43.283 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5PeyNjTT4Fvl6u4H4jBNHa3V7kU= 2024-03-04T18:24:43.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5PeyNjTT4Fvl6u4H4jBNHa3V7kU=', 'Content-Length': '0'} 2024-03-04T18:24:43.447 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000053e9186788df6115-0065e611eb-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:43.447 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-54 2024-03-04T18:24:43.447 DEBUG:boto:path=/ppfahn-54/ 2024-03-04T18:24:43.447 DEBUG:boto:auth_path=/ppfahn-54/ 2024-03-04T18:24:43.447 DEBUG:boto:Method: PUT 2024-03-04T18:24:43.447 DEBUG:boto:Path: /ppfahn-54/ 2024-03-04T18:24:43.448 DEBUG:boto:Data: 2024-03-04T18:24:43.448 DEBUG:boto:Headers: {} 2024-03-04T18:24:43.448 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:24:43.448 DEBUG:boto:Port: 8001 2024-03-04T18:24:43.448 DEBUG:boto:Params: {} 2024-03-04T18:24:43.448 DEBUG:boto:Token: None 2024-03-04T18:24:43.448 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:24:43 GMT /ppfahn-54/ 2024-03-04T18:24:43.448 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cxreXZUXiOKIacmohlP/eLAra7I= 2024-03-04T18:24:43.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cxreXZUXiOKIacmohlP/eLAra7I=', 'Content-Length': '0'} 2024-03-04T18:24:43.513 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000824e62a94727510d-0065e611eb-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:43.514 DEBUG:boto:path=/ppfahn-52/ 2024-03-04T18:24:43.514 DEBUG:boto:auth_path=/ppfahn-52/ 2024-03-04T18:24:43.514 DEBUG:boto:path=/ppfahn-52/?policy 2024-03-04T18:24:43.514 DEBUG:boto:auth_path=/ppfahn-52/?policy 2024-03-04T18:24:43.514 DEBUG:boto:Method: PUT 2024-03-04T18:24:43.514 DEBUG:boto:Path: /ppfahn-52/?policy 2024-03-04T18:24:43.514 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-03-04T18:24:43.514 DEBUG:boto:Headers: {} 2024-03-04T18:24:43.514 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:24:43.515 DEBUG:boto:Port: 8000 2024-03-04T18:24:43.515 DEBUG:boto:Params: {} 2024-03-04T18:24:43.515 DEBUG:boto:Token: None 2024-03-04T18:24:43.515 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:24:43 GMT /ppfahn-52/?policy 2024-03-04T18:24:43.515 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ifcTiRig7f1PRdk+1DV36WsHW7w= 2024-03-04T18:24:43.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ifcTiRig7f1PRdk+1DV36WsHW7w=', 'Content-Length': '97'} 2024-03-04T18:24:43.548 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001d3c0b3bad1b52fb-0065e611eb-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:43.548 DEBUG:boto:path=/ppfahn-52/ 2024-03-04T18:24:43.548 DEBUG:boto:auth_path=/ppfahn-52/ 2024-03-04T18:24:43.548 DEBUG:boto:path=/ppfahn-52/?policy 2024-03-04T18:24:43.548 DEBUG:boto:auth_path=/ppfahn-52/?policy 2024-03-04T18:24:43.549 DEBUG:boto:Method: GET 2024-03-04T18:24:43.549 DEBUG:boto:Path: /ppfahn-52/?policy 2024-03-04T18:24:43.549 DEBUG:boto:Data: 2024-03-04T18:24:43.549 DEBUG:boto:Headers: {} 2024-03-04T18:24:43.549 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:24:43.549 DEBUG:boto:Port: 8000 2024-03-04T18:24:43.549 DEBUG:boto:Params: {} 2024-03-04T18:24:43.549 DEBUG:boto:Token: None 2024-03-04T18:24:43.549 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 18:24:43 GMT /ppfahn-52/?policy 2024-03-04T18:24:43.550 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:FRXN2kO5jgKYdfQ4JtL9p9Eim08= 2024-03-04T18:24:43.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FRXN2kO5jgKYdfQ4JtL9p9Eim08=', 'Content-Length': '0'} 2024-03-04T18:24:43.551 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000196366ca3455b1ba-0065e611eb-5376-test-zone1'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '97'), ('Date', 'Mon, 04 Mar 2024 18:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:43.551 DEBUG:boto:path=/ppfahn-53/ 2024-03-04T18:24:43.551 DEBUG:boto:auth_path=/ppfahn-53/ 2024-03-04T18:24:43.551 DEBUG:boto:path=/ppfahn-53/?policy 2024-03-04T18:24:43.551 DEBUG:boto:auth_path=/ppfahn-53/?policy 2024-03-04T18:24:43.552 DEBUG:boto:Method: PUT 2024-03-04T18:24:43.552 DEBUG:boto:Path: /ppfahn-53/?policy 2024-03-04T18:24:43.552 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-03-04T18:24:43.552 DEBUG:boto:Headers: {} 2024-03-04T18:24:43.552 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:24:43.552 DEBUG:boto:Port: 8000 2024-03-04T18:24:43.552 DEBUG:boto:Params: {} 2024-03-04T18:24:43.552 DEBUG:boto:Token: None 2024-03-04T18:24:43.552 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:24:43 GMT /ppfahn-53/?policy 2024-03-04T18:24:43.553 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VSWQty4WP5V9QBX9vJqt2JcbnyA= 2024-03-04T18:24:43.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VSWQty4WP5V9QBX9vJqt2JcbnyA=', 'Content-Length': '97'} 2024-03-04T18:24:44.576 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000041292098e192bf03-0065e611eb-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:24:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:44.577 DEBUG:boto:path=/ppfahn-53/ 2024-03-04T18:24:44.577 DEBUG:boto:auth_path=/ppfahn-53/ 2024-03-04T18:24:44.577 DEBUG:boto:path=/ppfahn-53/?policy 2024-03-04T18:24:44.577 DEBUG:boto:auth_path=/ppfahn-53/?policy 2024-03-04T18:24:44.577 DEBUG:boto:Method: GET 2024-03-04T18:24:44.577 DEBUG:boto:Path: /ppfahn-53/?policy 2024-03-04T18:24:44.577 DEBUG:boto:Data: 2024-03-04T18:24:44.578 DEBUG:boto:Headers: {} 2024-03-04T18:24:44.578 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:24:44.578 DEBUG:boto:Port: 8000 2024-03-04T18:24:44.578 DEBUG:boto:Params: {} 2024-03-04T18:24:44.578 DEBUG:boto:Token: None 2024-03-04T18:24:44.578 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 18:24:44 GMT /ppfahn-53/?policy 2024-03-04T18:24:44.578 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eFWJG54rKxMlZ+XnnhkKHXMe3gQ= 2024-03-04T18:24:44.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eFWJG54rKxMlZ+XnnhkKHXMe3gQ=', 'Content-Length': '0'} 2024-03-04T18:24:44.580 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007f395aab9ffc0476-0065e611ec-7099-test-zone2'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '97'), ('Date', 'Mon, 04 Mar 2024 18:24:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:44.580 DEBUG:boto:path=/ppfahn-54/ 2024-03-04T18:24:44.580 DEBUG:boto:auth_path=/ppfahn-54/ 2024-03-04T18:24:44.581 DEBUG:boto:path=/ppfahn-54/?policy 2024-03-04T18:24:44.581 DEBUG:boto:auth_path=/ppfahn-54/?policy 2024-03-04T18:24:44.581 DEBUG:boto:Method: PUT 2024-03-04T18:24:44.581 DEBUG:boto:Path: /ppfahn-54/?policy 2024-03-04T18:24:44.581 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-03-04T18:24:44.581 DEBUG:boto:Headers: {} 2024-03-04T18:24:44.581 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:24:44.581 DEBUG:boto:Port: 8001 2024-03-04T18:24:44.581 DEBUG:boto:Params: {} 2024-03-04T18:24:44.581 DEBUG:boto:Token: None 2024-03-04T18:24:44.582 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:24:44 GMT /ppfahn-54/?policy 2024-03-04T18:24:44.582 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:PUWzSXVy/SYdwb6FgD865179TVM= 2024-03-04T18:24:44.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:PUWzSXVy/SYdwb6FgD865179TVM=', 'Content-Length': '97'} 2024-03-04T18:24:45.604 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004c49ce6851f6788e-0065e611ec-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:45.604 DEBUG:boto:path=/ppfahn-54/ 2024-03-04T18:24:45.604 DEBUG:boto:auth_path=/ppfahn-54/ 2024-03-04T18:24:45.604 DEBUG:boto:path=/ppfahn-54/?policy 2024-03-04T18:24:45.604 DEBUG:boto:auth_path=/ppfahn-54/?policy 2024-03-04T18:24:45.604 DEBUG:boto:Method: GET 2024-03-04T18:24:45.605 DEBUG:boto:Path: /ppfahn-54/?policy 2024-03-04T18:24:45.605 DEBUG:boto:Data: 2024-03-04T18:24:45.605 DEBUG:boto:Headers: {} 2024-03-04T18:24:45.605 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:24:45.605 DEBUG:boto:Port: 8001 2024-03-04T18:24:45.605 DEBUG:boto:Params: {} 2024-03-04T18:24:45.605 DEBUG:boto:Token: None 2024-03-04T18:24:45.605 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 18:24:45 GMT /ppfahn-54/?policy 2024-03-04T18:24:45.605 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:g2z6N0DV+taU1quwIe1lYhB17Dw= 2024-03-04T18:24:45.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:g2z6N0DV+taU1quwIe1lYhB17Dw=', 'Content-Length': '0'} 2024-03-04T18:24:45.607 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000444e4ae4cef5e035-0065e611ed-5379-test-zone3'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '97'), ('Date', 'Mon, 04 Mar 2024 18:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:45.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2024-03-04T18:24:45.608 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-55 2024-03-04T18:24:45.608 DEBUG:boto:path=/ppfahn-55/ 2024-03-04T18:24:45.608 DEBUG:boto:auth_path=/ppfahn-55/ 2024-03-04T18:24:45.608 DEBUG:boto:Method: PUT 2024-03-04T18:24:45.609 DEBUG:boto:Path: /ppfahn-55/ 2024-03-04T18:24:45.609 DEBUG:boto:Data: 2024-03-04T18:24:45.609 DEBUG:boto:Headers: {} 2024-03-04T18:24:45.609 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:24:45.609 DEBUG:boto:Port: 8000 2024-03-04T18:24:45.609 DEBUG:boto:Params: {} 2024-03-04T18:24:45.609 DEBUG:boto:Token: None 2024-03-04T18:24:45.609 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:24:45 GMT /ppfahn-55/ 2024-03-04T18:24:45.609 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:841diROdo0OV8CZWOUWyS4tBtGM= 2024-03-04T18:24:45.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:841diROdo0OV8CZWOUWyS4tBtGM=', 'Content-Length': '0'} 2024-03-04T18:24:45.755 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ad25d504e8f3e13d-0065e611ed-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:45.755 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-56 2024-03-04T18:24:45.755 DEBUG:boto:path=/ppfahn-56/ 2024-03-04T18:24:45.755 DEBUG:boto:auth_path=/ppfahn-56/ 2024-03-04T18:24:45.755 DEBUG:boto:Method: PUT 2024-03-04T18:24:45.755 DEBUG:boto:Path: /ppfahn-56/ 2024-03-04T18:24:45.756 DEBUG:boto:Data: 2024-03-04T18:24:45.756 DEBUG:boto:Headers: {} 2024-03-04T18:24:45.756 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:24:45.756 DEBUG:boto:Port: 8000 2024-03-04T18:24:45.756 DEBUG:boto:Params: {} 2024-03-04T18:24:45.756 DEBUG:boto:Token: None 2024-03-04T18:24:45.756 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:24:45 GMT /ppfahn-56/ 2024-03-04T18:24:45.756 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ivtCSb6jxy9e16f8a5SQBgSW9ms= 2024-03-04T18:24:45.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ivtCSb6jxy9e16f8a5SQBgSW9ms=', 'Content-Length': '0'} 2024-03-04T18:24:45.851 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8c6ba875ae8b083-0065e611ed-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:45.851 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-57 2024-03-04T18:24:45.851 DEBUG:boto:path=/ppfahn-57/ 2024-03-04T18:24:45.851 DEBUG:boto:auth_path=/ppfahn-57/ 2024-03-04T18:24:45.851 DEBUG:boto:Method: PUT 2024-03-04T18:24:45.851 DEBUG:boto:Path: /ppfahn-57/ 2024-03-04T18:24:45.851 DEBUG:boto:Data: 2024-03-04T18:24:45.851 DEBUG:boto:Headers: {} 2024-03-04T18:24:45.852 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:24:45.852 DEBUG:boto:Port: 8001 2024-03-04T18:24:45.852 DEBUG:boto:Params: {} 2024-03-04T18:24:45.852 DEBUG:boto:Token: None 2024-03-04T18:24:45.852 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:24:45 GMT /ppfahn-57/ 2024-03-04T18:24:45.852 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JzQQyDpOHH013bf4V1Jlm5eSOg0= 2024-03-04T18:24:45.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JzQQyDpOHH013bf4V1Jlm5eSOg0=', 'Content-Length': '0'} 2024-03-04T18:24:45.898 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000af3c2b93f1b415b7-0065e611ed-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:24:45.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:45.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:45.899 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:45.965 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:45.965 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:46.169 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T18:24:46.225 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709576685.881266_517.1","last_update":"2024-03-04T18:24:45.881266Z"}] 2024-03-04T18:24:46.225 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:24:46.225 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:24:46.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:46.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:46.226 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:46.267 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:46.267 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:46.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:46.325+0000 7f4a55473980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:46.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:46.325+0000 7f4a55473980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:46.370 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:46Z"} 2024-03-04T18:24:46.371 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:46Z"} 2024-03-04T18:24:46.371 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:24:46.371 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:24:46.371 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:24:46.371 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.310269_305.1 2024-03-04T18:24:51.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:51.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:51.373 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:51.415 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:51.415 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:51.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:51.537+0000 7f50cefab980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:51.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:51.537+0000 7f50cefab980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:51.566 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:51Z"} 2024-03-04T18:24:51.566 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:51Z"} 2024-03-04T18:24:51.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:24:51.567 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:24:51.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:24:51.567 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:24:56.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:56.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:56.568 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:56.610 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:56.610 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:56.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:56.669+0000 7f685c190980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:56.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:24:56.669+0000 7f685c190980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:56.699 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:56Z"} 2024-03-04T18:24:56.699 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:56Z"} 2024-03-04T18:24:56.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:24:56.700 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:24:56.700 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:24:56.700 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:25:01.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:01.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:01.701 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:01.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:01.853+0000 7f38fd94e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:01.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:01.853+0000 7f38fd94e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:01.885 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:01Z"} 2024-03-04T18:25:01.885 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:01Z"} 2024-03-04T18:25:01.885 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:25:01.885 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:25:01.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:25:01.886 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:25:06.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:06.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:06.887 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:06.930 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:06.930 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:07.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:07.013+0000 7f0a3fff7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:07.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:07.017+0000 7f0a3fff7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:07.067 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:07Z"} 2024-03-04T18:25:07.067 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:07Z"} 2024-03-04T18:25:07.067 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:25:07.067 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:25:07.067 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:25:07.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:25:12.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:12.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:12.069 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:12.112 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:12.112 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:12.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:12.181+0000 7fcf962d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:12.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:12.181+0000 7fcf962d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:12.214 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:12Z"} 2024-03-04T18:25:12.214 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:12Z"} 2024-03-04T18:25:12.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:25:12.215 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:25:12.215 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:25:12.215 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:25:17.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:17.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:17.216 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:17.264 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:17.264 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:17.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:17.333+0000 7f4b5c370980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:17.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:17.333+0000 7f4b5c370980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:17.364 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:17Z"} 2024-03-04T18:25:17.364 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:17Z"} 2024-03-04T18:25:17.365 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.292440_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.292440Z', 'realm_epoch': 2}}] 2024-03-04T18:25:17.365 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:25:17.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.292440_304.1'} 2024-03-04T18:25:17.365 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.292440_304.1 2024-03-04T18:25:22.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:22.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:22.367 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:22.409 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:22.409 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:22.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:22.481+0000 7fa7f18c3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:22.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:22.481+0000 7fa7f18c3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:22.514 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.828001_352.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.828001Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:22Z"} 2024-03-04T18:25:22.514 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.828001_352.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.828001Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:22Z"} 2024-03-04T18:25:22.514 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575385.828001_352.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:03:05.828001Z', 'realm_epoch': 2}}] 2024-03-04T18:25:22.514 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:25:22.514 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575385.828001_352.1'} 2024-03-04T18:25:22.514 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575385.828001_352.1 2024-03-04T18:25:27.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:27.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:27.516 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:27.559 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:27.559 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:27.625+0000 7f6e7bdce980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:27.629+0000 7f6e7bdce980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:27.658 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:27Z"} 2024-03-04T18:25:27.658 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:27Z"} 2024-03-04T18:25:27.658 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:25:27.658 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:25:27.658 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:25:27.658 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:25:32.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:32.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:32.660 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:32.702 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:32.703 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:32.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:32.777+0000 7ff2c0e82980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:32.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:32.777+0000 7ff2c0e82980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:32.815 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:32Z"} 2024-03-04T18:25:32.815 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:32Z"} 2024-03-04T18:25:32.815 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:25:32.815 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:25:32.816 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:25:32.816 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:25:37.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:37.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:37.817 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:37.861 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:37.861 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:37.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:37.925+0000 7f84d576a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:37.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:37.925+0000 7f84d576a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:37.957 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:37Z"} 2024-03-04T18:25:37.958 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:37Z"} 2024-03-04T18:25:37.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:25:37.958 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:25:37.958 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:25:37.958 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:25:42.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:42.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:42.960 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:43.002 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:43.002 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:43.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:43.073+0000 7f94f6593980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:43.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:43.077+0000 7f94f6593980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:43.105 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:43Z"} 2024-03-04T18:25:43.105 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:43Z"} 2024-03-04T18:25:43.105 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:25:43.105 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:25:43.106 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:25:43.106 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:25:48.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:48.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:48.107 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:48.150 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:48.150 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:48.216+0000 7f86cda90980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:48.216+0000 7f86cda90980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:48.247 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:48Z"} 2024-03-04T18:25:48.247 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:48Z"} 2024-03-04T18:25:48.248 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:25:48.248 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:25:48.248 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:25:48.248 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:25:53.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:53.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:53.249 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:53.297 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:53.298 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:53.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:53.352+0000 7fb5c10c4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:53.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:53.352+0000 7fb5c10c4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:53.383 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:53Z"} 2024-03-04T18:25:53.383 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:53Z"} 2024-03-04T18:25:53.383 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:25:53.383 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:25:53.383 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:25:53.384 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:25:58.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:58.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:58.385 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:58.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:58.492+0000 7ff6adba3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:58.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:25:58.492+0000 7ff6adba3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:58.520 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:58Z"} 2024-03-04T18:25:58.521 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:58Z"} 2024-03-04T18:25:58.521 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:25:58.521 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:25:58.521 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:25:58.521 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:26:03.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:03.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:03.523 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:03.561 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:03.561 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:03.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:03.616+0000 7f404a4f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:03.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:03.616+0000 7f404a4f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:03.647 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:03Z"} 2024-03-04T18:26:03.647 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:03Z"} 2024-03-04T18:26:03.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:26:03.648 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:26:03.648 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:26:03.648 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:26:08.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:08.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:08.649 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:08.692 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:08.693 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:08.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:08.748+0000 7f5b5a08f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:08.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:08.752+0000 7f5b5a08f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:08.779 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:08Z"} 2024-03-04T18:26:08.779 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:08Z"} 2024-03-04T18:26:08.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:26:08.780 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:26:08.780 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:26:08.780 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:26:13.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:13.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:13.782 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:13.838 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:13.838 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:13.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:13.900+0000 7faeb9fbb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:13.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:13.904+0000 7faeb9fbb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:13.930 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:13Z"} 2024-03-04T18:26:13.931 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:13Z"} 2024-03-04T18:26:13.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:26:13.931 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:26:13.931 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:26:13.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:26:14.049 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:26:13.940+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:26:14.149 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:26:14.148+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:26:18.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:18.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:18.932 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:18.975 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:18.975 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:19.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:19.036+0000 7f5a9224c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:19.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:19.036+0000 7f5a9224c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:19.065 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:19Z"} 2024-03-04T18:26:19.065 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:19Z"} 2024-03-04T18:26:19.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.422483_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.422483Z', 'realm_epoch': 2}}] 2024-03-04T18:26:19.066 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:26:19.066 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.422483_312.1'} 2024-03-04T18:26:19.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.422483_312.1 2024-03-04T18:26:24.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:24.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:24.067 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:24.110 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:24.110 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:24.176+0000 7f68d0680980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:24.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:24.176+0000 7f68d0680980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:24.209 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:24Z"} 2024-03-04T18:26:24.209 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:24Z"} 2024-03-04T18:26:24.209 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:26:24.209 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:26:24.209 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:26:24.210 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:26:29.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:29.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:29.211 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:29.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:29.328+0000 7f3c337e5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:29.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:29.328+0000 7f3c337e5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:29.365 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:29Z"} 2024-03-04T18:26:29.366 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:29Z"} 2024-03-04T18:26:29.366 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:26:29.366 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:26:29.366 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:26:29.366 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:26:34.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:34.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:34.367 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:34.409 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:34.409 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:34.504+0000 7f2d29a8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:34.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:34.504+0000 7f2d29a8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:34.565 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:34Z"} 2024-03-04T18:26:34.565 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:34Z"} 2024-03-04T18:26:34.565 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:26:34.565 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:26:34.565 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:26:34.565 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:26:39.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:39.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:39.567 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:39.611 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:39.611 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:39.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:39.668+0000 7f5bcb103980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:39.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:39.668+0000 7f5bcb103980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:39.696 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:39Z"} 2024-03-04T18:26:39.697 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:39Z"} 2024-03-04T18:26:39.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:26:39.697 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:26:39.697 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:26:39.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:26:44.113 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:26:44.115+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:26:44.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:44.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:44.699 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:44.742 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:44.742 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:44.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:44.800+0000 7f8c69c39980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:44.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:44.800+0000 7f8c69c39980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:44.831 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:44Z"} 2024-03-04T18:26:44.831 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:44Z"} 2024-03-04T18:26:44.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:26:44.831 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:26:44.831 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:26:44.831 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:26:49.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:49.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:49.833 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:49.875 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:49.876 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:49.939+0000 7f3b6b5ab980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:49.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:49.939+0000 7f3b6b5ab980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:49.973 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:49Z"} 2024-03-04T18:26:49.973 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:49Z"} 2024-03-04T18:26:49.973 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:26:49.973 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:26:49.974 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:26:49.974 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:26:54.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:54.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:54.974 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:55.017 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:55.017 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:55.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:55.071+0000 7fef41422980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:55.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:26:55.075+0000 7fef41422980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:55.103 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:55Z"} 2024-03-04T18:26:55.104 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:55Z"} 2024-03-04T18:26:55.104 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:26:55.104 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:26:55.104 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:26:55.104 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.310269_305.1 2024-03-04T18:27:00.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:00.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:00.106 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:00.147 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:00.147 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:00.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:00.203+0000 7f4aaecfe980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:00.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:00.203+0000 7f4aaecfe980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:00.236 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:00Z"} 2024-03-04T18:27:00.236 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:00Z"} 2024-03-04T18:27:00.236 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:27:00.236 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:27:00.237 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:27:00.237 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.310269_305.1 2024-03-04T18:27:05.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:05.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:05.238 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:05.282 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:05.282 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:05.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:05.351+0000 7fa3ad5b8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:05.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:05.351+0000 7fa3ad5b8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:05.387 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:05Z"} 2024-03-04T18:27:05.387 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:05Z"} 2024-03-04T18:27:05.387 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:27:05.388 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:27:05.388 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:27:05.388 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:27:10.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:10.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:10.389 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:10.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:10.495+0000 7ff2bd40b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:10.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:10.495+0000 7ff2bd40b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:10.527 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:10Z"} 2024-03-04T18:27:10.527 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:10Z"} 2024-03-04T18:27:10.528 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:27:10.528 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:27:10.528 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:27:10.528 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:27:15.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:15.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:15.529 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:15.572 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:15.572 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:15.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:15.627+0000 7f5ba9624980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:15.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:15.631+0000 7f5ba9624980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:15.663 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:15Z"} 2024-03-04T18:27:15.663 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:15Z"} 2024-03-04T18:27:15.664 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:27:15.664 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:27:15.664 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:27:15.664 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:27:20.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:20.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:20.666 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:20.709 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:20.710 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:20.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:20.775+0000 7fe3bd2f9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:20.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:20.775+0000 7fe3bd2f9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:20.807 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:20Z"} 2024-03-04T18:27:20.808 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:20Z"} 2024-03-04T18:27:20.808 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.315397_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.315397Z', 'realm_epoch': 2}}] 2024-03-04T18:27:20.808 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:27:20.808 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.315397_306.1'} 2024-03-04T18:27:20.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.315397_306.1 2024-03-04T18:27:25.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:25.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:25.809 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:25.853 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:25.853 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:25.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:25.915+0000 7f209feea980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:25.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:25.919+0000 7f209feea980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:25.952 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:25Z"} 2024-03-04T18:27:25.952 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:25Z"} 2024-03-04T18:27:25.952 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:27:25.952 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:27:25.953 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:27:25.953 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:27:30.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:30.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:30.954 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:30.995 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:30.995 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:31.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:31.055+0000 7f806043d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:31.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:31.055+0000 7f806043d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:31.085 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:31Z"} 2024-03-04T18:27:31.085 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:31Z"} 2024-03-04T18:27:31.085 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:27:31.085 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:27:31.085 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:27:31.085 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:27:36.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:36.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:36.087 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:36.129 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:36.129 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:36.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:36.195+0000 7f5fff8b8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:36.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:36.195+0000 7f5fff8b8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:36.225 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:36Z"} 2024-03-04T18:27:36.225 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:36Z"} 2024-03-04T18:27:36.225 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:27:36.225 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:27:36.226 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:27:36.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:27:41.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:41.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:41.227 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:41.271 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:41.271 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:41.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:41.395+0000 7fd4ee95f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:41.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:41.395+0000 7fd4ee95f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:41.422 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:41Z"} 2024-03-04T18:27:41.423 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:41Z"} 2024-03-04T18:27:41.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:27:41.423 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:27:41.423 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:27:41.423 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.310269_305.1 2024-03-04T18:27:46.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:46.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:46.425 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:46.467 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:46.468 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:46.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:46.539+0000 7f0dd7115980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:46.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:46.543+0000 7f0dd7115980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:46.569 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:46Z"} 2024-03-04T18:27:46.569 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:46Z"} 2024-03-04T18:27:46.569 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:27:46.569 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:27:46.569 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:27:46.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:27:51.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:51.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:51.571 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:51.612 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:51.612 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:51.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:51.686+0000 7f2ac121a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:51.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:51.686+0000 7f2ac121a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:51.714 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:51Z"} 2024-03-04T18:27:51.715 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:51Z"} 2024-03-04T18:27:51.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:27:51.715 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:27:51.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:27:51.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:27:56.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:56.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:56.716 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:56.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:56.822+0000 7f9678996980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:56.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:27:56.822+0000 7f9678996980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:56.849 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:56Z"} 2024-03-04T18:27:56.849 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:56Z"} 2024-03-04T18:27:56.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:27:56.850 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:27:56.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:27:56.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.310269_305.1 2024-03-04T18:28:01.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:01.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:01.851 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:01.890 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:01.891 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:01.962+0000 7f2e51b3c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:01.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:01.966+0000 7f2e51b3c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:01.999 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.820809_351.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.820809Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:01Z"} 2024-03-04T18:28:01.999 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.820809_351.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.820809Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:01Z"} 2024-03-04T18:28:01.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575385.820809_351.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:03:05.820809Z', 'realm_epoch': 2}}] 2024-03-04T18:28:01.999 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:28:01.999 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575385.820809_351.1'} 2024-03-04T18:28:02.000 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575385.820809_351.1 2024-03-04T18:28:07.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:07.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:07.001 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:07.044 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:07.044 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:07.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:07.106+0000 7f2180113980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:07.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:07.106+0000 7f2180113980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:07.136 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.051072_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.051072Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:07Z"} 2024-03-04T18:28:07.136 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.051072_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.051072Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:07Z"} 2024-03-04T18:28:07.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.051072_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.051072Z', 'realm_epoch': 2}}] 2024-03-04T18:28:07.137 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:28:07.137 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.051072_382.1'} 2024-03-04T18:28:07.137 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575694.051072_382.1 2024-03-04T18:28:12.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:12.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:12.138 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:12.181 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:12.181 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:12.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:12.250+0000 7fd21b6cc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:12.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:12.250+0000 7fd21b6cc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:12.279 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.051072_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.051072Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:12Z"} 2024-03-04T18:28:12.279 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.051072_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.051072Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:12Z"} 2024-03-04T18:28:12.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.051072_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.051072Z', 'realm_epoch': 2}}] 2024-03-04T18:28:12.279 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:28:12.279 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.051072_382.1'} 2024-03-04T18:28:12.279 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575694.051072_382.1 2024-03-04T18:28:17.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:17.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:17.281 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:17.323 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:17.323 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:17.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:17.398+0000 7f669c096980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:17.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:17.398+0000 7f669c096980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:17.434 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.051072_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.051072Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:17Z"} 2024-03-04T18:28:17.435 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.051072_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.051072Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:17Z"} 2024-03-04T18:28:17.435 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.051072_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.051072Z', 'realm_epoch': 2}}] 2024-03-04T18:28:17.435 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:28:17.435 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.051072_382.1'} 2024-03-04T18:28:17.435 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575694.051072_382.1 2024-03-04T18:28:22.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:22.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:22.437 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:22.480 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:22.480 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:22.558+0000 7fb3ff426980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:22.558+0000 7fb3ff426980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:22.588 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:22Z"} 2024-03-04T18:28:22.588 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:22Z"} 2024-03-04T18:28:22.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:28:22.589 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:28:22.589 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:28:22.589 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:28:27.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:27.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:27.590 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:27.710+0000 7f7b8b42d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:27.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:27.710+0000 7f7b8b42d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:27.745 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:27Z"} 2024-03-04T18:28:27.745 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:27Z"} 2024-03-04T18:28:27.745 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:28:27.746 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:28:27.746 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:28:27.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:28:32.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:32.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:32.748 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:32.791 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:32.792 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:32.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:32.870+0000 7f31889e4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:32.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:32.874+0000 7f31889e4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:32.918 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:32Z"} 2024-03-04T18:28:32.919 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:32Z"} 2024-03-04T18:28:32.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:28:32.921 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:28:32.921 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:28:32.921 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:28:37.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:37.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:37.924 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:37.967 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:37.967 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:38.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:38.046+0000 7f1325625980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:38.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:38.046+0000 7f1325625980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:38.073 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:38Z"} 2024-03-04T18:28:38.073 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:38Z"} 2024-03-04T18:28:38.073 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:28:38.073 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:28:38.073 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:28:38.074 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:28:43.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:43.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:43.075 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:43.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:43.174+0000 7f28d9b4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:43.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:43.178+0000 7f28d9b4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:43.203 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:43Z"} 2024-03-04T18:28:43.203 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:43Z"} 2024-03-04T18:28:43.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:28:43.204 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:28:43.204 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:28:43.204 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:28:48.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:48.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:48.205 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:48.269 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:48.269 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:48.338+0000 7fc3aa735980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:48.342+0000 7fc3aa735980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:48.375 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:48Z"} 2024-03-04T18:28:48.375 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:48Z"} 2024-03-04T18:28:48.375 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:28:48.375 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:28:48.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:28:48.376 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:28:53.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:53.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:53.377 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:53.420 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:53.420 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:53.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:53.489+0000 7f1b99f29980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:53.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:53.489+0000 7f1b99f29980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:53.522 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:53Z"} 2024-03-04T18:28:53.522 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:53Z"} 2024-03-04T18:28:53.522 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:28:53.522 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:28:53.522 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:28:53.522 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:28:58.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:58.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:58.524 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:58.576 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:58.576 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:58.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:58.637+0000 7f018a62b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:58.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:28:58.637+0000 7f018a62b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:58.673 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:58Z"} 2024-03-04T18:28:58.673 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:58Z"} 2024-03-04T18:28:58.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:28:58.674 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:28:58.674 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:28:58.674 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:29:03.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:03.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:03.675 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:03.718 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:03.718 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:03.797+0000 7f91410ba980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:03.801+0000 7f91410ba980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:03.835 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:03Z"} 2024-03-04T18:29:03.835 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:03Z"} 2024-03-04T18:29:03.835 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:29:03.836 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:29:03.836 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:29:03.836 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:29:08.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:08.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:08.837 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:08.888 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:08.889 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:08.965+0000 7fcdd359d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:08.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:08.965+0000 7fcdd359d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:08.993 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:08Z"} 2024-03-04T18:29:08.994 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:08Z"} 2024-03-04T18:29:08.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.422483_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.422483Z', 'realm_epoch': 2}}] 2024-03-04T18:29:08.994 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:29:08.994 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.422483_312.1'} 2024-03-04T18:29:08.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.422483_312.1 2024-03-04T18:29:13.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:13.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:13.996 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:14.039 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:14.039 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:14.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:14.109+0000 7fa5106bf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:14.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:14.113+0000 7fa5106bf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:14.142 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:14Z"} 2024-03-04T18:29:14.143 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:14Z"} 2024-03-04T18:29:14.143 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.422483_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.422483Z', 'realm_epoch': 2}}] 2024-03-04T18:29:14.143 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:29:14.143 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.422483_312.1'} 2024-03-04T18:29:14.143 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.422483_312.1 2024-03-04T18:29:19.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:19.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:19.146 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:19.249+0000 7fdb3b52d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:19.249+0000 7fdb3b52d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:19.277 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:19Z"} 2024-03-04T18:29:19.277 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:19Z"} 2024-03-04T18:29:19.277 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:29:19.277 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:29:19.277 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:29:19.277 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:29:24.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:24.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:24.279 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:24.401+0000 7fac98cd1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:24.401+0000 7fac98cd1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:24.429 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:24Z"} 2024-03-04T18:29:24.430 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:24Z"} 2024-03-04T18:29:24.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:29:24.430 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:29:24.430 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:29:24.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:29:29.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:29.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:29.432 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:29.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:29.553+0000 7fc2b5578980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:29.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:29.553+0000 7fc2b5578980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:29.589 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:29Z"} 2024-03-04T18:29:29.589 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:29Z"} 2024-03-04T18:29:29.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:29:29.589 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:29:29.589 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:29:29.589 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:29:34.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:34.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:34.591 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:34.634 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:34.634 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:34.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:34.693+0000 7fc5cbdb4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:34.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:34.693+0000 7fc5cbdb4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:34.721 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:34Z"} 2024-03-04T18:29:34.722 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:34Z"} 2024-03-04T18:29:34.722 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:29:34.722 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:29:34.722 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:29:34.722 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.310269_305.1 2024-03-04T18:29:39.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:39.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:39.724 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:39.767 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:39.767 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:39.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:39.845+0000 7fed6d8e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:39.849+0000 7fed6d8e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:39.883 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:39Z"} 2024-03-04T18:29:39.883 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:39Z"} 2024-03-04T18:29:39.883 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:29:39.883 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:29:39.883 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:29:39.884 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T18:29:44.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:44.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:44.885 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:44.985+0000 7f1a20e67980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:44.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:44.985+0000 7f1a20e67980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:45.013 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:44Z"} 2024-03-04T18:29:45.014 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:44Z"} 2024-03-04T18:29:45.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:29:45.014 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:29:45.014 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:29:45.014 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:29:50.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:50.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:50.015 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:50.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:50.133+0000 7f8ddc347980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:50.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:50.137+0000 7f8ddc347980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:50.168 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:50Z"} 2024-03-04T18:29:50.168 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:50Z"} 2024-03-04T18:29:50.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:29:50.169 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T18:29:50.169 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:29:50.169 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T18:29:55.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... FAIL 2024-03-04T18:29:55.173 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-58 2024-03-04T18:29:55.173 DEBUG:boto:path=/ppfahn-58/ 2024-03-04T18:29:55.173 DEBUG:boto:auth_path=/ppfahn-58/ 2024-03-04T18:29:55.173 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.174 DEBUG:boto:Path: /ppfahn-58/ 2024-03-04T18:29:55.174 DEBUG:boto:Data: 2024-03-04T18:29:55.174 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.174 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:29:55.174 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.174 DEBUG:boto:Params: {} 2024-03-04T18:29:55.174 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:29:55.174 DEBUG:boto:Token: None 2024-03-04T18:29:55.175 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-58/ 2024-03-04T18:29:55.175 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:wcaZ6dQexmkmcG19krQyzPo8OfY= 2024-03-04T18:29:55.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wcaZ6dQexmkmcG19krQyzPo8OfY=', 'Content-Length': '0'} 2024-03-04T18:29:55.218 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e09eba8f275cd2df-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.219 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-59 2024-03-04T18:29:55.219 DEBUG:boto:path=/ppfahn-59/ 2024-03-04T18:29:55.219 DEBUG:boto:auth_path=/ppfahn-59/ 2024-03-04T18:29:55.219 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.219 DEBUG:boto:Path: /ppfahn-59/ 2024-03-04T18:29:55.219 DEBUG:boto:Data: 2024-03-04T18:29:55.219 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.219 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:29:55.219 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.219 DEBUG:boto:Params: {} 2024-03-04T18:29:55.219 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:29:55.220 DEBUG:boto:Token: None 2024-03-04T18:29:55.220 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-59/ 2024-03-04T18:29:55.220 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:McfwhZdwN7HNRa8Bmfr5dyWCiaU= 2024-03-04T18:29:55.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:McfwhZdwN7HNRa8Bmfr5dyWCiaU=', 'Content-Length': '0'} 2024-03-04T18:29:55.300 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000578f8edf459d0b84-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.300 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-60 2024-03-04T18:29:55.301 DEBUG:boto:path=/ppfahn-60/ 2024-03-04T18:29:55.301 DEBUG:boto:auth_path=/ppfahn-60/ 2024-03-04T18:29:55.301 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.301 DEBUG:boto:Path: /ppfahn-60/ 2024-03-04T18:29:55.301 DEBUG:boto:Data: 2024-03-04T18:29:55.301 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.301 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:29:55.301 DEBUG:boto:Port: 8001 2024-03-04T18:29:55.301 DEBUG:boto:Params: {} 2024-03-04T18:29:55.301 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T18:29:55.302 DEBUG:boto:Token: None 2024-03-04T18:29:55.302 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-60/ 2024-03-04T18:29:55.302 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:hGh8SprXApuusEMdFji7IxTp5hQ= 2024-03-04T18:29:55.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hGh8SprXApuusEMdFji7IxTp5hQ=', 'Content-Length': '0'} 2024-03-04T18:29:55.479 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000752292a78d0bdb4f-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.479 DEBUG:boto:path=/ppfahn-58/ 2024-03-04T18:29:55.480 DEBUG:boto:auth_path=/ppfahn-58/ 2024-03-04T18:29:55.480 DEBUG:boto:Method: HEAD 2024-03-04T18:29:55.480 DEBUG:boto:Path: /ppfahn-58/ 2024-03-04T18:29:55.480 DEBUG:boto:Data: 2024-03-04T18:29:55.480 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.480 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:29:55.480 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.480 DEBUG:boto:Params: {} 2024-03-04T18:29:55.480 DEBUG:boto:Token: None 2024-03-04T18:29:55.480 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-58/ 2024-03-04T18:29:55.480 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8= 2024-03-04T18:29:55.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8=', 'Content-Length': '0'} 2024-03-04T18:29:55.485 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000579e4df355a7835b-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.485 DEBUG:boto:path=/ppfahn-58/obj1 2024-03-04T18:29:55.485 DEBUG:boto:auth_path=/ppfahn-58/obj1 2024-03-04T18:29:55.485 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.485 DEBUG:boto:Path: /ppfahn-58/obj1 2024-03-04T18:29:55.485 DEBUG:boto:Data: 2024-03-04T18:29:55.486 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:29:55.486 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:29:55.486 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.486 DEBUG:boto:Params: {} 2024-03-04T18:29:55.486 DEBUG:boto:Token: None 2024-03-04T18:29:55.486 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-58/obj1 2024-03-04T18:29:55.486 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CTGuT2/bp4NGJey/YytDSnoZTJw= 2024-03-04T18:29:55.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CTGuT2/bp4NGJey/YytDSnoZTJw='} 2024-03-04T18:29:55.493 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a34638bd2157d24c-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.494 DEBUG:boto:path=/ppfahn-58/ 2024-03-04T18:29:55.494 DEBUG:boto:auth_path=/ppfahn-58/ 2024-03-04T18:29:55.494 DEBUG:boto:Method: HEAD 2024-03-04T18:29:55.494 DEBUG:boto:Path: /ppfahn-58/ 2024-03-04T18:29:55.494 DEBUG:boto:Data: 2024-03-04T18:29:55.494 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.494 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:29:55.494 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.494 DEBUG:boto:Params: {} 2024-03-04T18:29:55.494 DEBUG:boto:Token: None 2024-03-04T18:29:55.494 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-58/ 2024-03-04T18:29:55.495 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8= 2024-03-04T18:29:55.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8=', 'Content-Length': '0'} 2024-03-04T18:29:55.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe43d72cb6bc10f4-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.497 DEBUG:boto:path=/ppfahn-58/obj2 2024-03-04T18:29:55.497 DEBUG:boto:auth_path=/ppfahn-58/obj2 2024-03-04T18:29:55.497 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.497 DEBUG:boto:Path: /ppfahn-58/obj2 2024-03-04T18:29:55.497 DEBUG:boto:Data: 2024-03-04T18:29:55.497 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:29:55.497 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:29:55.497 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.497 DEBUG:boto:Params: {} 2024-03-04T18:29:55.497 DEBUG:boto:Token: None 2024-03-04T18:29:55.498 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-58/obj2 2024-03-04T18:29:55.498 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:0rOaXR+qvgqyG04qCJkcH/W8C1Q= 2024-03-04T18:29:55.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0rOaXR+qvgqyG04qCJkcH/W8C1Q='} 2024-03-04T18:29:55.540 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e83ef1b357e3f57-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.540 DEBUG:boto:path=/ppfahn-58/ 2024-03-04T18:29:55.540 DEBUG:boto:auth_path=/ppfahn-58/ 2024-03-04T18:29:55.540 DEBUG:boto:Method: HEAD 2024-03-04T18:29:55.541 DEBUG:boto:Path: /ppfahn-58/ 2024-03-04T18:29:55.541 DEBUG:boto:Data: 2024-03-04T18:29:55.541 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.541 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:29:55.541 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.541 DEBUG:boto:Params: {} 2024-03-04T18:29:55.541 DEBUG:boto:Token: None 2024-03-04T18:29:55.541 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-58/ 2024-03-04T18:29:55.541 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8= 2024-03-04T18:29:55.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8=', 'Content-Length': '0'} 2024-03-04T18:29:55.544 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000734492fae90aac08-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.544 DEBUG:boto:path=/ppfahn-58/obj3 2024-03-04T18:29:55.544 DEBUG:boto:auth_path=/ppfahn-58/obj3 2024-03-04T18:29:55.544 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.544 DEBUG:boto:Path: /ppfahn-58/obj3 2024-03-04T18:29:55.544 DEBUG:boto:Data: 2024-03-04T18:29:55.545 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:29:55.545 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:29:55.545 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.545 DEBUG:boto:Params: {} 2024-03-04T18:29:55.545 DEBUG:boto:Token: None 2024-03-04T18:29:55.545 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-58/obj3 2024-03-04T18:29:55.545 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:pPyhmg+WAjaG+NdAg+e9c21353I= 2024-03-04T18:29:55.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pPyhmg+WAjaG+NdAg+e9c21353I='} 2024-03-04T18:29:55.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004bb01be47f2f51c8-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.555 DEBUG:boto:path=/ppfahn-58/ 2024-03-04T18:29:55.555 DEBUG:boto:auth_path=/ppfahn-58/ 2024-03-04T18:29:55.555 DEBUG:boto:Method: HEAD 2024-03-04T18:29:55.555 DEBUG:boto:Path: /ppfahn-58/ 2024-03-04T18:29:55.555 DEBUG:boto:Data: 2024-03-04T18:29:55.555 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.555 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:29:55.555 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.555 DEBUG:boto:Params: {} 2024-03-04T18:29:55.555 DEBUG:boto:Token: None 2024-03-04T18:29:55.556 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-58/ 2024-03-04T18:29:55.556 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8= 2024-03-04T18:29:55.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8=', 'Content-Length': '0'} 2024-03-04T18:29:55.558 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d067a75d48a7997-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.558 DEBUG:boto:path=/ppfahn-58/obj4 2024-03-04T18:29:55.558 DEBUG:boto:auth_path=/ppfahn-58/obj4 2024-03-04T18:29:55.558 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.558 DEBUG:boto:Path: /ppfahn-58/obj4 2024-03-04T18:29:55.558 DEBUG:boto:Data: 2024-03-04T18:29:55.558 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:29:55.559 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:29:55.559 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.559 DEBUG:boto:Params: {} 2024-03-04T18:29:55.559 DEBUG:boto:Token: None 2024-03-04T18:29:55.559 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-58/obj4 2024-03-04T18:29:55.559 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:oVdRLw8AEXXSPNLG4xvnUxU0P2g= 2024-03-04T18:29:55.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oVdRLw8AEXXSPNLG4xvnUxU0P2g='} 2024-03-04T18:29:55.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c467f606f8f60cf9-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.567 DEBUG:boto:path=/ppfahn-59/ 2024-03-04T18:29:55.567 DEBUG:boto:auth_path=/ppfahn-59/ 2024-03-04T18:29:55.567 DEBUG:boto:Method: HEAD 2024-03-04T18:29:55.567 DEBUG:boto:Path: /ppfahn-59/ 2024-03-04T18:29:55.568 DEBUG:boto:Data: 2024-03-04T18:29:55.568 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.568 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:29:55.568 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.568 DEBUG:boto:Params: {} 2024-03-04T18:29:55.568 DEBUG:boto:Token: None 2024-03-04T18:29:55.568 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-59/ 2024-03-04T18:29:55.568 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU= 2024-03-04T18:29:55.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU=', 'Content-Length': '0'} 2024-03-04T18:29:55.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5c8861c089ad5ed-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.578 DEBUG:boto:path=/ppfahn-59/obj1 2024-03-04T18:29:55.578 DEBUG:boto:auth_path=/ppfahn-59/obj1 2024-03-04T18:29:55.578 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.578 DEBUG:boto:Path: /ppfahn-59/obj1 2024-03-04T18:29:55.578 DEBUG:boto:Data: 2024-03-04T18:29:55.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:29:55.578 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:29:55.578 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.578 DEBUG:boto:Params: {} 2024-03-04T18:29:55.578 DEBUG:boto:Token: None 2024-03-04T18:29:55.579 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-59/obj1 2024-03-04T18:29:55.579 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GPwi37L+gmpO0VTuylbeo3yP6C0= 2024-03-04T18:29:55.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GPwi37L+gmpO0VTuylbeo3yP6C0='} 2024-03-04T18:29:55.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7b2468713a2e5c2-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.603 DEBUG:boto:path=/ppfahn-59/ 2024-03-04T18:29:55.604 DEBUG:boto:auth_path=/ppfahn-59/ 2024-03-04T18:29:55.604 DEBUG:boto:Method: HEAD 2024-03-04T18:29:55.604 DEBUG:boto:Path: /ppfahn-59/ 2024-03-04T18:29:55.604 DEBUG:boto:Data: 2024-03-04T18:29:55.604 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.604 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:29:55.604 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.604 DEBUG:boto:Params: {} 2024-03-04T18:29:55.604 DEBUG:boto:Token: None 2024-03-04T18:29:55.604 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-59/ 2024-03-04T18:29:55.605 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU= 2024-03-04T18:29:55.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU=', 'Content-Length': '0'} 2024-03-04T18:29:55.608 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002500251ec509ff90-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.608 DEBUG:boto:path=/ppfahn-59/obj2 2024-03-04T18:29:55.608 DEBUG:boto:auth_path=/ppfahn-59/obj2 2024-03-04T18:29:55.608 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.608 DEBUG:boto:Path: /ppfahn-59/obj2 2024-03-04T18:29:55.608 DEBUG:boto:Data: 2024-03-04T18:29:55.608 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:29:55.609 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:29:55.609 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.609 DEBUG:boto:Params: {} 2024-03-04T18:29:55.609 DEBUG:boto:Token: None 2024-03-04T18:29:55.609 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-59/obj2 2024-03-04T18:29:55.609 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:qaRzw6/fQk1cYNlpiq0foT9KrfY= 2024-03-04T18:29:55.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qaRzw6/fQk1cYNlpiq0foT9KrfY='} 2024-03-04T18:29:55.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004515fe7b3995ce03-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.620 DEBUG:boto:path=/ppfahn-59/ 2024-03-04T18:29:55.620 DEBUG:boto:auth_path=/ppfahn-59/ 2024-03-04T18:29:55.621 DEBUG:boto:Method: HEAD 2024-03-04T18:29:55.621 DEBUG:boto:Path: /ppfahn-59/ 2024-03-04T18:29:55.621 DEBUG:boto:Data: 2024-03-04T18:29:55.621 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.621 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:29:55.621 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.621 DEBUG:boto:Params: {} 2024-03-04T18:29:55.621 DEBUG:boto:Token: None 2024-03-04T18:29:55.621 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-59/ 2024-03-04T18:29:55.621 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU= 2024-03-04T18:29:55.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU=', 'Content-Length': '0'} 2024-03-04T18:29:55.624 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092744c9ad8d3d792-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.625 DEBUG:boto:path=/ppfahn-59/obj3 2024-03-04T18:29:55.625 DEBUG:boto:auth_path=/ppfahn-59/obj3 2024-03-04T18:29:55.625 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.625 DEBUG:boto:Path: /ppfahn-59/obj3 2024-03-04T18:29:55.625 DEBUG:boto:Data: 2024-03-04T18:29:55.625 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:29:55.625 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:29:55.625 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.625 DEBUG:boto:Params: {} 2024-03-04T18:29:55.626 DEBUG:boto:Token: None 2024-03-04T18:29:55.626 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-59/obj3 2024-03-04T18:29:55.626 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5Vh6PCkKZRlvPRGVa3sKRIGKi/I= 2024-03-04T18:29:55.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Vh6PCkKZRlvPRGVa3sKRIGKi/I='} 2024-03-04T18:29:55.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb865bc975d278e1-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.668 DEBUG:boto:path=/ppfahn-59/ 2024-03-04T18:29:55.668 DEBUG:boto:auth_path=/ppfahn-59/ 2024-03-04T18:29:55.668 DEBUG:boto:Method: HEAD 2024-03-04T18:29:55.669 DEBUG:boto:Path: /ppfahn-59/ 2024-03-04T18:29:55.669 DEBUG:boto:Data: 2024-03-04T18:29:55.669 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.669 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:29:55.669 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.669 DEBUG:boto:Params: {} 2024-03-04T18:29:55.669 DEBUG:boto:Token: None 2024-03-04T18:29:55.669 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-59/ 2024-03-04T18:29:55.669 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU= 2024-03-04T18:29:55.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU=', 'Content-Length': '0'} 2024-03-04T18:29:55.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000717c007c42199623-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.674 DEBUG:boto:path=/ppfahn-59/obj4 2024-03-04T18:29:55.674 DEBUG:boto:auth_path=/ppfahn-59/obj4 2024-03-04T18:29:55.674 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.674 DEBUG:boto:Path: /ppfahn-59/obj4 2024-03-04T18:29:55.674 DEBUG:boto:Data: 2024-03-04T18:29:55.674 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:29:55.674 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:29:55.674 DEBUG:boto:Port: 8000 2024-03-04T18:29:55.674 DEBUG:boto:Params: {} 2024-03-04T18:29:55.675 DEBUG:boto:Token: None 2024-03-04T18:29:55.675 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-59/obj4 2024-03-04T18:29:55.675 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8opWzLzevpYNVafadPjZoIEA/MI= 2024-03-04T18:29:55.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8opWzLzevpYNVafadPjZoIEA/MI='} 2024-03-04T18:29:55.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e028d41bb076493-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.689 DEBUG:boto:path=/ppfahn-60/ 2024-03-04T18:29:55.689 DEBUG:boto:auth_path=/ppfahn-60/ 2024-03-04T18:29:55.689 DEBUG:boto:Method: HEAD 2024-03-04T18:29:55.689 DEBUG:boto:Path: /ppfahn-60/ 2024-03-04T18:29:55.689 DEBUG:boto:Data: 2024-03-04T18:29:55.689 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.689 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:29:55.689 DEBUG:boto:Port: 8001 2024-03-04T18:29:55.689 DEBUG:boto:Params: {} 2024-03-04T18:29:55.690 DEBUG:boto:Token: None 2024-03-04T18:29:55.690 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-60/ 2024-03-04T18:29:55.690 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio= 2024-03-04T18:29:55.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio=', 'Content-Length': '0'} 2024-03-04T18:29:55.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d723d8bd88490c71-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.695 DEBUG:boto:path=/ppfahn-60/obj1 2024-03-04T18:29:55.695 DEBUG:boto:auth_path=/ppfahn-60/obj1 2024-03-04T18:29:55.695 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.695 DEBUG:boto:Path: /ppfahn-60/obj1 2024-03-04T18:29:55.695 DEBUG:boto:Data: 2024-03-04T18:29:55.695 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:29:55.695 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:29:55.695 DEBUG:boto:Port: 8001 2024-03-04T18:29:55.695 DEBUG:boto:Params: {} 2024-03-04T18:29:55.696 DEBUG:boto:Token: None 2024-03-04T18:29:55.696 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-60/obj1 2024-03-04T18:29:55.699 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GLhHy8wwY6CA7o1Xokle4yk81uE= 2024-03-04T18:29:55.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GLhHy8wwY6CA7o1Xokle4yk81uE='} 2024-03-04T18:29:55.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005daa37c12dbd891b-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.715 DEBUG:boto:path=/ppfahn-60/ 2024-03-04T18:29:55.716 DEBUG:boto:auth_path=/ppfahn-60/ 2024-03-04T18:29:55.716 DEBUG:boto:Method: HEAD 2024-03-04T18:29:55.716 DEBUG:boto:Path: /ppfahn-60/ 2024-03-04T18:29:55.716 DEBUG:boto:Data: 2024-03-04T18:29:55.716 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.716 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:29:55.716 DEBUG:boto:Port: 8001 2024-03-04T18:29:55.716 DEBUG:boto:Params: {} 2024-03-04T18:29:55.716 DEBUG:boto:Token: None 2024-03-04T18:29:55.717 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-60/ 2024-03-04T18:29:55.717 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio= 2024-03-04T18:29:55.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio=', 'Content-Length': '0'} 2024-03-04T18:29:55.721 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004be8b08183969e64-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.721 DEBUG:boto:path=/ppfahn-60/obj2 2024-03-04T18:29:55.722 DEBUG:boto:auth_path=/ppfahn-60/obj2 2024-03-04T18:29:55.722 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.722 DEBUG:boto:Path: /ppfahn-60/obj2 2024-03-04T18:29:55.722 DEBUG:boto:Data: 2024-03-04T18:29:55.722 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:29:55.722 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:29:55.722 DEBUG:boto:Port: 8001 2024-03-04T18:29:55.722 DEBUG:boto:Params: {} 2024-03-04T18:29:55.722 DEBUG:boto:Token: None 2024-03-04T18:29:55.722 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-60/obj2 2024-03-04T18:29:55.723 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uden4NgtxYaHWZ0ecOZtkMFFO48= 2024-03-04T18:29:55.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uden4NgtxYaHWZ0ecOZtkMFFO48='} 2024-03-04T18:29:55.768 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc6663e77abbb991-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.768 DEBUG:boto:path=/ppfahn-60/ 2024-03-04T18:29:55.768 DEBUG:boto:auth_path=/ppfahn-60/ 2024-03-04T18:29:55.769 DEBUG:boto:Method: HEAD 2024-03-04T18:29:55.769 DEBUG:boto:Path: /ppfahn-60/ 2024-03-04T18:29:55.769 DEBUG:boto:Data: 2024-03-04T18:29:55.769 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.769 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:29:55.769 DEBUG:boto:Port: 8001 2024-03-04T18:29:55.769 DEBUG:boto:Params: {} 2024-03-04T18:29:55.769 DEBUG:boto:Token: None 2024-03-04T18:29:55.769 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-60/ 2024-03-04T18:29:55.769 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio= 2024-03-04T18:29:55.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio=', 'Content-Length': '0'} 2024-03-04T18:29:55.773 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e382c2bb2d75b68e-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.774 DEBUG:boto:path=/ppfahn-60/obj3 2024-03-04T18:29:55.774 DEBUG:boto:auth_path=/ppfahn-60/obj3 2024-03-04T18:29:55.774 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.774 DEBUG:boto:Path: /ppfahn-60/obj3 2024-03-04T18:29:55.774 DEBUG:boto:Data: 2024-03-04T18:29:55.774 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:29:55.774 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:29:55.774 DEBUG:boto:Port: 8001 2024-03-04T18:29:55.774 DEBUG:boto:Params: {} 2024-03-04T18:29:55.774 DEBUG:boto:Token: None 2024-03-04T18:29:55.775 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-60/obj3 2024-03-04T18:29:55.775 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:lPuIlXurJf303NWLY5mVhhp1eL4= 2024-03-04T18:29:55.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lPuIlXurJf303NWLY5mVhhp1eL4='} 2024-03-04T18:29:55.788 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001971693ee3a404f5-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.789 DEBUG:boto:path=/ppfahn-60/ 2024-03-04T18:29:55.789 DEBUG:boto:auth_path=/ppfahn-60/ 2024-03-04T18:29:55.789 DEBUG:boto:Method: HEAD 2024-03-04T18:29:55.789 DEBUG:boto:Path: /ppfahn-60/ 2024-03-04T18:29:55.789 DEBUG:boto:Data: 2024-03-04T18:29:55.789 DEBUG:boto:Headers: {} 2024-03-04T18:29:55.789 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:29:55.789 DEBUG:boto:Port: 8001 2024-03-04T18:29:55.789 DEBUG:boto:Params: {} 2024-03-04T18:29:55.790 DEBUG:boto:Token: None 2024-03-04T18:29:55.790 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-60/ 2024-03-04T18:29:55.790 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio= 2024-03-04T18:29:55.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio=', 'Content-Length': '0'} 2024-03-04T18:29:55.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d25ea4c53dd4c59-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.794 DEBUG:boto:path=/ppfahn-60/obj4 2024-03-04T18:29:55.794 DEBUG:boto:auth_path=/ppfahn-60/obj4 2024-03-04T18:29:55.795 DEBUG:boto:Method: PUT 2024-03-04T18:29:55.795 DEBUG:boto:Path: /ppfahn-60/obj4 2024-03-04T18:29:55.795 DEBUG:boto:Data: 2024-03-04T18:29:55.795 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:29:55.795 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:29:55.795 DEBUG:boto:Port: 8001 2024-03-04T18:29:55.795 DEBUG:boto:Params: {} 2024-03-04T18:29:55.795 DEBUG:boto:Token: None 2024-03-04T18:29:55.795 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:29:55 GMT /ppfahn-60/obj4 2024-03-04T18:29:55.795 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:g7MzNMdRCENrBO4fTsERu/L71Cc= 2024-03-04T18:29:55.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:g7MzNMdRCENrBO4fTsERu/L71Cc='} 2024-03-04T18:29:55.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004295622cb06c7141-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:29:55.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:55.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:55.841 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:55.981 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:55.981 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:56.091 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T18:29:56.116 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709576995.430314_531.1","last_update":"2024-03-04T18:29:55.430314Z"}] 2024-03-04T18:29:56.116 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:29:56.117 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:29:56.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:56.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:56.117 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:56.220+0000 7f9bb5077980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:29:56.220+0000 7f9bb5077980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:56.253 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:56Z"} 2024-03-04T18:29:56.253 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:56Z"} 2024-03-04T18:29:56.253 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:29:56.253 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:29:56.254 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:29:56.254 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T18:30:01.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:01.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:01.255 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:01.299 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:01.299 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:01.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:01.376+0000 7fa04225c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:01.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:01.380+0000 7fa04225c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:01.410 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:01Z"} 2024-03-04T18:30:01.411 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:01Z"} 2024-03-04T18:30:01.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:30:01.411 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:30:01.411 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:30:01.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T18:30:04.116 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:30:04.113+0000 7f5c190f6640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:30:06.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:06.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:06.413 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:06.464 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:06.464 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:06.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:06.520+0000 7f881b671980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:06.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:06.524+0000 7f881b671980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:06.552 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.690390_325.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.690390Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:06Z"} 2024-03-04T18:30:06.552 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.690390_325.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.690390Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:06Z"} 2024-03-04T18:30:06.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.690390_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.690390Z', 'realm_epoch': 2}}] 2024-03-04T18:30:06.553 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:30:06.553 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.690390_325.1'} 2024-03-04T18:30:06.553 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575077.690390_325.1 2024-03-04T18:30:11.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:11.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:11.555 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:11.598 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:11.598 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:11.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:11.672+0000 7fc494f6d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:11.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:11.672+0000 7fc494f6d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:11.704 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:11Z"} 2024-03-04T18:30:11.704 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:11Z"} 2024-03-04T18:30:11.704 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:30:11.704 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:30:11.705 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:30:11.705 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:30:16.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:16.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:16.707 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:16.750 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:16.751 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:16.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:16.820+0000 7f64c014a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:16.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:16.824+0000 7f64c014a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:16.854 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:16Z"} 2024-03-04T18:30:16.854 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:16Z"} 2024-03-04T18:30:16.855 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:30:16.855 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:30:16.855 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:30:16.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:30:21.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:21.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:21.856 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:21.900 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:21.900 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:21.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:21.968+0000 7f2ffc3a6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:21.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:21.972+0000 7f2ffc3a6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:22.004 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:21Z"} 2024-03-04T18:30:22.005 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:21Z"} 2024-03-04T18:30:22.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:30:22.005 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:30:22.005 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:30:22.005 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:30:27.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:27.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:27.006 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:27.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:27.120+0000 7fc6839f4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:27.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:27.120+0000 7fc6839f4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:27.151 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:27Z"} 2024-03-04T18:30:27.151 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:27Z"} 2024-03-04T18:30:27.151 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:30:27.151 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:30:27.151 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:30:27.152 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:30:32.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:32.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:32.153 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:32.199 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:32.199 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:32.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:32.276+0000 7f6d8ff96980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:32.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:32.276+0000 7f6d8ff96980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:32.338 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.486968_316.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.486968Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:32Z"} 2024-03-04T18:30:32.338 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.486968_316.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.486968Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:32Z"} 2024-03-04T18:30:32.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.486968_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.486968Z', 'realm_epoch': 2}}] 2024-03-04T18:30:32.338 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:30:32.338 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.486968_316.1'} 2024-03-04T18:30:32.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.486968_316.1 2024-03-04T18:30:37.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:37.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:37.340 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:37.383 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:37.408 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:37.440+0000 7fd09b4e7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:37.444+0000 7fd09b4e7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:37.471 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:37Z"} 2024-03-04T18:30:37.472 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:37Z"} 2024-03-04T18:30:37.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:30:37.472 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:30:37.472 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:30:37.472 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:30:42.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:42.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:42.473 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:42.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:42.588+0000 7f97e1577980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:42.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:42.592+0000 7f97e1577980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:42.621 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:42Z"} 2024-03-04T18:30:42.622 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:42Z"} 2024-03-04T18:30:42.622 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:30:42.622 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:30:42.622 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:30:42.622 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:30:47.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:47.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:47.624 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:47.666 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:47.666 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:47.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:47.740+0000 7f1e1014f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:47.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:47.740+0000 7f1e1014f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:47.788 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:47Z"} 2024-03-04T18:30:47.788 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:47Z"} 2024-03-04T18:30:47.788 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:30:47.789 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:30:47.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:30:47.789 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:30:52.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:52.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:52.791 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:52.835 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:52.835 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:52.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:52.908+0000 7f314b36c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:52.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:52.908+0000 7f314b36c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:52.940 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:52Z"} 2024-03-04T18:30:52.940 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:52Z"} 2024-03-04T18:30:52.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:30:52.943 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:30:52.944 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:30:52.944 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:30:57.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:57.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:57.945 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:57.987 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:57.987 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:58.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:58.056+0000 7f533a7c2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:58.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:30:58.056+0000 7f533a7c2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:58.097 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:58Z"} 2024-03-04T18:30:58.097 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:58Z"} 2024-03-04T18:30:58.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.523626_319.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.523626Z', 'realm_epoch': 2}}] 2024-03-04T18:30:58.097 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:30:58.097 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.523626_319.1'} 2024-03-04T18:30:58.098 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.523626_319.1 2024-03-04T18:31:03.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:03.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:03.099 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:03.142 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:03.143 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:03.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:03.207+0000 7ff3bdb51980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:03.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:03.207+0000 7ff3bdb51980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:03.237 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:03Z"} 2024-03-04T18:31:03.237 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:03Z"} 2024-03-04T18:31:03.237 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:31:03.237 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:31:03.237 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:31:03.237 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T18:31:08.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:08.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:08.239 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:08.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:08.383+0000 7f23e7917980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:08.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:08.387+0000 7f23e7917980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:08.414 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:08Z"} 2024-03-04T18:31:08.415 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:08Z"} 2024-03-04T18:31:08.415 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:31:08.415 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:31:08.415 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:31:08.415 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.310269_305.1 2024-03-04T18:31:13.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:13.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:13.416 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:13.460 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:13.460 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:13.523+0000 7fa896f52980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:13.523+0000 7fa896f52980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:13.553 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:13Z"} 2024-03-04T18:31:13.554 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:13Z"} 2024-03-04T18:31:13.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:31:13.554 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:31:13.554 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:31:13.554 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:31:18.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:18.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:18.556 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:18.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:18.659+0000 7fb6b004f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:18.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:18.663+0000 7fb6b004f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:18.699 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:18Z"} 2024-03-04T18:31:18.699 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:18Z"} 2024-03-04T18:31:18.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:31:18.699 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:31:18.699 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:31:18.699 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.310269_305.1 2024-03-04T18:31:23.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:23.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:23.701 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:23.743 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:23.743 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:23.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:23.799+0000 7f3a9d3b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:23.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:23.799+0000 7f3a9d3b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:23.831 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:23Z"} 2024-03-04T18:31:23.832 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:23Z"} 2024-03-04T18:31:23.832 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:31:23.832 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:31:23.832 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:31:23.832 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:31:23.954 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:31:23.951+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:31:28.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:28.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:28.833 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:28.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:28.959+0000 7f7c77c6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:28.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:28.959+0000 7f7c77c6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:29.003 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:28Z"} 2024-03-04T18:31:29.003 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:28Z"} 2024-03-04T18:31:29.003 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:31:29.003 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:31:29.004 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:31:29.004 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T18:31:34.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:34.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:34.007 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:34.050 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:34.050 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:34.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:34.107+0000 7f8b33ed6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:34.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:34.111+0000 7f8b33ed6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:34.143 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:34Z"} 2024-03-04T18:31:34.143 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:34Z"} 2024-03-04T18:31:34.143 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:31:34.144 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:31:34.144 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:31:34.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:31:35.375 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:31:35.375+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:31:39.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:39.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:39.146 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:39.189 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:39.189 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:39.259+0000 7f5574fde980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:39.259+0000 7f5574fde980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:39.293 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:39Z"} 2024-03-04T18:31:39.293 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:39Z"} 2024-03-04T18:31:39.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T18:31:39.294 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:31:39.294 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T18:31:39.294 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.380518_307.1 2024-03-04T18:31:44.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:44.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:44.296 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:44.342 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:44.342 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:44.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:44.411+0000 7faa15d99980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:44.411+0000 7faa15d99980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:44.444 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:44Z"} 2024-03-04T18:31:44.444 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:44Z"} 2024-03-04T18:31:44.444 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:31:44.444 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:31:44.444 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:31:44.445 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:31:49.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:49.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:49.446 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:49.488 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:49.488 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:49.555+0000 7ff7aab1b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:49.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:49.559+0000 7ff7aab1b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:49.584 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:49Z"} 2024-03-04T18:31:49.585 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:49Z"} 2024-03-04T18:31:49.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:31:49.585 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:31:49.585 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:31:49.585 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:31:54.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:54.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:54.587 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:54.630 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:54.630 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:54.711+0000 7f18f6b0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:54.711+0000 7f18f6b0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:54.742 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:54Z"} 2024-03-04T18:31:54.742 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:54Z"} 2024-03-04T18:31:54.742 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:31:54.742 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:31:54.742 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:31:54.742 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T18:31:59.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:59.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:59.743 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:59.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:59.859+0000 7f4a51244980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:59.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:31:59.859+0000 7f4a51244980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:59.884 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:59Z"} 2024-03-04T18:31:59.884 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:59Z"} 2024-03-04T18:31:59.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:31:59.885 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:31:59.885 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:31:59.885 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:32:04.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:04.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:04.886 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:04.931 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:04.931 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:05.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:05.010+0000 7f5a80faf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:05.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:05.010+0000 7f5a80faf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:05.043 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:05Z"} 2024-03-04T18:32:05.043 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:05Z"} 2024-03-04T18:32:05.044 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:32:05.044 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:32:05.044 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:32:05.044 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:32:10.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:10.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:10.045 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:10.089 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:10.089 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:10.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:10.170+0000 7fc41f4e3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:10.170+0000 7fc41f4e3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:10.205 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:10Z"} 2024-03-04T18:32:10.205 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:10Z"} 2024-03-04T18:32:10.205 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:32:10.206 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:32:10.206 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:32:10.206 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:32:15.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:15.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:15.207 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:15.249 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:15.249 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:15.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:15.310+0000 7fdc20fbb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:15.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:15.310+0000 7fdc20fbb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:15.338 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:15Z"} 2024-03-04T18:32:15.339 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:15Z"} 2024-03-04T18:32:15.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:32:15.339 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:32:15.339 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:32:15.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T18:32:20.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:20.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:20.341 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:20.385 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:20.385 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:20.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:20.446+0000 7f5519727980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:20.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:20.446+0000 7f5519727980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:20.472 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:20Z"} 2024-03-04T18:32:20.472 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:20Z"} 2024-03-04T18:32:20.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:32:20.473 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:32:20.473 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:32:20.473 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T18:32:24.146 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:32:24.142+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:32:25.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:25.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:25.475 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:25.523 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:25.524 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:25.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:25.594+0000 7f570f1ac980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:25.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:25.598+0000 7f570f1ac980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:25.625 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:25Z"} 2024-03-04T18:32:25.625 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:25Z"} 2024-03-04T18:32:25.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:32:25.626 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:32:25.626 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:32:25.626 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:32:30.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:30.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:30.627 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:30.669 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:30.669 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:30.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:30.742+0000 7f448c9d5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:30.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:30.742+0000 7f448c9d5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:30.772 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:30Z"} 2024-03-04T18:32:30.772 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:30Z"} 2024-03-04T18:32:30.772 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:32:30.773 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:32:30.773 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:32:30.773 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:32:35.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:35.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:35.774 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:35.816 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:35.816 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:35.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:35.878+0000 7f45f890d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:35.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:35.878+0000 7f45f890d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:35.906 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.489493_317.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.489493Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:35Z"} 2024-03-04T18:32:35.907 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.489493_317.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.489493Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:35Z"} 2024-03-04T18:32:35.907 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.489493_317.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.489493Z', 'realm_epoch': 2}}] 2024-03-04T18:32:35.907 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:32:35.907 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.489493_317.1'} 2024-03-04T18:32:35.907 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.489493_317.1 2024-03-04T18:32:40.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:40.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:40.909 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:40.951 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:40.952 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:41.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:41.030+0000 7f1dcf18c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:41.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:41.030+0000 7f1dcf18c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:41.058 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:41Z"} 2024-03-04T18:32:41.058 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:41Z"} 2024-03-04T18:32:41.058 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:32:41.059 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:32:41.059 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:32:41.059 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:32:46.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:46.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:46.060 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:46.104 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:46.104 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:46.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:46.166+0000 7fc0195f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:46.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:46.166+0000 7fc0195f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:46.195 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:46Z"} 2024-03-04T18:32:46.196 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:46Z"} 2024-03-04T18:32:46.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:32:46.196 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:32:46.196 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:32:46.196 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:32:51.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:51.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:51.197 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:51.240 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:51.240 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:51.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:51.306+0000 7f8f00ab0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:51.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:51.306+0000 7f8f00ab0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:51.333 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:51Z"} 2024-03-04T18:32:51.333 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:51Z"} 2024-03-04T18:32:51.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:32:51.334 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:32:51.334 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:32:51.334 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:32:56.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:56.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:56.336 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:56.378 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:56.379 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:56.454+0000 7fdedefa2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:56.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:32:56.454+0000 7fdedefa2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:56.490 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:56Z"} 2024-03-04T18:32:56.490 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:56Z"} 2024-03-04T18:32:56.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:32:56.490 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:32:56.490 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:32:56.490 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:33:01.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:01.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:01.492 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:01.533 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:01.533 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:01.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:01.598+0000 7fac4e753980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:01.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:01.602+0000 7fac4e753980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:01.629 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:01Z"} 2024-03-04T18:33:01.629 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:01Z"} 2024-03-04T18:33:01.630 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:33:01.630 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:33:01.630 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:33:01.630 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:33:06.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:06.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:06.631 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:06.675 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:06.675 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:06.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:06.757+0000 7f7651954980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:06.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:06.761+0000 7f7651954980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:06.807 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:06Z"} 2024-03-04T18:33:06.807 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:06Z"} 2024-03-04T18:33:06.808 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:33:06.808 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:33:06.808 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:33:06.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T18:33:11.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:11.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:11.810 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:11.852 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:11.852 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:11.925+0000 7f5d75d0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:11.925+0000 7f5d75d0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:11.956 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:11Z"} 2024-03-04T18:33:11.956 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:11Z"} 2024-03-04T18:33:11.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:33:11.956 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:33:11.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:33:11.957 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T18:33:16.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:16.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:16.958 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:17.003 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:17.003 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:17.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:17.073+0000 7fb3c2d59980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:17.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:17.077+0000 7fb3c2d59980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:17.103 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:17Z"} 2024-03-04T18:33:17.103 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:17Z"} 2024-03-04T18:33:17.103 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:33:17.103 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:33:17.103 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:33:17.103 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:33:22.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:22.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:22.105 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:22.150 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:22.150 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:22.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:22.209+0000 7f61781e9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:22.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:22.209+0000 7f61781e9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:22.239 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:22Z"} 2024-03-04T18:33:22.239 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:22Z"} 2024-03-04T18:33:22.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:33:22.239 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:33:22.239 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:33:22.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T18:33:27.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:27.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:27.240 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:27.284 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:27.284 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:27.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:27.357+0000 7fe095ffa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:27.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:27.357+0000 7fe095ffa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:27.389 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:27Z"} 2024-03-04T18:33:27.390 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:27Z"} 2024-03-04T18:33:27.390 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:33:27.390 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:33:27.390 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:33:27.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T18:33:32.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:32.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:32.392 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:32.442 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:32.443 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:32.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:32.509+0000 7fee7556e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:32.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:32.513+0000 7fee7556e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:32.543 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:32Z"} 2024-03-04T18:33:32.543 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:32Z"} 2024-03-04T18:33:32.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T18:33:32.544 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:33:32.544 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T18:33:32.544 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.380518_307.1 2024-03-04T18:33:37.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:37.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:37.545 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:37.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:37.653+0000 7f9836333980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:37.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:37.653+0000 7f9836333980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:37.681 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:37Z"} 2024-03-04T18:33:37.681 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:37Z"} 2024-03-04T18:33:37.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:33:37.682 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:33:37.682 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:33:37.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T18:33:42.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:42.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:42.683 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:42.725 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:42.726 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:42.785+0000 7fd4b0120980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:42.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:42.789+0000 7fd4b0120980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:42.823 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:42Z"} 2024-03-04T18:33:42.824 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:42Z"} 2024-03-04T18:33:42.824 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:33:42.824 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:33:42.824 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:33:42.824 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:33:47.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:47.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:47.826 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:47.868 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:47.868 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:47.941+0000 7f565a499980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:47.941+0000 7f565a499980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:47.978 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:47Z"} 2024-03-04T18:33:47.978 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:47Z"} 2024-03-04T18:33:47.978 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:33:47.978 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:33:47.978 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:33:47.978 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:33:52.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:52.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:52.979 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:53.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:53.089+0000 7f325dcc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:53.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:53.093+0000 7f325dcc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:53.119 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:53Z"} 2024-03-04T18:33:53.119 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:53Z"} 2024-03-04T18:33:53.119 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:33:53.120 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:33:53.120 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:33:53.120 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.310269_305.1 2024-03-04T18:33:58.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:58.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:58.121 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:58.164 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:58.164 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:58.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:58.241+0000 7f86cf5b6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:58.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:33:58.245+0000 7f86cf5b6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:58.271 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:58Z"} 2024-03-04T18:33:58.272 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:58Z"} 2024-03-04T18:33:58.272 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:33:58.272 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:33:58.272 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:33:58.272 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.310269_305.1 2024-03-04T18:34:03.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:03.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:03.273 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:03.317 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:03.317 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:03.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:03.381+0000 7f30f9b93980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:03.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:03.381+0000 7f30f9b93980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:03Z"} 2024-03-04T18:34:03.412 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:03Z"} 2024-03-04T18:34:03.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:34:03.412 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:34:03.412 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:34:03.412 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T18:34:08.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:08.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:08.414 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:08.455 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:08.455 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:08.517+0000 7f4808985980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:08.521+0000 7f4808985980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:08.549 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:08Z"} 2024-03-04T18:34:08.549 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:08Z"} 2024-03-04T18:34:08.549 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:34:08.549 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:34:08.549 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:34:08.549 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:34:13.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:13.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:13.551 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:13.593 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:13.593 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:13.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:13.660+0000 7f86da25c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:13.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:13.660+0000 7f86da25c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:13.688 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:13Z"} 2024-03-04T18:34:13.688 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:13Z"} 2024-03-04T18:34:13.688 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:34:13.689 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:34:13.689 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:34:13.689 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:34:18.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:18.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:18.690 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:18.734 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:18.734 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:18.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:18.796+0000 7fc0a5403980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:18.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:18.796+0000 7fc0a5403980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:18.825 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:18Z"} 2024-03-04T18:34:18.825 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:18Z"} 2024-03-04T18:34:18.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:34:18.826 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:34:18.826 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:34:18.826 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:34:23.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:23.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:23.827 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:23.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:23.940+0000 7f2c6527d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:23.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:23.940+0000 7f2c6527d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:23.973 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:23Z"} 2024-03-04T18:34:23.973 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:23Z"} 2024-03-04T18:34:23.973 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.422483_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.422483Z', 'realm_epoch': 2}}] 2024-03-04T18:34:23.973 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:34:23.973 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.422483_312.1'} 2024-03-04T18:34:23.973 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.422483_312.1 2024-03-04T18:34:28.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:28.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:28.975 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:29.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:29.092+0000 7f487825e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:29.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:29.092+0000 7f487825e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:29.126 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:29Z"} 2024-03-04T18:34:29.126 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:29Z"} 2024-03-04T18:34:29.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:34:29.126 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:34:29.126 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:34:29.127 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:34:34.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:34.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:34.128 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:34.178 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:34.178 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:34.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:34.240+0000 7fe6e55fb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:34.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:34.240+0000 7fe6e55fb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:34.270 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:34Z"} 2024-03-04T18:34:34.271 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:34Z"} 2024-03-04T18:34:34.271 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:34:34.271 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:34:34.271 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:34:34.271 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:34:39.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:39.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:39.272 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:39.315 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:39.315 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:39.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:39.388+0000 7fda685bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:39.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:39.388+0000 7fda685bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:39.421 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:39Z"} 2024-03-04T18:34:39.421 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:39Z"} 2024-03-04T18:34:39.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:34:39.421 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:34:39.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:34:39.422 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:34:44.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:44.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:44.423 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:44.464 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:44.464 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:44.536+0000 7f181fb6b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:44.536+0000 7f181fb6b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:44.570 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:44Z"} 2024-03-04T18:34:44.570 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:44Z"} 2024-03-04T18:34:44.570 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:34:44.570 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:34:44.571 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:34:44.571 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:34:49.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:49.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:49.573 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:49.618 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:49.618 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:49.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:49.684+0000 7f1d5bff2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:49.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:49.688+0000 7f1d5bff2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:49.718 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:49Z"} 2024-03-04T18:34:49.718 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:49Z"} 2024-03-04T18:34:49.719 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:34:49.719 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:34:49.719 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:34:49.719 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:34:54.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:54.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:54.721 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:54.832+0000 7fcd2c44f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:54.832+0000 7fcd2c44f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:54.862 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:54Z"} 2024-03-04T18:34:54.862 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:54Z"} 2024-03-04T18:34:54.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:34:54.863 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:34:54.863 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:34:54.863 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:34:59.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:59.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:59.864 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:59.899 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:59.899 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:59.968+0000 7f44f2b17980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:34:59.968+0000 7f44f2b17980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:59.999 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:59Z"} 2024-03-04T18:34:59.999 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:59Z"} 2024-03-04T18:35:00.000 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:35:00.000 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T18:35:00.000 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:35:00.000 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T18:35:05.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2024-03-04T18:35:05.004 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-61 2024-03-04T18:35:05.004 DEBUG:boto:path=/ppfahn-61/ 2024-03-04T18:35:05.004 DEBUG:boto:auth_path=/ppfahn-61/ 2024-03-04T18:35:05.004 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.004 DEBUG:boto:Path: /ppfahn-61/ 2024-03-04T18:35:05.004 DEBUG:boto:Data: 2024-03-04T18:35:05.004 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.005 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:35:05.005 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.005 DEBUG:boto:Params: {} 2024-03-04T18:35:05.005 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:35:05.005 DEBUG:boto:Token: None 2024-03-04T18:35:05.005 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-61/ 2024-03-04T18:35:05.006 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tD/RyF0dJuJj1eNNLOsAX0LkwuA= 2024-03-04T18:35:05.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tD/RyF0dJuJj1eNNLOsAX0LkwuA=', 'Content-Length': '0'} 2024-03-04T18:35:05.052 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f60fffd7d08f4af8-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.052 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-62 2024-03-04T18:35:05.052 DEBUG:boto:path=/ppfahn-62/ 2024-03-04T18:35:05.052 DEBUG:boto:auth_path=/ppfahn-62/ 2024-03-04T18:35:05.052 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.052 DEBUG:boto:Path: /ppfahn-62/ 2024-03-04T18:35:05.052 DEBUG:boto:Data: 2024-03-04T18:35:05.052 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.053 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:35:05.053 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.053 DEBUG:boto:Params: {} 2024-03-04T18:35:05.053 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:35:05.053 DEBUG:boto:Token: None 2024-03-04T18:35:05.053 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-62/ 2024-03-04T18:35:05.053 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:T0J3F37cTJywCyWUCbcGWiexdKc= 2024-03-04T18:35:05.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T0J3F37cTJywCyWUCbcGWiexdKc=', 'Content-Length': '0'} 2024-03-04T18:35:05.189 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b57036635e476a8f-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.189 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-63 2024-03-04T18:35:05.189 DEBUG:boto:path=/ppfahn-63/ 2024-03-04T18:35:05.189 DEBUG:boto:auth_path=/ppfahn-63/ 2024-03-04T18:35:05.189 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.189 DEBUG:boto:Path: /ppfahn-63/ 2024-03-04T18:35:05.190 DEBUG:boto:Data: 2024-03-04T18:35:05.190 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.190 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:35:05.190 DEBUG:boto:Port: 8001 2024-03-04T18:35:05.190 DEBUG:boto:Params: {} 2024-03-04T18:35:05.190 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T18:35:05.190 DEBUG:boto:Token: None 2024-03-04T18:35:05.190 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-63/ 2024-03-04T18:35:05.191 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2oEw/0OQ+aJdyjuB7vCLnhaKiJQ= 2024-03-04T18:35:05.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2oEw/0OQ+aJdyjuB7vCLnhaKiJQ=', 'Content-Length': '0'} 2024-03-04T18:35:05.305 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e6d9c3ed2a38a86f-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.305 DEBUG:boto:path=/ppfahn-61/ 2024-03-04T18:35:05.305 DEBUG:boto:auth_path=/ppfahn-61/ 2024-03-04T18:35:05.305 DEBUG:boto:Method: HEAD 2024-03-04T18:35:05.305 DEBUG:boto:Path: /ppfahn-61/ 2024-03-04T18:35:05.305 DEBUG:boto:Data: 2024-03-04T18:35:05.305 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.305 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:35:05.305 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.306 DEBUG:boto:Params: {} 2024-03-04T18:35:05.306 DEBUG:boto:Token: None 2024-03-04T18:35:05.306 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-61/ 2024-03-04T18:35:05.306 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg= 2024-03-04T18:35:05.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg=', 'Content-Length': '0'} 2024-03-04T18:35:05.308 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a70995c43080d9a-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.309 DEBUG:boto:path=/ppfahn-61/obj1 2024-03-04T18:35:05.309 DEBUG:boto:auth_path=/ppfahn-61/obj1 2024-03-04T18:35:05.309 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.309 DEBUG:boto:Path: /ppfahn-61/obj1 2024-03-04T18:35:05.309 DEBUG:boto:Data: 2024-03-04T18:35:05.309 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:35:05.309 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:35:05.309 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.309 DEBUG:boto:Params: {} 2024-03-04T18:35:05.309 DEBUG:boto:Token: None 2024-03-04T18:35:05.310 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-61/obj1 2024-03-04T18:35:05.310 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:5B/s7WLinUK4gtsQzce/3IOX96Y= 2024-03-04T18:35:05.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5B/s7WLinUK4gtsQzce/3IOX96Y='} 2024-03-04T18:35:05.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9c0f6fa902cf358-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.317 DEBUG:boto:path=/ppfahn-61/ 2024-03-04T18:35:05.317 DEBUG:boto:auth_path=/ppfahn-61/ 2024-03-04T18:35:05.317 DEBUG:boto:Method: HEAD 2024-03-04T18:35:05.317 DEBUG:boto:Path: /ppfahn-61/ 2024-03-04T18:35:05.317 DEBUG:boto:Data: 2024-03-04T18:35:05.317 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.317 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:35:05.317 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.317 DEBUG:boto:Params: {} 2024-03-04T18:35:05.317 DEBUG:boto:Token: None 2024-03-04T18:35:05.318 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-61/ 2024-03-04T18:35:05.318 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg= 2024-03-04T18:35:05.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg=', 'Content-Length': '0'} 2024-03-04T18:35:05.320 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005498287dd0b58bca-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.320 DEBUG:boto:path=/ppfahn-61/obj2 2024-03-04T18:35:05.320 DEBUG:boto:auth_path=/ppfahn-61/obj2 2024-03-04T18:35:05.320 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.320 DEBUG:boto:Path: /ppfahn-61/obj2 2024-03-04T18:35:05.320 DEBUG:boto:Data: 2024-03-04T18:35:05.321 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:35:05.321 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:35:05.321 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.321 DEBUG:boto:Params: {} 2024-03-04T18:35:05.321 DEBUG:boto:Token: None 2024-03-04T18:35:05.321 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-61/obj2 2024-03-04T18:35:05.321 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cGewrvDG/wTE1IgAJMmnMVzp8Jc= 2024-03-04T18:35:05.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cGewrvDG/wTE1IgAJMmnMVzp8Jc='} 2024-03-04T18:35:05.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e607e4328837802-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.327 DEBUG:boto:path=/ppfahn-61/ 2024-03-04T18:35:05.327 DEBUG:boto:auth_path=/ppfahn-61/ 2024-03-04T18:35:05.327 DEBUG:boto:Method: HEAD 2024-03-04T18:35:05.327 DEBUG:boto:Path: /ppfahn-61/ 2024-03-04T18:35:05.327 DEBUG:boto:Data: 2024-03-04T18:35:05.327 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.327 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:35:05.328 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.328 DEBUG:boto:Params: {} 2024-03-04T18:35:05.328 DEBUG:boto:Token: None 2024-03-04T18:35:05.328 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-61/ 2024-03-04T18:35:05.328 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg= 2024-03-04T18:35:05.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg=', 'Content-Length': '0'} 2024-03-04T18:35:05.330 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddf5a637dd142859-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.331 DEBUG:boto:path=/ppfahn-61/obj3 2024-03-04T18:35:05.331 DEBUG:boto:auth_path=/ppfahn-61/obj3 2024-03-04T18:35:05.331 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.331 DEBUG:boto:Path: /ppfahn-61/obj3 2024-03-04T18:35:05.331 DEBUG:boto:Data: 2024-03-04T18:35:05.331 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:35:05.331 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:35:05.331 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.331 DEBUG:boto:Params: {} 2024-03-04T18:35:05.331 DEBUG:boto:Token: None 2024-03-04T18:35:05.331 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-61/obj3 2024-03-04T18:35:05.332 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+ZY+E/mQhgGqxRdHFnx85Hru0Lk= 2024-03-04T18:35:05.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+ZY+E/mQhgGqxRdHFnx85Hru0Lk='} 2024-03-04T18:35:05.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082f7c65ca622fe1e-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.376 DEBUG:boto:path=/ppfahn-61/ 2024-03-04T18:35:05.376 DEBUG:boto:auth_path=/ppfahn-61/ 2024-03-04T18:35:05.376 DEBUG:boto:Method: HEAD 2024-03-04T18:35:05.377 DEBUG:boto:Path: /ppfahn-61/ 2024-03-04T18:35:05.377 DEBUG:boto:Data: 2024-03-04T18:35:05.377 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.377 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:35:05.377 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.377 DEBUG:boto:Params: {} 2024-03-04T18:35:05.377 DEBUG:boto:Token: None 2024-03-04T18:35:05.377 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-61/ 2024-03-04T18:35:05.377 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg= 2024-03-04T18:35:05.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg=', 'Content-Length': '0'} 2024-03-04T18:35:05.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000757a3fd48401f1ab-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.380 DEBUG:boto:path=/ppfahn-61/obj4 2024-03-04T18:35:05.380 DEBUG:boto:auth_path=/ppfahn-61/obj4 2024-03-04T18:35:05.380 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.380 DEBUG:boto:Path: /ppfahn-61/obj4 2024-03-04T18:35:05.380 DEBUG:boto:Data: 2024-03-04T18:35:05.381 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:35:05.381 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:35:05.381 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.381 DEBUG:boto:Params: {} 2024-03-04T18:35:05.381 DEBUG:boto:Token: None 2024-03-04T18:35:05.381 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-61/obj4 2024-03-04T18:35:05.381 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:F7RNz9yUvfFyKBuTjJ/If3NgTvo= 2024-03-04T18:35:05.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F7RNz9yUvfFyKBuTjJ/If3NgTvo='} 2024-03-04T18:35:05.388 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d03606d6d5216535-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.388 DEBUG:boto:path=/ppfahn-62/ 2024-03-04T18:35:05.388 DEBUG:boto:auth_path=/ppfahn-62/ 2024-03-04T18:35:05.389 DEBUG:boto:Method: HEAD 2024-03-04T18:35:05.389 DEBUG:boto:Path: /ppfahn-62/ 2024-03-04T18:35:05.389 DEBUG:boto:Data: 2024-03-04T18:35:05.389 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.389 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:35:05.389 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.389 DEBUG:boto:Params: {} 2024-03-04T18:35:05.389 DEBUG:boto:Token: None 2024-03-04T18:35:05.389 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-62/ 2024-03-04T18:35:05.389 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E= 2024-03-04T18:35:05.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E=', 'Content-Length': '0'} 2024-03-04T18:35:05.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005504c68284f78706-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.395 DEBUG:boto:path=/ppfahn-62/obj1 2024-03-04T18:35:05.395 DEBUG:boto:auth_path=/ppfahn-62/obj1 2024-03-04T18:35:05.395 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.395 DEBUG:boto:Path: /ppfahn-62/obj1 2024-03-04T18:35:05.396 DEBUG:boto:Data: 2024-03-04T18:35:05.396 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:35:05.396 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:35:05.396 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.396 DEBUG:boto:Params: {} 2024-03-04T18:35:05.396 DEBUG:boto:Token: None 2024-03-04T18:35:05.396 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-62/obj1 2024-03-04T18:35:05.396 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jxl+ksOvIfHxATiwJcuuB0mArG0= 2024-03-04T18:35:05.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jxl+ksOvIfHxATiwJcuuB0mArG0='} 2024-03-04T18:35:05.419 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d848fde8df0b0eb1-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.419 DEBUG:boto:path=/ppfahn-62/ 2024-03-04T18:35:05.419 DEBUG:boto:auth_path=/ppfahn-62/ 2024-03-04T18:35:05.419 DEBUG:boto:Method: HEAD 2024-03-04T18:35:05.420 DEBUG:boto:Path: /ppfahn-62/ 2024-03-04T18:35:05.420 DEBUG:boto:Data: 2024-03-04T18:35:05.420 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.420 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:35:05.420 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.420 DEBUG:boto:Params: {} 2024-03-04T18:35:05.420 DEBUG:boto:Token: None 2024-03-04T18:35:05.420 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-62/ 2024-03-04T18:35:05.420 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E= 2024-03-04T18:35:05.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E=', 'Content-Length': '0'} 2024-03-04T18:35:05.424 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000853acf679b243e9e-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.424 DEBUG:boto:path=/ppfahn-62/obj2 2024-03-04T18:35:05.424 DEBUG:boto:auth_path=/ppfahn-62/obj2 2024-03-04T18:35:05.424 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.424 DEBUG:boto:Path: /ppfahn-62/obj2 2024-03-04T18:35:05.424 DEBUG:boto:Data: 2024-03-04T18:35:05.425 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:35:05.425 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:35:05.425 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.425 DEBUG:boto:Params: {} 2024-03-04T18:35:05.425 DEBUG:boto:Token: None 2024-03-04T18:35:05.425 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-62/obj2 2024-03-04T18:35:05.425 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:uER3QCLEEtxGDUd+h+xxSNZJxsM= 2024-03-04T18:35:05.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uER3QCLEEtxGDUd+h+xxSNZJxsM='} 2024-03-04T18:35:05.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f3e3c71f0ec03843-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.434 DEBUG:boto:path=/ppfahn-62/ 2024-03-04T18:35:05.434 DEBUG:boto:auth_path=/ppfahn-62/ 2024-03-04T18:35:05.434 DEBUG:boto:Method: HEAD 2024-03-04T18:35:05.434 DEBUG:boto:Path: /ppfahn-62/ 2024-03-04T18:35:05.434 DEBUG:boto:Data: 2024-03-04T18:35:05.434 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.434 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:35:05.434 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.434 DEBUG:boto:Params: {} 2024-03-04T18:35:05.435 DEBUG:boto:Token: None 2024-03-04T18:35:05.435 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-62/ 2024-03-04T18:35:05.435 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E= 2024-03-04T18:35:05.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E=', 'Content-Length': '0'} 2024-03-04T18:35:05.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1f7ddb5c81e02f1-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.438 DEBUG:boto:path=/ppfahn-62/obj3 2024-03-04T18:35:05.438 DEBUG:boto:auth_path=/ppfahn-62/obj3 2024-03-04T18:35:05.438 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.438 DEBUG:boto:Path: /ppfahn-62/obj3 2024-03-04T18:35:05.438 DEBUG:boto:Data: 2024-03-04T18:35:05.438 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:35:05.438 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:35:05.438 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.438 DEBUG:boto:Params: {} 2024-03-04T18:35:05.438 DEBUG:boto:Token: None 2024-03-04T18:35:05.439 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-62/obj3 2024-03-04T18:35:05.439 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:n4PzhrSkDO7n6zF87Hl8pWggfsQ= 2024-03-04T18:35:05.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:n4PzhrSkDO7n6zF87Hl8pWggfsQ='} 2024-03-04T18:35:05.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cabb989e1956581c-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.484 DEBUG:boto:path=/ppfahn-62/ 2024-03-04T18:35:05.484 DEBUG:boto:auth_path=/ppfahn-62/ 2024-03-04T18:35:05.485 DEBUG:boto:Method: HEAD 2024-03-04T18:35:05.485 DEBUG:boto:Path: /ppfahn-62/ 2024-03-04T18:35:05.485 DEBUG:boto:Data: 2024-03-04T18:35:05.485 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.485 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:35:05.485 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.485 DEBUG:boto:Params: {} 2024-03-04T18:35:05.485 DEBUG:boto:Token: None 2024-03-04T18:35:05.485 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-62/ 2024-03-04T18:35:05.485 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E= 2024-03-04T18:35:05.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E=', 'Content-Length': '0'} 2024-03-04T18:35:05.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004a1c8e9a093d665-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.489 DEBUG:boto:path=/ppfahn-62/obj4 2024-03-04T18:35:05.489 DEBUG:boto:auth_path=/ppfahn-62/obj4 2024-03-04T18:35:05.489 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.489 DEBUG:boto:Path: /ppfahn-62/obj4 2024-03-04T18:35:05.489 DEBUG:boto:Data: 2024-03-04T18:35:05.489 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:35:05.489 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:35:05.489 DEBUG:boto:Port: 8000 2024-03-04T18:35:05.490 DEBUG:boto:Params: {} 2024-03-04T18:35:05.490 DEBUG:boto:Token: None 2024-03-04T18:35:05.490 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-62/obj4 2024-03-04T18:35:05.490 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:dKx4IUsHXpNmx9XaNicHKSCST8Y= 2024-03-04T18:35:05.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dKx4IUsHXpNmx9XaNicHKSCST8Y='} 2024-03-04T18:35:05.501 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003db527dd09c4a11f-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.501 DEBUG:boto:path=/ppfahn-63/ 2024-03-04T18:35:05.502 DEBUG:boto:auth_path=/ppfahn-63/ 2024-03-04T18:35:05.502 DEBUG:boto:Method: HEAD 2024-03-04T18:35:05.502 DEBUG:boto:Path: /ppfahn-63/ 2024-03-04T18:35:05.502 DEBUG:boto:Data: 2024-03-04T18:35:05.502 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.502 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:35:05.502 DEBUG:boto:Port: 8001 2024-03-04T18:35:05.502 DEBUG:boto:Params: {} 2024-03-04T18:35:05.502 DEBUG:boto:Token: None 2024-03-04T18:35:05.503 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-63/ 2024-03-04T18:35:05.503 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I= 2024-03-04T18:35:05.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I=', 'Content-Length': '0'} 2024-03-04T18:35:05.507 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057e535deb2d41037-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.507 DEBUG:boto:path=/ppfahn-63/obj1 2024-03-04T18:35:05.507 DEBUG:boto:auth_path=/ppfahn-63/obj1 2024-03-04T18:35:05.507 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.507 DEBUG:boto:Path: /ppfahn-63/obj1 2024-03-04T18:35:05.507 DEBUG:boto:Data: 2024-03-04T18:35:05.508 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:35:05.508 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:35:05.508 DEBUG:boto:Port: 8001 2024-03-04T18:35:05.508 DEBUG:boto:Params: {} 2024-03-04T18:35:05.508 DEBUG:boto:Token: None 2024-03-04T18:35:05.508 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-63/obj1 2024-03-04T18:35:05.508 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:tmesgIAPlTt1mjVvhh1B2yK/HBU= 2024-03-04T18:35:05.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tmesgIAPlTt1mjVvhh1B2yK/HBU='} 2024-03-04T18:35:05.525 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f5cbc958550adc6-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.525 DEBUG:boto:path=/ppfahn-63/ 2024-03-04T18:35:05.525 DEBUG:boto:auth_path=/ppfahn-63/ 2024-03-04T18:35:05.525 DEBUG:boto:Method: HEAD 2024-03-04T18:35:05.525 DEBUG:boto:Path: /ppfahn-63/ 2024-03-04T18:35:05.525 DEBUG:boto:Data: 2024-03-04T18:35:05.525 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.526 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:35:05.526 DEBUG:boto:Port: 8001 2024-03-04T18:35:05.526 DEBUG:boto:Params: {} 2024-03-04T18:35:05.526 DEBUG:boto:Token: None 2024-03-04T18:35:05.526 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-63/ 2024-03-04T18:35:05.526 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I= 2024-03-04T18:35:05.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I=', 'Content-Length': '0'} 2024-03-04T18:35:05.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7222a2aab779e97-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.532 DEBUG:boto:path=/ppfahn-63/obj2 2024-03-04T18:35:05.532 DEBUG:boto:auth_path=/ppfahn-63/obj2 2024-03-04T18:35:05.532 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.532 DEBUG:boto:Path: /ppfahn-63/obj2 2024-03-04T18:35:05.532 DEBUG:boto:Data: 2024-03-04T18:35:05.532 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:35:05.532 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:35:05.532 DEBUG:boto:Port: 8001 2024-03-04T18:35:05.532 DEBUG:boto:Params: {} 2024-03-04T18:35:05.533 DEBUG:boto:Token: None 2024-03-04T18:35:05.533 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-63/obj2 2024-03-04T18:35:05.533 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:7oUzQ4UXMTaBlQD7f6uhyBs5eC0= 2024-03-04T18:35:05.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7oUzQ4UXMTaBlQD7f6uhyBs5eC0='} 2024-03-04T18:35:05.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6a426bbcb482ee0-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.576 DEBUG:boto:path=/ppfahn-63/ 2024-03-04T18:35:05.576 DEBUG:boto:auth_path=/ppfahn-63/ 2024-03-04T18:35:05.576 DEBUG:boto:Method: HEAD 2024-03-04T18:35:05.577 DEBUG:boto:Path: /ppfahn-63/ 2024-03-04T18:35:05.577 DEBUG:boto:Data: 2024-03-04T18:35:05.577 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.577 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:35:05.577 DEBUG:boto:Port: 8001 2024-03-04T18:35:05.577 DEBUG:boto:Params: {} 2024-03-04T18:35:05.577 DEBUG:boto:Token: None 2024-03-04T18:35:05.577 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-63/ 2024-03-04T18:35:05.577 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I= 2024-03-04T18:35:05.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I=', 'Content-Length': '0'} 2024-03-04T18:35:05.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000096cb4bc59e8bda8-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.582 DEBUG:boto:path=/ppfahn-63/obj3 2024-03-04T18:35:05.582 DEBUG:boto:auth_path=/ppfahn-63/obj3 2024-03-04T18:35:05.582 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.582 DEBUG:boto:Path: /ppfahn-63/obj3 2024-03-04T18:35:05.582 DEBUG:boto:Data: 2024-03-04T18:35:05.582 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:35:05.582 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:35:05.582 DEBUG:boto:Port: 8001 2024-03-04T18:35:05.582 DEBUG:boto:Params: {} 2024-03-04T18:35:05.583 DEBUG:boto:Token: None 2024-03-04T18:35:05.583 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-63/obj3 2024-03-04T18:35:05.583 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:3PCJjuh3VNHJLTKXdRue6aWzVaw= 2024-03-04T18:35:05.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3PCJjuh3VNHJLTKXdRue6aWzVaw='} 2024-03-04T18:35:05.597 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000ea510c2b34d40e-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.597 DEBUG:boto:path=/ppfahn-63/ 2024-03-04T18:35:05.598 DEBUG:boto:auth_path=/ppfahn-63/ 2024-03-04T18:35:05.598 DEBUG:boto:Method: HEAD 2024-03-04T18:35:05.598 DEBUG:boto:Path: /ppfahn-63/ 2024-03-04T18:35:05.598 DEBUG:boto:Data: 2024-03-04T18:35:05.598 DEBUG:boto:Headers: {} 2024-03-04T18:35:05.598 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:35:05.598 DEBUG:boto:Port: 8001 2024-03-04T18:35:05.598 DEBUG:boto:Params: {} 2024-03-04T18:35:05.598 DEBUG:boto:Token: None 2024-03-04T18:35:05.598 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-63/ 2024-03-04T18:35:05.599 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I= 2024-03-04T18:35:05.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I=', 'Content-Length': '0'} 2024-03-04T18:35:05.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b513b4c3ed355395-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.603 DEBUG:boto:path=/ppfahn-63/obj4 2024-03-04T18:35:05.603 DEBUG:boto:auth_path=/ppfahn-63/obj4 2024-03-04T18:35:05.603 DEBUG:boto:Method: PUT 2024-03-04T18:35:05.603 DEBUG:boto:Path: /ppfahn-63/obj4 2024-03-04T18:35:05.603 DEBUG:boto:Data: 2024-03-04T18:35:05.603 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T18:35:05.604 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:35:05.604 DEBUG:boto:Port: 8001 2024-03-04T18:35:05.604 DEBUG:boto:Params: {} 2024-03-04T18:35:05.604 DEBUG:boto:Token: None 2024-03-04T18:35:05.604 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 18:35:05 GMT /ppfahn-63/obj4 2024-03-04T18:35:05.604 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:q0hzhMo+JlZYrTnQuKwpyzeHQIs= 2024-03-04T18:35:05.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:q0hzhMo+JlZYrTnQuKwpyzeHQIs='} 2024-03-04T18:35:05.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ab0e1111a4f1bb7-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:35:05.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:05.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:05.649 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:05.865 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:05.865 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:05.952 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T18:35:05.980 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709577305.271001_560.1","last_update":"2024-03-04T18:35:05.271001Z"}] 2024-03-04T18:35:05.981 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:35:05.981 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:35:05.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:05.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:05.982 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:06.024 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:06.024 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:06.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:06.096+0000 7f187e28f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:06.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:06.096+0000 7f187e28f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:06.124 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:06Z"} 2024-03-04T18:35:06.124 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:06Z"} 2024-03-04T18:35:06.124 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:35:06.124 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:35:06.124 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:35:06.125 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:35:06.588 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:35:06.586+0000 7f5c2310a640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T18:35:11.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:11.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:11.127 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:11.169 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:11.170 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:11.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:11.228+0000 7f19dfd49980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:11.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:11.232+0000 7f19dfd49980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:11.263 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:11Z"} 2024-03-04T18:35:11.263 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:11Z"} 2024-03-04T18:35:11.263 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:35:11.263 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:35:11.264 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:35:11.264 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:35:14.129 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:35:14.126+0000 7f5c190f6640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:35:14.130 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:35:14.126+0000 7f5c2390b640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T18:35:16.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:16.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:16.265 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:16.308 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:16.308 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:16.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:16.383+0000 7f7c1bca2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:16.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:16.383+0000 7f7c1bca2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:16.427 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:16Z"} 2024-03-04T18:35:16.427 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:16Z"} 2024-03-04T18:35:16.427 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:35:16.427 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:35:16.427 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:35:16.428 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:35:21.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:21.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:21.429 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:21.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:21.531+0000 7f77e6944980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:21.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:21.531+0000 7f77e6944980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:21.562 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:21Z"} 2024-03-04T18:35:21.563 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:21Z"} 2024-03-04T18:35:21.563 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:35:21.563 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:35:21.563 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:35:21.563 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:35:26.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:26.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:26.565 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:26.609 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:26.609 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:26.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:26.683+0000 7f4544104980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:26.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:26.683+0000 7f4544104980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:26.713 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:26Z"} 2024-03-04T18:35:26.713 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:26Z"} 2024-03-04T18:35:26.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.315397_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.315397Z', 'realm_epoch': 2}}] 2024-03-04T18:35:26.713 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:35:26.713 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.315397_306.1'} 2024-03-04T18:35:26.713 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.315397_306.1 2024-03-04T18:35:31.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:31.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:31.714 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:31.756 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:31.756 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:31.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:31.823+0000 7fa0d9648980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:31.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:31.827+0000 7fa0d9648980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:31.854 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:31Z"} 2024-03-04T18:35:31.855 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:31Z"} 2024-03-04T18:35:31.855 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:35:31.855 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:35:31.855 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:35:31.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:35:36.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:36.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:36.858 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:36.900 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:36.900 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:36.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:36.967+0000 7fda06434980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:36.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:36.967+0000 7fda06434980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:37.004 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.815737_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.815737Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:36Z"} 2024-03-04T18:35:37.005 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.815737_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.815737Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:36Z"} 2024-03-04T18:35:37.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.815737_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.815737Z', 'realm_epoch': 2}}] 2024-03-04T18:35:37.005 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:35:37.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.815737_334.1'} 2024-03-04T18:35:37.006 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575077.815737_334.1 2024-03-04T18:35:42.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:42.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:42.007 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:42.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:42.111+0000 7f6654249980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:42.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:42.111+0000 7f6654249980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:42.140 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:42Z"} 2024-03-04T18:35:42.140 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:42Z"} 2024-03-04T18:35:42.141 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:35:42.141 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:35:42.141 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:35:42.141 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:35:47.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:47.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:47.142 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:47.184 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:47.184 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:47.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:47.259+0000 7f8a561dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:47.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:47.259+0000 7f8a561dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:47.297 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:47Z"} 2024-03-04T18:35:47.297 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:47Z"} 2024-03-04T18:35:47.297 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:35:47.297 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:35:47.298 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:35:47.298 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:35:52.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:52.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:52.299 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:52.343 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:52.344 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:52.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:52.415+0000 7f516f530980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:52.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:52.419+0000 7f516f530980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:52.451 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:52Z"} 2024-03-04T18:35:52.451 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:52Z"} 2024-03-04T18:35:52.451 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:35:52.451 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:35:52.451 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:35:52.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:35:57.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:57.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:57.453 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:57.555+0000 7f53d78ad980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:35:57.555+0000 7f53d78ad980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:57.585 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:57Z"} 2024-03-04T18:35:57.585 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:57Z"} 2024-03-04T18:35:57.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:35:57.586 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:35:57.586 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:35:57.586 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:36:02.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:02.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:02.587 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:02.630 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:02.630 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:02.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:02.695+0000 7fd318a54980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:02.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:02.695+0000 7fd318a54980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:02.720 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:02Z"} 2024-03-04T18:36:02.720 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:02Z"} 2024-03-04T18:36:02.721 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:36:02.721 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:36:02.721 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:36:02.721 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:36:07.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:07.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:07.723 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:07.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:07.831+0000 7f376fe99980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:07.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:07.831+0000 7f376fe99980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:07.857 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:07Z"} 2024-03-04T18:36:07.858 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:07Z"} 2024-03-04T18:36:07.858 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:36:07.858 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:36:07.858 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:36:07.858 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:36:12.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:12.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:12.860 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:12.903 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:12.903 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:12.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:12.967+0000 7f83aaabd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:12.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:12.967+0000 7f83aaabd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:12.997 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:12Z"} 2024-03-04T18:36:12.997 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:12Z"} 2024-03-04T18:36:12.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.523626_319.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.523626Z', 'realm_epoch': 2}}] 2024-03-04T18:36:12.998 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:36:12.998 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.523626_319.1'} 2024-03-04T18:36:12.998 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.523626_319.1 2024-03-04T18:36:17.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:17.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:18.000 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:18.043 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:18.043 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:18.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:18.111+0000 7f0c9674b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:18.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:18.111+0000 7f0c9674b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:18.139 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:18Z"} 2024-03-04T18:36:18.139 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:18Z"} 2024-03-04T18:36:18.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:36:18.140 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:36:18.140 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:36:18.140 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:36:23.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:23.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:23.141 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:23.185 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:23.185 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:23.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:23.254+0000 7fa4e1455980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:23.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:23.254+0000 7fa4e1455980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:23.280 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.489493_317.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.489493Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:23Z"} 2024-03-04T18:36:23.280 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.489493_317.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.489493Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:23Z"} 2024-03-04T18:36:23.280 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.489493_317.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.489493Z', 'realm_epoch': 2}}] 2024-03-04T18:36:23.281 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:36:23.281 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.489493_317.1'} 2024-03-04T18:36:23.281 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.489493_317.1 2024-03-04T18:36:26.314 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:36:26.312+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:36:28.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:28.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:28.283 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:28.326 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:28.326 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:28.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:28.390+0000 7ff47019b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:28.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:28.390+0000 7ff47019b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:28.420 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:28Z"} 2024-03-04T18:36:28.420 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:28Z"} 2024-03-04T18:36:28.420 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:36:28.421 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:36:28.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:36:28.421 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:36:33.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:33.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:33.422 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:33.465 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:33.465 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:33.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:33.542+0000 7f79d6c24980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:33.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:33.542+0000 7f79d6c24980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:33.572 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:33Z"} 2024-03-04T18:36:33.573 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:33Z"} 2024-03-04T18:36:33.573 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:36:33.573 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:36:33.573 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:36:33.573 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:36:38.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:38.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:38.575 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:38.627 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:38.627 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:38.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:38.698+0000 7f62fb4ae980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:38.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:38.698+0000 7f62fb4ae980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:38.726 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:38Z"} 2024-03-04T18:36:38.726 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:38Z"} 2024-03-04T18:36:38.726 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:36:38.726 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:36:38.726 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:36:38.726 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:36:43.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:43.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:43.729 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:43.773 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:43.773 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:43.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:43.850+0000 7f68b4574980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:43.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:43.850+0000 7f68b4574980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:43.883 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:43Z"} 2024-03-04T18:36:43.883 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:43Z"} 2024-03-04T18:36:43.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:36:43.884 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:36:43.884 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:36:43.884 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:36:48.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:48.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:48.886 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:48.926 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:48.926 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:48.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:48.994+0000 7fdd546da980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:48.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:48.998+0000 7fdd546da980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:49.023 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.486968_316.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.486968Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:49Z"} 2024-03-04T18:36:49.023 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.486968_316.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.486968Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:49Z"} 2024-03-04T18:36:49.023 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.486968_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.486968Z', 'realm_epoch': 2}}] 2024-03-04T18:36:49.023 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:36:49.024 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.486968_316.1'} 2024-03-04T18:36:49.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.486968_316.1 2024-03-04T18:36:54.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:54.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:54.026 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:54.069 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:54.069 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:54.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:54.138+0000 7fc8493fd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:54.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:54.138+0000 7fc8493fd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:54.170 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:54Z"} 2024-03-04T18:36:54.170 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:54Z"} 2024-03-04T18:36:54.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:36:54.171 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:36:54.171 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:36:54.171 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:36:59.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:59.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:59.172 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:59.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:59.282+0000 7f469afbf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:59.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:36:59.282+0000 7f469afbf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:59.313 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:59Z"} 2024-03-04T18:36:59.313 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:59Z"} 2024-03-04T18:36:59.313 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:36:59.314 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:36:59.314 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:36:59.314 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:37:04.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:04.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:04.315 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:04.361 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:04.361 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:04.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:04.434+0000 7fb54f8d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:04.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:04.438+0000 7fb54f8d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:04.468 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:04Z"} 2024-03-04T18:37:04.468 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:04Z"} 2024-03-04T18:37:04.468 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:37:04.468 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:37:04.469 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:37:04.469 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:37:09.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:09.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:09.471 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:09.513 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:09.513 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:09.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:09.590+0000 7f1d639c1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:09.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:09.590+0000 7f1d639c1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:09.637 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:09Z"} 2024-03-04T18:37:09.637 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:09Z"} 2024-03-04T18:37:09.637 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:37:09.638 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:37:09.638 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:37:09.638 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:37:14.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:14.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:14.639 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:14.681 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:14.682 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:14.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:14.742+0000 7efd90a35980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:14.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:14.742+0000 7efd90a35980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:14.767 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:14Z"} 2024-03-04T18:37:14.768 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:14Z"} 2024-03-04T18:37:14.768 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:37:14.768 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:37:14.768 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:37:14.768 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:37:19.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:19.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:19.769 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:19.811 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:19.812 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:19.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:19.893+0000 7f439007d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:19.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:19.893+0000 7f439007d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:19.921 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:19Z"} 2024-03-04T18:37:19.922 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:19Z"} 2024-03-04T18:37:19.922 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:37:19.922 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:37:19.922 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:37:19.922 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:37:24.166 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:37:24.163+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:37:24.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:24.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:24.923 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:25.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:25.033+0000 7fe73934e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:25.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:25.037+0000 7fe73934e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:25.064 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:25Z"} 2024-03-04T18:37:25.065 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:25Z"} 2024-03-04T18:37:25.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:37:25.065 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:37:25.065 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:37:25.065 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:37:30.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:30.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:30.066 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:30.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:30.169+0000 7f7f8cfcb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:30.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:30.169+0000 7f7f8cfcb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:30.198 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:30Z"} 2024-03-04T18:37:30.198 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:30Z"} 2024-03-04T18:37:30.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:37:30.199 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:37:30.199 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:37:30.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:37:35.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:35.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:35.201 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:35.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:35.305+0000 7f9722560980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:35.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:35.305+0000 7f9722560980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:35.333 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:35Z"} 2024-03-04T18:37:35.333 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:35Z"} 2024-03-04T18:37:35.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:37:35.333 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:37:35.334 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:37:35.334 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:37:40.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:40.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:40.335 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:40.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:40.441+0000 7f44a8266980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:40.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:40.441+0000 7f44a8266980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:40.470 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:40Z"} 2024-03-04T18:37:40.470 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:40Z"} 2024-03-04T18:37:40.470 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:37:40.470 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:37:40.470 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:37:40.470 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:37:45.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:45.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:45.472 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:45.515 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:45.515 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:45.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:45.589+0000 7f098bbf2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:45.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:45.593+0000 7f098bbf2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:45.622 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:45Z"} 2024-03-04T18:37:45.622 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:45Z"} 2024-03-04T18:37:45.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:37:45.623 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:37:45.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:37:45.623 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:37:50.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:50.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:50.625 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:50.668 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:50.668 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:50.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:50.749+0000 7ffad88e0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:50.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:50.749+0000 7ffad88e0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:50.781 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:50Z"} 2024-03-04T18:37:50.781 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:50Z"} 2024-03-04T18:37:50.782 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:37:50.782 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:37:50.782 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:37:50.782 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:37:55.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:55.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:55.784 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:55.827 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:55.827 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:55.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:55.905+0000 7f9383542980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:55.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:37:55.909+0000 7f9383542980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:55.944 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:55Z"} 2024-03-04T18:37:55.944 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:55Z"} 2024-03-04T18:37:55.944 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:37:55.945 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:37:55.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:37:55.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:38:00.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:00.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:00.946 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:01.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:01.053+0000 7fbb3ea18980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:01.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:01.057+0000 7fbb3ea18980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:01.085 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:01Z"} 2024-03-04T18:38:01.085 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:01Z"} 2024-03-04T18:38:01.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:38:01.086 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:38:01.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:38:01.086 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:38:06.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:06.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:06.089 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:06.131 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:06.131 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:06.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:06.201+0000 7f12d57d5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:06.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:06.205+0000 7f12d57d5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:06.230 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:06Z"} 2024-03-04T18:38:06.230 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:06Z"} 2024-03-04T18:38:06.231 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:38:06.231 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:38:06.231 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:38:06.231 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:38:11.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:11.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:11.233 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:11.293 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:11.293 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:11.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:11.369+0000 7f1d9b30c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:11.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:11.373+0000 7f1d9b30c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:11.411 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:11Z"} 2024-03-04T18:38:11.411 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:11Z"} 2024-03-04T18:38:11.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:38:11.411 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:38:11.412 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:38:11.412 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:38:16.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:16.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:16.413 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:16.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:16.524+0000 7f2a58587980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:16.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:16.524+0000 7f2a58587980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:16.554 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:16Z"} 2024-03-04T18:38:16.554 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:16Z"} 2024-03-04T18:38:16.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:38:16.555 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:38:16.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:38:16.555 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:38:21.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:21.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:21.557 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:21.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:21.680+0000 7f003839c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:21.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:21.680+0000 7f003839c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:21.729 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:21Z"} 2024-03-04T18:38:21.729 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:21Z"} 2024-03-04T18:38:21.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:38:21.729 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:38:21.729 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:38:21.729 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:38:26.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:26.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:26.731 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:26.774 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:26.775 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:26.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:26.848+0000 7ff311305980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:26.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:26.852+0000 7ff311305980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:26.883 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:26Z"} 2024-03-04T18:38:26.884 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:26Z"} 2024-03-04T18:38:26.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:38:26.884 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:38:26.884 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:38:26.884 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:38:31.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:31.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:31.886 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:31.929 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:31.929 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:32.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:32.056+0000 7f1b49276980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:32.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:32.060+0000 7f1b49276980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:32.085 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:32Z"} 2024-03-04T18:38:32.086 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:32Z"} 2024-03-04T18:38:32.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:38:32.086 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:38:32.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:38:32.087 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:38:37.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:37.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:37.088 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:37.132 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:37.132 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:37.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:37.200+0000 7f9c8b1dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:37.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:37.204+0000 7f9c8b1dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:37.234 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:37Z"} 2024-03-04T18:38:37.234 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:37Z"} 2024-03-04T18:38:37.234 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:38:37.235 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:38:37.235 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:38:37.235 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:38:42.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:42.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:42.236 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:42.273 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:42.274 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:42.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:42.356+0000 7f4e4e0b2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:42.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:42.356+0000 7f4e4e0b2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:42.391 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:42Z"} 2024-03-04T18:38:42.391 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:42Z"} 2024-03-04T18:38:42.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575076.582500_321.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:56.582500Z', 'realm_epoch': 2}}] 2024-03-04T18:38:42.392 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:38:42.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575076.582500_321.1'} 2024-03-04T18:38:42.392 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575076.582500_321.1 2024-03-04T18:38:47.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:47.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:47.393 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:47.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:47.508+0000 7fb3c4ad9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:47.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:47.508+0000 7fb3c4ad9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:47.537 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:47Z"} 2024-03-04T18:38:47.538 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:47Z"} 2024-03-04T18:38:47.538 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:38:47.538 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:38:47.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:38:47.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:38:52.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:52.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:52.539 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:52.581 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:52.581 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:52.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:52.640+0000 7f26466c7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:52.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:52.640+0000 7f26466c7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:52.668 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:52Z"} 2024-03-04T18:38:52.669 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:52Z"} 2024-03-04T18:38:52.669 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.388355_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.388355Z', 'realm_epoch': 2}}] 2024-03-04T18:38:52.669 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:38:52.669 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.388355_309.1'} 2024-03-04T18:38:52.669 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.388355_309.1 2024-03-04T18:38:57.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:57.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:57.672 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:57.715 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:57.715 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:57.788+0000 7f9f496e7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:38:57.788+0000 7f9f496e7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:57.818 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:57Z"} 2024-03-04T18:38:57.818 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:57Z"} 2024-03-04T18:38:57.819 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:38:57.819 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:38:57.819 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:38:57.819 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:39:02.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:02.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:02.821 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:02.864 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:02.864 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:02.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:02.932+0000 7fda8db68980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:02.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:02.936+0000 7fda8db68980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:02.965 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:02Z"} 2024-03-04T18:39:02.965 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:02Z"} 2024-03-04T18:39:02.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:39:02.965 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:39:02.965 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:39:02.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:39:07.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:07.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:07.967 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:08.009 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:08.009 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:08.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:08.068+0000 7fe5f5b57980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:08.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:08.068+0000 7fe5f5b57980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:08.098 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:08Z"} 2024-03-04T18:39:08.099 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:08Z"} 2024-03-04T18:39:08.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:39:08.099 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:39:08.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:39:08.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:39:13.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:13.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:13.101 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:13.143 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:13.143 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:13.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:13.199+0000 7f55ad198980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:13.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:13.199+0000 7f55ad198980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:13.233 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:13Z"} 2024-03-04T18:39:13.234 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:13Z"} 2024-03-04T18:39:13.234 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:39:13.234 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:39:13.234 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:39:13.234 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:39:18.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:18.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:18.236 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:18.296 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:18.296 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:18.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:18.391+0000 7fb89c867980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:18.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:18.391+0000 7fb89c867980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:18.422 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:18Z"} 2024-03-04T18:39:18.422 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:18Z"} 2024-03-04T18:39:18.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:39:18.423 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:39:18.423 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:39:18.423 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:39:23.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:23.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:23.425 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:23.468 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:23.468 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:23.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:23.527+0000 7f7095bf8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:23.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:23.527+0000 7f7095bf8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:23.557 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:23Z"} 2024-03-04T18:39:23.558 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:23Z"} 2024-03-04T18:39:23.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:39:23.558 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:39:23.558 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:39:23.558 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:39:28.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:28.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:28.560 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:28.603 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:28.603 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:28.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:28.663+0000 7f0b7d0de980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:28.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:28.663+0000 7f0b7d0de980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:28.699 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:28Z"} 2024-03-04T18:39:28.699 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:28Z"} 2024-03-04T18:39:28.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:39:28.699 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:39:28.699 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:39:28.699 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:39:33.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:33.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:33.701 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:33.744 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:33.744 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:33.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:33.803+0000 7f7851513980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:33.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:33.803+0000 7f7851513980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:33.833 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:33Z"} 2024-03-04T18:39:33.834 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:33Z"} 2024-03-04T18:39:33.834 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:39:33.834 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:39:33.834 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:39:33.834 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:39:38.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:38.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:38.836 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:38.880 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:38.880 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:38.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:38.979+0000 7f3a1f1a3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:38.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:38.983+0000 7f3a1f1a3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:39.011 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:38Z"} 2024-03-04T18:39:39.011 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:38Z"} 2024-03-04T18:39:39.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:39:39.012 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:39:39.012 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:39:39.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:39:44.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:44.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:44.014 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:44.058 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:44.058 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:44.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:44.127+0000 7f8c5df09980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:44.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:44.131+0000 7f8c5df09980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:44.160 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:44Z"} 2024-03-04T18:39:44.160 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:44Z"} 2024-03-04T18:39:44.161 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:39:44.161 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:39:44.161 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:39:44.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:39:49.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:49.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:49.163 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:49.206 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:49.206 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:49.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:49.279+0000 7fb19ac68980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:49.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:49.279+0000 7fb19ac68980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:49.314 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.393343_310.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.393343Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:49Z"} 2024-03-04T18:39:49.315 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.393343_310.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.393343Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:49Z"} 2024-03-04T18:39:49.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.393343_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.393343Z', 'realm_epoch': 2}}] 2024-03-04T18:39:49.315 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:39:49.315 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.393343_310.1'} 2024-03-04T18:39:49.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.393343_310.1 2024-03-04T18:39:54.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:54.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:54.317 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:54.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:54.447+0000 7f36bc21e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:54.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:54.447+0000 7f36bc21e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:54.480 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:54Z"} 2024-03-04T18:39:54.480 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:54Z"} 2024-03-04T18:39:54.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:39:54.480 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:39:54.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:39:54.480 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:39:59.482 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:59.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:59.482 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:59.526 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:59.526 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:59.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:59.591+0000 7fb3aa45d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:59.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:39:59.595+0000 7fb3aa45d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:59.620 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:59Z"} 2024-03-04T18:39:59.620 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:59Z"} 2024-03-04T18:39:59.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:39:59.621 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:39:59.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:39:59.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:40:04.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:04.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:04.622 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:04.665 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:04.665 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:04.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:04.746+0000 7fcfe65dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:04.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:04.746+0000 7fcfe65dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:04.781 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:04Z"} 2024-03-04T18:40:04.781 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:04Z"} 2024-03-04T18:40:04.781 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:40:04.781 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:40:04.782 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:40:04.782 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T18:40:09.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:09.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:09.783 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:09.827 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:09.827 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:09.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:09.902+0000 7f831af5f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:09.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:09.902+0000 7f831af5f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:09.932 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:09Z"} 2024-03-04T18:40:09.932 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:09Z"} 2024-03-04T18:40:09.932 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:40:09.932 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T18:40:09.933 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:40:09.933 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T18:40:14.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2024-03-04T18:40:14.936 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-64 2024-03-04T18:40:14.936 DEBUG:boto:path=/ppfahn-64/ 2024-03-04T18:40:14.937 DEBUG:boto:auth_path=/ppfahn-64/ 2024-03-04T18:40:14.937 DEBUG:boto:Method: PUT 2024-03-04T18:40:14.937 DEBUG:boto:Path: /ppfahn-64/ 2024-03-04T18:40:14.937 DEBUG:boto:Data: 2024-03-04T18:40:14.937 DEBUG:boto:Headers: {} 2024-03-04T18:40:14.937 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:40:14.937 DEBUG:boto:Port: 8000 2024-03-04T18:40:14.937 DEBUG:boto:Params: {} 2024-03-04T18:40:14.938 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:40:14.938 DEBUG:boto:Token: None 2024-03-04T18:40:14.938 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:40:14 GMT /ppfahn-64/ 2024-03-04T18:40:14.938 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Ms8x8N00fAzKrfDrajNRldFAfSQ= 2024-03-04T18:40:14.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:40:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Ms8x8N00fAzKrfDrajNRldFAfSQ=', 'Content-Length': '0'} 2024-03-04T18:40:14.988 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004b3193439ab16fbc-0065e6158e-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:40:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:40:14.988 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ppfahn-65 2024-03-04T18:40:14.988 DEBUG:boto:path=/ppfahn-65/ 2024-03-04T18:40:14.988 DEBUG:boto:auth_path=/ppfahn-65/ 2024-03-04T18:40:14.988 DEBUG:boto:Method: PUT 2024-03-04T18:40:14.988 DEBUG:boto:Path: /ppfahn-65/ 2024-03-04T18:40:14.988 DEBUG:boto:Data: 2024-03-04T18:40:14.988 DEBUG:boto:Headers: {} 2024-03-04T18:40:14.988 DEBUG:boto:Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T18:40:14.989 DEBUG:boto:Port: 8000 2024-03-04T18:40:14.989 DEBUG:boto:Params: {} 2024-03-04T18:40:14.989 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:40:14.989 DEBUG:boto:Token: None 2024-03-04T18:40:14.989 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:40:14 GMT /ppfahn-65/ 2024-03-04T18:40:14.989 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ldi8I+ILnrtT98YyNek8Rhcmeyw= 2024-03-04T18:40:14.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:40:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ldi8I+ILnrtT98YyNek8Rhcmeyw=', 'Content-Length': '0'} 2024-03-04T18:40:15.074 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f660999ea68c93d1-0065e6158e-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:40:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:40:15.074 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ppfahn-66 2024-03-04T18:40:15.074 DEBUG:boto:path=/ppfahn-66/ 2024-03-04T18:40:15.074 DEBUG:boto:auth_path=/ppfahn-66/ 2024-03-04T18:40:15.074 DEBUG:boto:Method: PUT 2024-03-04T18:40:15.074 DEBUG:boto:Path: /ppfahn-66/ 2024-03-04T18:40:15.074 DEBUG:boto:Data: 2024-03-04T18:40:15.075 DEBUG:boto:Headers: {} 2024-03-04T18:40:15.075 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T18:40:15.075 DEBUG:boto:Port: 8001 2024-03-04T18:40:15.075 DEBUG:boto:Params: {} 2024-03-04T18:40:15.075 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T18:40:15.075 DEBUG:boto:Token: None 2024-03-04T18:40:15.075 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:40:15 GMT /ppfahn-66/ 2024-03-04T18:40:15.075 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eWLsa8IjL79wy0xwd8B26FT6NJ8= 2024-03-04T18:40:15.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:40:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eWLsa8IjL79wy0xwd8B26FT6NJ8=', 'Content-Length': '0'} 2024-03-04T18:40:15.272 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f79a72231b10b73a-0065e6158f-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:40:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:40:15.272 DEBUG:boto:path=/ppfahn-64/MULTIPART 2024-03-04T18:40:15.272 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART 2024-03-04T18:40:15.273 DEBUG:boto:path=/ppfahn-64/MULTIPART?uploads 2024-03-04T18:40:15.273 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART?uploads 2024-03-04T18:40:15.273 DEBUG:boto:Method: POST 2024-03-04T18:40:15.273 DEBUG:boto:Path: /ppfahn-64/MULTIPART?uploads 2024-03-04T18:40:15.273 DEBUG:boto:Data: 2024-03-04T18:40:15.273 DEBUG:boto:Headers: {} 2024-03-04T18:40:15.273 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:40:15.273 DEBUG:boto:Port: 8000 2024-03-04T18:40:15.273 DEBUG:boto:Params: {} 2024-03-04T18:40:15.273 DEBUG:boto:Token: None 2024-03-04T18:40:15.274 DEBUG:boto:StringToSign: POST Mon, 04 Mar 2024 18:40:15 GMT /ppfahn-64/MULTIPART?uploads 2024-03-04T18:40:15.274 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:zmjPkLaBWjRzg9Um/qfMgk3TXig= 2024-03-04T18:40:15.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:40:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zmjPkLaBWjRzg9Um/qfMgk3TXig=', 'Content-Length': '0'} 2024-03-04T18:40:17.331 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000040a980df6df2a661-0065e6158f-5376-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '249'), ('Date', 'Mon, 04 Mar 2024 18:40:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:40:17.331 DEBUG:boto:b'ppfahn-64MULTIPART2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb' 2024-03-04T18:40:17.375 DEBUG:boto:path=/ppfahn-64/MULTIPART 2024-03-04T18:40:17.375 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART 2024-03-04T18:40:17.375 DEBUG:boto:path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=1 2024-03-04T18:40:17.375 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=1 2024-03-04T18:40:17.375 DEBUG:boto:Method: PUT 2024-03-04T18:40:17.375 DEBUG:boto:Path: /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=1 2024-03-04T18:40:17.375 DEBUG:boto:Data: 2024-03-04T18:40:17.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T18:40:17.376 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:40:17.376 DEBUG:boto:Port: 8000 2024-03-04T18:40:17.376 DEBUG:boto:Params: {} 2024-03-04T18:40:17.376 DEBUG:boto:Token: None 2024-03-04T18:40:17.376 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Mon, 04 Mar 2024 18:40:17 GMT /ppfahn-64/MULTIPART?partNumber=1&uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T18:40:17.376 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:2MDtzWDPf9tYwmpH14vxOh/vYUI= 2024-03-04T18:40:17.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:40:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2MDtzWDPf9tYwmpH14vxOh/vYUI='} 2024-03-04T18:40:17.507 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0d22a7e28828b93-0065e61591-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:40:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:40:17.540 DEBUG:boto:path=/ppfahn-64/MULTIPART 2024-03-04T18:40:17.540 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART 2024-03-04T18:40:17.540 DEBUG:boto:path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=2 2024-03-04T18:40:17.540 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=2 2024-03-04T18:40:17.540 DEBUG:boto:Method: PUT 2024-03-04T18:40:17.540 DEBUG:boto:Path: /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=2 2024-03-04T18:40:17.541 DEBUG:boto:Data: 2024-03-04T18:40:17.541 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T18:40:17.541 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:40:17.541 DEBUG:boto:Port: 8000 2024-03-04T18:40:17.541 DEBUG:boto:Params: {} 2024-03-04T18:40:17.541 DEBUG:boto:Token: None 2024-03-04T18:40:17.541 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Mon, 04 Mar 2024 18:40:17 GMT /ppfahn-64/MULTIPART?partNumber=2&uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T18:40:17.541 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:TxG+MiKSesPFiucNo2NKPH0la0A= 2024-03-04T18:40:17.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:40:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TxG+MiKSesPFiucNo2NKPH0la0A='} 2024-03-04T18:40:17.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077353b7b32d526b8-0065e61591-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:40:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:40:17.669 DEBUG:boto:path=/ppfahn-64/MULTIPART 2024-03-04T18:40:17.669 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART 2024-03-04T18:40:17.669 DEBUG:boto:path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=3 2024-03-04T18:40:17.669 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=3 2024-03-04T18:40:17.670 DEBUG:boto:Method: PUT 2024-03-04T18:40:17.670 DEBUG:boto:Path: /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=3 2024-03-04T18:40:17.670 DEBUG:boto:Data: 2024-03-04T18:40:17.670 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T18:40:17.670 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:40:17.670 DEBUG:boto:Port: 8000 2024-03-04T18:40:17.670 DEBUG:boto:Params: {} 2024-03-04T18:40:17.670 DEBUG:boto:Token: None 2024-03-04T18:40:17.670 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Mon, 04 Mar 2024 18:40:17 GMT /ppfahn-64/MULTIPART?partNumber=3&uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T18:40:17.671 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:XBXsXck1g9Aba5AX2524jmGuwZk= 2024-03-04T18:40:17.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:40:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XBXsXck1g9Aba5AX2524jmGuwZk='} 2024-03-04T18:40:17.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f771588707147bf8-0065e61591-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:40:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:40:17.866 DEBUG:boto:path=/ppfahn-64/MULTIPART 2024-03-04T18:40:17.866 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART 2024-03-04T18:40:17.866 DEBUG:boto:path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=4 2024-03-04T18:40:17.866 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=4 2024-03-04T18:40:17.866 DEBUG:boto:Method: PUT 2024-03-04T18:40:17.866 DEBUG:boto:Path: /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=4 2024-03-04T18:40:17.866 DEBUG:boto:Data: 2024-03-04T18:40:17.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T18:40:17.867 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:40:17.867 DEBUG:boto:Port: 8000 2024-03-04T18:40:17.867 DEBUG:boto:Params: {} 2024-03-04T18:40:17.867 DEBUG:boto:Token: None 2024-03-04T18:40:17.867 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Mon, 04 Mar 2024 18:40:17 GMT /ppfahn-64/MULTIPART?partNumber=4&uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T18:40:17.867 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gAskm4AqdIlcMf5TtHbUgsmLfQY= 2024-03-04T18:40:17.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:40:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gAskm4AqdIlcMf5TtHbUgsmLfQY='} 2024-03-04T18:40:17.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5b13278920ef77a-0065e61591-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:40:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:40:17.943 DEBUG:boto:path=/ppfahn-64/MULTIPART 2024-03-04T18:40:17.943 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART 2024-03-04T18:40:17.943 DEBUG:boto:path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T18:40:17.943 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T18:40:17.943 DEBUG:boto:Method: GET 2024-03-04T18:40:17.944 DEBUG:boto:Path: /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T18:40:17.944 DEBUG:boto:Data: 2024-03-04T18:40:17.944 DEBUG:boto:Headers: {} 2024-03-04T18:40:17.944 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:40:17.944 DEBUG:boto:Port: 8000 2024-03-04T18:40:17.944 DEBUG:boto:Params: {} 2024-03-04T18:40:17.944 DEBUG:boto:Token: None 2024-03-04T18:40:17.944 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 18:40:17 GMT /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T18:40:17.944 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:cJCjF6S/tZOQMehP7njNux535uM= 2024-03-04T18:40:17.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:40:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cJCjF6S/tZOQMehP7njNux535uM=', 'Content-Length': '0'} 2024-03-04T18:40:17.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070e8e105ce78d9cb-0065e61591-5376-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:40:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:40:17.988 DEBUG:boto:path=/ppfahn-64/MULTIPART 2024-03-04T18:40:17.989 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART 2024-03-04T18:40:17.989 DEBUG:boto:path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T18:40:17.989 DEBUG:boto:auth_path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T18:40:17.989 DEBUG:boto:Method: POST 2024-03-04T18:40:17.989 DEBUG:boto:Path: /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T18:40:17.989 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2024-03-04T18:40:17.989 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2024-03-04T18:40:17.989 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:40:17.989 DEBUG:boto:Port: 8000 2024-03-04T18:40:17.989 DEBUG:boto:Params: {} 2024-03-04T18:40:17.989 DEBUG:boto:Token: None 2024-03-04T18:40:17.990 DEBUG:boto:StringToSign: POST text/xml Mon, 04 Mar 2024 18:40:17 GMT /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T18:40:17.990 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:VbHiKdhZZGRkE1SjfyGv8wzBUbg= 2024-03-04T18:40:17.990 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:40:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VbHiKdhZZGRkE1SjfyGv8wzBUbg=', 'Content-Length': '460'} 2024-03-04T18:40:18.011 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008c920bbd5e3bb902-0065e61591-5376-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '325'), ('Date', 'Mon, 04 Mar 2024 18:40:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:40:18.012 DEBUG:boto:http://smithi136.front.sepia.ceph.com:8000/ppfahn-64/MULTIPARTppfahn-64MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-03-04T18:40:18.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:18.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:18.012 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:18.089 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:18.089 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:18.162 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T18:40:18.195 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709577615.212032_585.1","last_update":"2024-03-04T18:40:15.212032Z"}] 2024-03-04T18:40:18.195 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:40:18.195 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:40:18.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:18.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:18.196 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:18.238 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:18.238 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:18.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:18.318+0000 7f91f8f98980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:18.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:18.322+0000 7f91f8f98980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:18.355 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:18Z"} 2024-03-04T18:40:18.355 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:18Z"} 2024-03-04T18:40:18.356 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:40:18.356 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:40:18.356 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:40:18.356 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T18:40:23.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:23.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:23.357 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:23.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:23.474+0000 7fc1e1423980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:23.474+0000 7fc1e1423980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:23.511 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:23Z"} 2024-03-04T18:40:23.511 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:23Z"} 2024-03-04T18:40:23.511 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:40:23.511 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:40:23.511 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:40:23.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T18:40:28.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:28.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:28.512 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:28.556 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:28.556 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:28.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:28.622+0000 7f2e82167980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:28.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:28.622+0000 7f2e82167980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:28.658 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.830517_353.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.830517Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:28Z"} 2024-03-04T18:40:28.658 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.830517_353.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.830517Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:28Z"} 2024-03-04T18:40:28.658 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575385.830517_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:03:05.830517Z', 'realm_epoch': 2}}] 2024-03-04T18:40:28.658 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:40:28.658 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575385.830517_353.1'} 2024-03-04T18:40:28.658 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575385.830517_353.1 2024-03-04T18:40:33.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:33.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:33.659 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:33.702 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:33.702 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:33.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:33.762+0000 7fea734da980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:33.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:33.762+0000 7fea734da980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:33.789 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:33Z"} 2024-03-04T18:40:33.789 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:33Z"} 2024-03-04T18:40:33.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:40:33.789 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:40:33.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:40:33.789 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:40:34.260 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:40:34.258+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:40:34.261 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:40:34.258+0000 7f5c2d91f640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T18:40:38.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:38.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:38.791 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:38.832 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:38.833 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:38.898+0000 7fd1e85a1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:38.902+0000 7fd1e85a1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:38.934 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:38Z"} 2024-03-04T18:40:38.935 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:38Z"} 2024-03-04T18:40:38.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:40:38.935 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:40:38.935 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:40:38.935 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:40:43.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:43.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:43.937 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:43.979 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:43.979 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:44.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:44.054+0000 7fd35bb39980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:44.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:44.054+0000 7fd35bb39980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:44.085 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:44Z"} 2024-03-04T18:40:44.086 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:44Z"} 2024-03-04T18:40:44.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:40:44.086 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:40:44.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:40:44.086 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:40:49.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:49.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:49.088 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:49.130 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:49.130 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:49.210+0000 7f7228689980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:49.214+0000 7f7228689980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:49.244 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:49Z"} 2024-03-04T18:40:49.244 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:49Z"} 2024-03-04T18:40:49.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:40:49.245 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:40:49.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:40:49.245 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:40:54.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:54.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:54.246 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:54.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:54.370+0000 7f3f51338980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:54.370+0000 7f3f51338980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:54.405 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:54Z"} 2024-03-04T18:40:54.406 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:54Z"} 2024-03-04T18:40:54.406 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T18:40:54.406 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:40:54.406 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T18:40:54.406 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.189999_297.1 2024-03-04T18:40:59.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:59.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:59.407 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:59.449 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:59.450 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:59.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:59.521+0000 7f533b517980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:59.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:40:59.521+0000 7f533b517980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:59.557 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:59Z"} 2024-03-04T18:40:59.557 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:59Z"} 2024-03-04T18:40:59.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T18:40:59.558 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:40:59.558 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T18:40:59.558 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.380518_307.1 2024-03-04T18:41:04.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:04.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:04.559 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:04.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:04.681+0000 7f75b3bfe980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:04.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:04.681+0000 7f75b3bfe980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:04.712 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:04Z"} 2024-03-04T18:41:04.712 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:04Z"} 2024-03-04T18:41:04.712 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:41:04.713 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:41:04.713 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:41:04.713 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T18:41:09.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:09.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:09.714 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:09.758 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:09.758 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:09.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:09.837+0000 7f3b311e8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:09.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:09.841+0000 7f3b311e8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:09.874 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:09Z"} 2024-03-04T18:41:09.875 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:09Z"} 2024-03-04T18:41:09.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:41:09.875 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:41:09.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:41:09.875 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T18:41:14.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:14.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:14.877 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:14.919 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:14.919 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:14.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:14.985+0000 7fd128359980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:14.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:14.985+0000 7fd128359980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:15.012 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:14Z"} 2024-03-04T18:41:15.012 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:14Z"} 2024-03-04T18:41:15.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:41:15.012 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:41:15.012 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:41:15.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:41:20.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:20.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:20.014 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:20.057 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:20.057 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:20.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:20.137+0000 7f11686b8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:20.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:20.137+0000 7f11686b8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:20.173 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:20Z"} 2024-03-04T18:41:20.173 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:20Z"} 2024-03-04T18:41:20.173 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:41:20.174 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:41:20.174 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:41:20.174 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:41:25.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:25.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:25.175 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:25.218 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:25.219 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:25.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:25.289+0000 7f7b48c0a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:25.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:25.289+0000 7f7b48c0a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:25.317 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:25Z"} 2024-03-04T18:41:25.318 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:25Z"} 2024-03-04T18:41:25.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:41:25.318 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:41:25.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:41:25.318 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:41:26.354 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:41:26.349+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:41:30.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:30.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:30.319 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:30.366 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:30.366 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:30.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:30.441+0000 7f0311c0e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:30.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:30.441+0000 7f0311c0e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:30.471 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:30Z"} 2024-03-04T18:41:30.471 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:30Z"} 2024-03-04T18:41:30.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:41:30.472 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:41:30.472 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:41:30.472 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T18:41:35.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:35.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:35.474 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:35.517 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:35.517 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:35.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:35.577+0000 7f10aa5da980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:35.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:35.581+0000 7f10aa5da980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:35.612 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:35Z"} 2024-03-04T18:41:35.612 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:35Z"} 2024-03-04T18:41:35.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:41:35.612 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:41:35.613 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:41:35.613 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:41:40.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:40.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:40.615 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:40.657 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:40.657 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:40.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:40.729+0000 7f2456e52980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:40.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:40.729+0000 7f2456e52980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:40.756 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:40Z"} 2024-03-04T18:41:40.756 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:40Z"} 2024-03-04T18:41:40.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.315397_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.315397Z', 'realm_epoch': 2}}] 2024-03-04T18:41:40.757 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:41:40.757 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.315397_306.1'} 2024-03-04T18:41:40.757 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.315397_306.1 2024-03-04T18:41:45.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:45.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:45.759 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:45.802 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:45.802 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:45.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:45.869+0000 7f48c9fda980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:45.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:45.873+0000 7f48c9fda980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:45.899 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:45Z"} 2024-03-04T18:41:45.899 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:45Z"} 2024-03-04T18:41:45.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:41:45.900 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:41:45.900 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:41:45.900 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:41:50.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:50.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:50.901 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:50.944 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:50.945 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:51.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:51.005+0000 7f25923ef980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:51.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:51.009+0000 7f25923ef980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:51.035 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:51Z"} 2024-03-04T18:41:51.035 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:51Z"} 2024-03-04T18:41:51.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:41:51.036 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:41:51.036 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:41:51.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:41:56.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:56.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:56.037 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:56.077 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:56.078 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:56.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:56.136+0000 7f48203eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:56.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:41:56.140+0000 7f48203eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:56.169 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:56Z"} 2024-03-04T18:41:56.169 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:56Z"} 2024-03-04T18:41:56.169 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:41:56.170 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:41:56.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:41:56.170 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:42:01.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:01.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:01.172 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:01.215 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:01.215 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:01.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:01.284+0000 7f13a2f84980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:01.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:01.284+0000 7f13a2f84980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:01.322 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:01Z"} 2024-03-04T18:42:01.322 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:01Z"} 2024-03-04T18:42:01.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:42:01.322 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:42:01.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:42:01.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T18:42:06.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:06.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:06.324 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:06.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:06.436+0000 7f23dc261980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:06.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:06.436+0000 7f23dc261980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:06.468 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.735016_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.735016Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:06Z"} 2024-03-04T18:42:06.468 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.735016_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.735016Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:06Z"} 2024-03-04T18:42:06.468 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.735016_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.735016Z', 'realm_epoch': 2}}] 2024-03-04T18:42:06.468 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:42:06.468 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.735016_329.1'} 2024-03-04T18:42:06.468 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575077.735016_329.1 2024-03-04T18:42:11.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:11.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:11.470 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:11.512 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:11.512 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:11.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:11.580+0000 7fcd83604980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:11.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:11.580+0000 7fcd83604980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:11.616 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:11Z"} 2024-03-04T18:42:11.616 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:11Z"} 2024-03-04T18:42:11.617 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:42:11.617 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:42:11.617 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:42:11.617 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T18:42:16.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:16.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:16.618 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:16.660 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:16.661 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:16.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:16.736+0000 7f1dca4e8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:16.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:16.744+0000 7f1dca4e8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:16.793 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:16Z"} 2024-03-04T18:42:16.793 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:16Z"} 2024-03-04T18:42:16.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:42:16.793 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:42:16.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:42:16.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:42:21.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:21.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:21.795 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:21.838 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:21.838 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:21.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:21.900+0000 7fc049239980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:21.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:21.900+0000 7fc049239980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:21.931 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:21Z"} 2024-03-04T18:42:21.931 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:21Z"} 2024-03-04T18:42:21.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:42:21.931 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:42:21.931 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:42:21.932 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:42:26.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:26.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:26.933 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:27.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:27.052+0000 7fce09840980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:27.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:27.052+0000 7fce09840980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:27.080 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:27Z"} 2024-03-04T18:42:27.080 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:27Z"} 2024-03-04T18:42:27.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:42:27.081 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:42:27.081 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:42:27.081 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:42:32.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:32.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:32.082 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:32.125 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:32.126 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:32.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:32.184+0000 7f8bea794980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:32.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:32.184+0000 7f8bea794980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:32.216 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:32Z"} 2024-03-04T18:42:32.216 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:32Z"} 2024-03-04T18:42:32.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:42:32.216 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:42:32.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:42:32.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:42:37.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:37.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:37.218 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:37.257 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:37.257 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:37.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:37.324+0000 7f30f277b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:37.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:37.324+0000 7f30f277b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:37.359 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:37Z"} 2024-03-04T18:42:37.360 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:37Z"} 2024-03-04T18:42:37.360 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:42:37.360 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:42:37.360 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:42:37.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:42:42.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:42.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:42.362 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:42.404 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:42.404 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:42.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:42.464+0000 7fcf0b38a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:42.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:42.468+0000 7fcf0b38a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:42.495 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:42Z"} 2024-03-04T18:42:42.495 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:42Z"} 2024-03-04T18:42:42.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:42:42.495 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:42:42.495 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:42:42.495 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:42:47.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:47.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:47.497 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:47.539 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:47.540 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:47.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:47.607+0000 7f05776bb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:47.611+0000 7f05776bb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:47.666 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:47Z"} 2024-03-04T18:42:47.666 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:47Z"} 2024-03-04T18:42:47.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:42:47.667 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:42:47.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:42:47.667 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:42:48.987 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:42:48.988+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:42:52.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:52.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:52.669 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:52.712 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:52.712 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:52.779+0000 7f578e37e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:52.779+0000 7f578e37e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:52.810 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:52Z"} 2024-03-04T18:42:52.810 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:52Z"} 2024-03-04T18:42:52.810 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.292440_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.292440Z', 'realm_epoch': 2}}] 2024-03-04T18:42:52.811 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:42:52.811 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.292440_304.1'} 2024-03-04T18:42:52.811 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.292440_304.1 2024-03-04T18:42:57.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:57.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:57.813 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:57.856 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:57.856 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:57.919+0000 7f274a045980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:57.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:42:57.923+0000 7f274a045980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:57.950 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:57Z"} 2024-03-04T18:42:57.951 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:57Z"} 2024-03-04T18:42:57.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:42:57.951 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:42:57.951 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:42:57.951 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T18:43:02.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:02.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:02.953 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:02.996 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:02.997 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:03.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:03.063+0000 7f67402e9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:03.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:03.063+0000 7f67402e9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:03.097 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:03Z"} 2024-03-04T18:43:03.097 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:03Z"} 2024-03-04T18:43:03.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:43:03.097 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:43:03.097 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:43:03.098 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:43:08.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:08.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:08.099 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:08.143 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:08.143 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:08.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:08.199+0000 7f543519f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:08.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:08.199+0000 7f543519f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:08.230 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:08Z"} 2024-03-04T18:43:08.230 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:08Z"} 2024-03-04T18:43:08.231 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:43:08.231 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:43:08.231 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:43:08.231 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:43:13.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:13.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:13.234 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:13.285 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:13.285 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:13.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:13.363+0000 7f4602649980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:13.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:13.363+0000 7f4602649980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:13.393 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:13Z"} 2024-03-04T18:43:13.393 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:13Z"} 2024-03-04T18:43:13.393 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:43:13.394 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:43:13.394 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:43:13.394 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:43:18.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:18.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:18.396 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:18.441 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:18.441 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:18.519+0000 7fdb46bb8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:18.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:18.519+0000 7fdb46bb8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:18.556 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:18Z"} 2024-03-04T18:43:18.557 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:18Z"} 2024-03-04T18:43:18.557 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:43:18.557 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:43:18.557 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:43:18.557 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:43:23.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:23.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:23.559 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:23.601 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:23.601 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:23.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:23.679+0000 7f5f35fea980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:23.679+0000 7f5f35fea980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:23.715 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:23Z"} 2024-03-04T18:43:23.715 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:23Z"} 2024-03-04T18:43:23.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:43:23.716 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:43:23.716 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:43:23.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T18:43:28.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:28.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:28.717 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:28.761 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:28.762 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:28.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:28.859+0000 7f661d4c8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:28.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:28.863+0000 7f661d4c8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:28.887 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:28Z"} 2024-03-04T18:43:28.887 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:28Z"} 2024-03-04T18:43:28.888 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:43:28.888 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:43:28.888 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:43:28.888 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:43:33.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:33.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:33.890 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:33.932 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:33.933 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:34.003+0000 7f04cd550980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:34.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:34.003+0000 7f04cd550980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:34.032 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.489493_317.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.489493Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:34Z"} 2024-03-04T18:43:34.032 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.489493_317.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.489493Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:34Z"} 2024-03-04T18:43:34.032 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.489493_317.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.489493Z', 'realm_epoch': 2}}] 2024-03-04T18:43:34.032 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:43:34.032 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.489493_317.1'} 2024-03-04T18:43:34.033 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.489493_317.1 2024-03-04T18:43:39.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:39.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:39.034 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:39.076 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:39.076 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:39.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:39.154+0000 7f7884a5c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:39.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:39.158+0000 7f7884a5c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:39.192 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:39Z"} 2024-03-04T18:43:39.193 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:39Z"} 2024-03-04T18:43:39.193 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:43:39.193 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:43:39.193 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:43:39.193 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:43:44.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:44.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:44.195 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:44.242 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:44.243 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:44.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:44.314+0000 7f197b5c4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:44.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:44.318+0000 7f197b5c4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:44.344 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:44Z"} 2024-03-04T18:43:44.344 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:44Z"} 2024-03-04T18:43:44.345 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:43:44.345 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:43:44.345 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:43:44.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:43:49.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:49.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:49.346 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:49.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:49.490+0000 7fd00287a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:49.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:49.490+0000 7fd00287a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:49.516 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:49Z"} 2024-03-04T18:43:49.517 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:49Z"} 2024-03-04T18:43:49.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:43:49.517 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:43:49.517 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:43:49.517 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:43:54.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:54.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:54.519 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:54.561 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:54.561 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:54.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:54.638+0000 7fb914f85980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:54.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:54.642+0000 7fb914f85980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:54.670 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:54Z"} 2024-03-04T18:43:54.670 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:54Z"} 2024-03-04T18:43:54.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:43:54.671 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:43:54.671 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:43:54.671 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:43:59.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:59.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:59.672 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:59.715 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:59.715 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:59.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:59.786+0000 7f19c68bf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:43:59.786+0000 7f19c68bf980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:59.813 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:59Z"} 2024-03-04T18:43:59.813 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:59Z"} 2024-03-04T18:43:59.814 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:43:59.814 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:43:59.814 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:43:59.814 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:44:04.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:04.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:04.816 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:04.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:04.950+0000 7f55ab4fa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:04.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:04.950+0000 7f55ab4fa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:04.977 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:04Z"} 2024-03-04T18:44:04.977 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:04Z"} 2024-03-04T18:44:04.977 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:44:04.977 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:44:04.977 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:44:04.978 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:44:09.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:09.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:09.979 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:10.022 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:10.022 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:10.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:10.086+0000 7f03e2f5f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:10.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:10.086+0000 7f03e2f5f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:10.114 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:10Z"} 2024-03-04T18:44:10.115 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:10Z"} 2024-03-04T18:44:10.115 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:44:10.115 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:44:10.115 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:44:10.115 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:44:15.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:15.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:15.118 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:15.161 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:15.161 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:15.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:15.226+0000 7f068cfd0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:15.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:15.226+0000 7f068cfd0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:15.254 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:15Z"} 2024-03-04T18:44:15.255 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:15Z"} 2024-03-04T18:44:15.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:44:15.255 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:44:15.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:44:15.255 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:44:20.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:20.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:20.257 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:20.302 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:20.303 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:20.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:20.382+0000 7fb96c3e3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:20.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:20.382+0000 7fb96c3e3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:20.410 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:20Z"} 2024-03-04T18:44:20.410 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:20Z"} 2024-03-04T18:44:20.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:44:20.410 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:44:20.410 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:44:20.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:44:25.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:25.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:25.412 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:25.455 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:25.455 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:25.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:25.530+0000 7f78bbe31980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:25.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:25.530+0000 7f78bbe31980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:25.560 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:25Z"} 2024-03-04T18:44:25.560 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:25Z"} 2024-03-04T18:44:25.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.292440_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.292440Z', 'realm_epoch': 2}}] 2024-03-04T18:44:25.561 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:44:25.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.292440_304.1'} 2024-03-04T18:44:25.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.292440_304.1 2024-03-04T18:44:30.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:30.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:30.562 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:30.605 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:30.606 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:30.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:30.678+0000 7f6d6af33980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:30.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:30.678+0000 7f6d6af33980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:30.709 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:30Z"} 2024-03-04T18:44:30.709 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:30Z"} 2024-03-04T18:44:30.709 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:44:30.709 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:44:30.709 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:44:30.709 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:44:35.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:35.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:35.711 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:35.753 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:35.753 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:35.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:35.821+0000 7f39573dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:35.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:35.821+0000 7f39573dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:35.856 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:35Z"} 2024-03-04T18:44:35.857 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:35Z"} 2024-03-04T18:44:35.857 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:44:35.857 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:44:35.857 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:44:35.857 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:44:40.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:40.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:40.859 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:40.909 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:40.910 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:40.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:40.985+0000 7f9f3adfc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:40.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:40.985+0000 7f9f3adfc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:41.014 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:40Z"} 2024-03-04T18:44:41.014 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:40Z"} 2024-03-04T18:44:41.015 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:44:41.015 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:44:41.015 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:44:41.015 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:44:46.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:46.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:46.016 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:46.059 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:46.059 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:46.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:46.125+0000 7fd991bdc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:46.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:46.125+0000 7fd991bdc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:46.152 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:46Z"} 2024-03-04T18:44:46.152 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:46Z"} 2024-03-04T18:44:46.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:44:46.152 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:44:46.152 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:44:46.153 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:44:51.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:51.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:51.154 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:51.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:51.257+0000 7f2a5475d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:51.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:51.257+0000 7f2a5475d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:51.285 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:51Z"} 2024-03-04T18:44:51.285 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:51Z"} 2024-03-04T18:44:51.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:44:51.286 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:44:51.286 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:44:51.286 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T18:44:56.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:56.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:56.287 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:56.341 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:56.341 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:56.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:56.425+0000 7f9434e28980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:56.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:44:56.425+0000 7f9434e28980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:56.458 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:56Z"} 2024-03-04T18:44:56.458 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:56Z"} 2024-03-04T18:44:56.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:44:56.458 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:44:56.459 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:44:56.459 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T18:45:01.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:01.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:01.460 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:01.502 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:01.503 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:01.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:01.581+0000 7f547a394980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:01.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:01.585+0000 7f547a394980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:01.612 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:01Z"} 2024-03-04T18:45:01.613 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:01Z"} 2024-03-04T18:45:01.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:45:01.613 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:45:01.613 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:45:01.613 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:45:06.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:06.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:06.614 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:06.649 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:06.649 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:06.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:06.721+0000 7fe5369b3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:06.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:06.721+0000 7fe5369b3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:06.753 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:06Z"} 2024-03-04T18:45:06.753 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:06Z"} 2024-03-04T18:45:06.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.523626_319.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.523626Z', 'realm_epoch': 2}}] 2024-03-04T18:45:06.753 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:45:06.753 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.523626_319.1'} 2024-03-04T18:45:06.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.523626_319.1 2024-03-04T18:45:11.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:11.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:11.755 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:11.800 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:11.800 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:11.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:11.881+0000 7f0dd0411980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:11.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:11.881+0000 7f0dd0411980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:11.915 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:11Z"} 2024-03-04T18:45:11.915 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:11Z"} 2024-03-04T18:45:11.916 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:45:11.916 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:45:11.916 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:45:11.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:45:16.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:16.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:16.917 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:16.959 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:16.960 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:17.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:17.025+0000 7f8e4b86b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:17.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:17.025+0000 7f8e4b86b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:17.054 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:17Z"} 2024-03-04T18:45:17.054 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:17Z"} 2024-03-04T18:45:17.054 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:45:17.055 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:45:17.055 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:45:17.055 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:45:22.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:22.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:22.057 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:22.100 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:22.101 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:22.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:22.185+0000 7fc0d9e04980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:22.185+0000 7fc0d9e04980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:22.216 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:22Z"} 2024-03-04T18:45:22.217 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:22Z"} 2024-03-04T18:45:22.217 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:45:22.217 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T18:45:22.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:45:22.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T18:45:27.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... FAIL 2024-03-04T18:45:27.221 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-67 2024-03-04T18:45:27.221 DEBUG:boto:path=/ppfahn-67/ 2024-03-04T18:45:27.221 DEBUG:boto:auth_path=/ppfahn-67/ 2024-03-04T18:45:27.222 DEBUG:boto:Method: PUT 2024-03-04T18:45:27.222 DEBUG:boto:Path: /ppfahn-67/ 2024-03-04T18:45:27.222 DEBUG:boto:Data: 2024-03-04T18:45:27.222 DEBUG:boto:Headers: {} 2024-03-04T18:45:27.222 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:45:27.222 DEBUG:boto:Port: 8000 2024-03-04T18:45:27.222 DEBUG:boto:Params: {} 2024-03-04T18:45:27.222 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:45:27.223 DEBUG:boto:Token: None 2024-03-04T18:45:27.223 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:45:27 GMT /ppfahn-67/ 2024-03-04T18:45:27.223 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:bGT6qDvkxaZV22cXL+tO+WF6TMQ= 2024-03-04T18:45:27.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:45:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bGT6qDvkxaZV22cXL+tO+WF6TMQ=', 'Content-Length': '0'} 2024-03-04T18:45:27.269 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000730c0352672deb99-0065e616c7-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:45:27.269 DEBUG:boto:path=/ppfahn-67/testobj-sse-c 2024-03-04T18:45:27.269 DEBUG:boto:auth_path=/ppfahn-67/testobj-sse-c 2024-03-04T18:45:27.270 DEBUG:boto:Method: PUT 2024-03-04T18:45:27.270 DEBUG:boto:Path: /ppfahn-67/testobj-sse-c 2024-03-04T18:45:27.270 DEBUG:boto:Data: 2024-03-04T18:45:27.270 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-03-04T18:45:27.270 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:45:27.270 DEBUG:boto:Port: 8000 2024-03-04T18:45:27.270 DEBUG:boto:Params: {} 2024-03-04T18:45:27.270 DEBUG:boto:Token: None 2024-03-04T18:45:27.270 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Mon, 04 Mar 2024 18:45:27 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /ppfahn-67/testobj-sse-c 2024-03-04T18:45:27.271 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JHgcBG/xAsffu0O8L15wlk3g+XU= 2024-03-04T18:45:27.271 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:45:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JHgcBG/xAsffu0O8L15wlk3g+XU='} 2024-03-04T18:45:27.333 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000c36442f349410c26-0065e616c7-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:45:27.334 DEBUG:boto:path=/ppfahn-67/testobj-sse-kms 2024-03-04T18:45:27.334 DEBUG:boto:auth_path=/ppfahn-67/testobj-sse-kms 2024-03-04T18:45:27.334 DEBUG:boto:Method: PUT 2024-03-04T18:45:27.334 DEBUG:boto:Path: /ppfahn-67/testobj-sse-kms 2024-03-04T18:45:27.334 DEBUG:boto:Data: 2024-03-04T18:45:27.334 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-03-04T18:45:27.334 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:45:27.334 DEBUG:boto:Port: 8000 2024-03-04T18:45:27.334 DEBUG:boto:Params: {} 2024-03-04T18:45:27.335 DEBUG:boto:Token: None 2024-03-04T18:45:27.335 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Mon, 04 Mar 2024 18:45:27 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /ppfahn-67/testobj-sse-kms 2024-03-04T18:45:27.335 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:NkVUiAyZrR6sGEKRd5C9Z4M/47U= 2024-03-04T18:45:27.335 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:45:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NkVUiAyZrR6sGEKRd5C9Z4M/47U='} 2024-03-04T18:45:27.396 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6cHBmYWhuLTY3L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000d24b4df8d2ae5d2c-0065e616c7-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:45:27.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:27.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:27.397 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:27.465 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:27.465 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:27.532 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T18:45:27.557 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709577927.265995_602.1","last_update":"2024-03-04T18:45:27.265995Z"}] 2024-03-04T18:45:27.557 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:45:27.557 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:45:27.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:27.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:27.558 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:27.602 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:27.602 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:27.673+0000 7f1c23bb2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:27.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:27.677+0000 7f1c23bb2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:27.715 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:27Z"} 2024-03-04T18:45:27.715 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:27Z"} 2024-03-04T18:45:27.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:45:27.715 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:45:27.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:45:27.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:45:32.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:32.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:32.717 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:32.778 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:32.778 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:32.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:32.840+0000 7fbf81cb3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:32.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:32.840+0000 7fbf81cb3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:32.875 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:32Z"} 2024-03-04T18:45:32.875 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:32Z"} 2024-03-04T18:45:32.876 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:45:32.876 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:45:32.876 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:45:32.876 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:45:35.729 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:45:35.728+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:45:35.744 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:45:35.728+0000 7f5c1f903640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T18:45:35.744 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:45:35.732+0000 7f5c21907640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T18:45:37.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:37.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:37.878 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:37.920 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:37.920 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:37.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:37.992+0000 7f7579c2c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:37.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:37.992+0000 7f7579c2c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:38.024 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:37Z"} 2024-03-04T18:45:38.024 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:37Z"} 2024-03-04T18:45:38.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:45:38.024 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:45:38.024 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:45:38.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:45:43.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:43.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:43.026 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:43.068 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:43.069 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:43.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:43.132+0000 7fdae5ce1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:43.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:43.132+0000 7fdae5ce1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:43.169 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:43Z"} 2024-03-04T18:45:43.170 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:43Z"} 2024-03-04T18:45:43.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:45:43.170 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:45:43.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:45:43.170 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:45:48.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:48.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:48.172 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:48.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:48.280+0000 7f0381018980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:48.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:48.280+0000 7f0381018980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:48.315 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:48Z"} 2024-03-04T18:45:48.316 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:48Z"} 2024-03-04T18:45:48.316 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:45:48.316 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:45:48.316 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:45:48.316 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:45:53.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:53.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:53.317 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:53.360 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:53.360 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:53.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:53.436+0000 7f5754291980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:53.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:53.440+0000 7f5754291980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:53.469 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:53Z"} 2024-03-04T18:45:53.469 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:53Z"} 2024-03-04T18:45:53.470 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:45:53.470 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:45:53.470 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:45:53.470 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:45:58.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:58.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:58.471 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:58.513 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:58.513 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:58.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:58.588+0000 7f2d14ef2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:45:58.588+0000 7f2d14ef2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:58.620 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:58Z"} 2024-03-04T18:45:58.621 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:58Z"} 2024-03-04T18:45:58.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:45:58.621 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:45:58.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:45:58.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:46:03.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:03.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:03.623 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:03.665 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:03.665 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:03.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:03.736+0000 7f12b6b3f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:03.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:03.736+0000 7f12b6b3f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:03.770 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:03Z"} 2024-03-04T18:46:03.770 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:03Z"} 2024-03-04T18:46:03.770 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:46:03.770 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:46:03.770 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:46:03.771 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:46:08.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:08.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:08.772 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:08.820 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:08.820 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:08.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:08.880+0000 7f8663f11980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:08.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:08.880+0000 7f8663f11980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:08.915 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:08Z"} 2024-03-04T18:46:08.915 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:08Z"} 2024-03-04T18:46:08.916 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:46:08.916 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:46:08.916 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:46:08.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:46:13.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:13.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:13.917 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:13.970 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:13.970 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:14.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:14.064+0000 7f537ebe7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:14.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:14.064+0000 7f537ebe7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:14.099 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:14Z"} 2024-03-04T18:46:14.099 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:14Z"} 2024-03-04T18:46:14.100 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:46:14.100 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:46:14.100 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:46:14.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:46:19.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:19.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:19.101 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:19.212+0000 7f87a34b2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:19.212+0000 7f87a34b2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:19.245 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:19Z"} 2024-03-04T18:46:19.245 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:19Z"} 2024-03-04T18:46:19.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:46:19.245 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:46:19.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:46:19.246 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:46:24.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:24.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:24.247 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:24.289 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:24.289 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:24.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:24.364+0000 7f5c9e39e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:24.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:24.364+0000 7f5c9e39e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:24.400 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:24Z"} 2024-03-04T18:46:24.400 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:24Z"} 2024-03-04T18:46:24.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:46:24.401 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:46:24.401 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:46:24.401 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:46:26.403 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:46:26.403+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:46:29.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:29.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:29.403 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:29.446 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:29.446 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:29.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:29.507+0000 7f616f76a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:29.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:29.511+0000 7f616f76a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:29.539 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:29Z"} 2024-03-04T18:46:29.540 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:29Z"} 2024-03-04T18:46:29.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:46:29.540 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:46:29.540 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:46:29.540 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:46:34.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:34.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:34.542 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:34.599 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:34.599 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:34.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:34.663+0000 7fd551a15980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:34.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:34.667+0000 7fd551a15980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:34.696 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:34Z"} 2024-03-04T18:46:34.696 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:34Z"} 2024-03-04T18:46:34.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:46:34.696 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:46:34.696 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:46:34.696 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:46:39.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:39.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:39.698 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:39.807+0000 7fefad610980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:39.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:39.811+0000 7fefad610980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:39.841 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.735016_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.735016Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:39Z"} 2024-03-04T18:46:39.841 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.735016_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.735016Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:39Z"} 2024-03-04T18:46:39.841 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.735016_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.735016Z', 'realm_epoch': 2}}] 2024-03-04T18:46:39.842 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:46:39.842 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.735016_329.1'} 2024-03-04T18:46:39.842 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575077.735016_329.1 2024-03-04T18:46:44.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:44.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:44.844 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:44.886 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:44.886 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:44.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:44.967+0000 7fef304ac980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:44.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:44.967+0000 7fef304ac980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:44.998 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:44Z"} 2024-03-04T18:46:44.998 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:44Z"} 2024-03-04T18:46:44.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:46:44.999 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:46:44.999 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:46:44.999 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:46:50.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:50.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:50.001 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:50.044 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:50.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:50.107+0000 7f4b3b6c9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:50.107+0000 7f4b3b6c9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:50.137 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:50Z"} 2024-03-04T18:46:50.137 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:50Z"} 2024-03-04T18:46:50.137 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:46:50.137 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:46:50.138 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:46:50.138 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:46:55.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:55.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:55.139 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:55.183 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:55.183 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:55.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:55.247+0000 7f5eb8a0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:55.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:46:55.247+0000 7f5eb8a0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:55.283 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:55Z"} 2024-03-04T18:46:55.283 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:55Z"} 2024-03-04T18:46:55.283 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:46:55.283 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:46:55.283 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:46:55.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:47:00.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:00.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:00.286 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:00.329 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:00.329 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:00.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:00.387+0000 7ff8666cb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:00.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:00.387+0000 7ff8666cb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:00.418 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:00Z"} 2024-03-04T18:47:00.418 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:00Z"} 2024-03-04T18:47:00.419 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:47:00.419 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:47:00.419 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:47:00.419 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:47:05.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:05.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:05.421 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:05.464 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:05.464 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:05.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:05.523+0000 7ff394ba6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:05.527+0000 7ff394ba6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:05.575 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:05Z"} 2024-03-04T18:47:05.576 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:05Z"} 2024-03-04T18:47:05.576 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:47:05.576 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:47:05.576 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:47:05.576 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:47:10.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:10.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:10.578 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:10.621 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:10.621 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:10.703+0000 7f82f0e63980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:10.703+0000 7f82f0e63980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:10.728 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:10Z"} 2024-03-04T18:47:10.729 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:10Z"} 2024-03-04T18:47:10.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:47:10.729 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:47:10.729 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:47:10.729 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:47:15.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:15.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:15.730 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:15.772 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:15.772 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:15.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:15.843+0000 7f855a856980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:15.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:15.847+0000 7f855a856980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:15.903 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:15Z"} 2024-03-04T18:47:15.903 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:15Z"} 2024-03-04T18:47:15.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:47:15.903 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:47:15.904 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:47:15.904 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:47:20.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:20.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:20.905 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:20.948 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:20.948 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:21.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:21.019+0000 7f2773d33980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:21.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:21.019+0000 7f2773d33980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:21.045 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.393343_310.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.393343Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:21Z"} 2024-03-04T18:47:21.045 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.393343_310.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.393343Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:21Z"} 2024-03-04T18:47:21.046 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.393343_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.393343Z', 'realm_epoch': 2}}] 2024-03-04T18:47:21.046 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:47:21.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.393343_310.1'} 2024-03-04T18:47:21.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.393343_310.1 2024-03-04T18:47:26.048 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:26.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:26.048 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:26.093 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:26.093 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:26.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:26.174+0000 7f44a1979980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:26.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:26.174+0000 7f44a1979980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:26.208 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:26Z"} 2024-03-04T18:47:26.208 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:26Z"} 2024-03-04T18:47:26.208 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:47:26.208 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:47:26.208 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:47:26.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:47:31.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:31.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:31.305 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:31.350 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:31.350 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:31.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:31.426+0000 7fb339f8d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:31.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:31.426+0000 7fb339f8d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:31.454 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:31Z"} 2024-03-04T18:47:31.454 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:31Z"} 2024-03-04T18:47:31.455 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:47:31.455 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:47:31.455 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:47:31.455 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:47:36.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:36.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:36.457 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:36.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:36.582+0000 7fdf49993980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:36.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:36.582+0000 7fdf49993980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:36.611 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:36Z"} 2024-03-04T18:47:36.612 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:36Z"} 2024-03-04T18:47:36.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:47:36.612 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:47:36.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:47:36.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:47:41.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:41.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:41.614 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:41.656 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:41.656 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:41.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:41.718+0000 7ff10bf4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:41.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:41.718+0000 7ff10bf4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:41.757 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:41Z"} 2024-03-04T18:47:41.757 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:41Z"} 2024-03-04T18:47:41.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:47:41.757 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:47:41.758 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:47:41.758 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:47:46.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:46.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:46.759 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:46.801 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:46.801 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:46.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:46.862+0000 7fe8fb535980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:46.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:46.862+0000 7fe8fb535980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:46.893 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:46Z"} 2024-03-04T18:47:46.893 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:46Z"} 2024-03-04T18:47:46.893 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:47:46.893 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:47:46.893 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:47:46.893 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.310269_305.1 2024-03-04T18:47:51.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:51.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:51.895 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:51.955 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:51.956 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:52.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:52.042+0000 7efd0d606980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:52.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:52.046+0000 7efd0d606980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:52.077 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:52Z"} 2024-03-04T18:47:52.077 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:52Z"} 2024-03-04T18:47:52.077 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:47:52.077 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:47:52.078 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:47:52.078 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:47:57.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:57.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:57.079 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:57.121 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:57.121 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:57.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:57.190+0000 7ff4ee3b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:57.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:47:57.190+0000 7ff4ee3b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:57.218 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:57Z"} 2024-03-04T18:47:57.218 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:57Z"} 2024-03-04T18:47:57.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:47:57.218 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:47:57.218 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:47:57.219 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:48:02.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:02.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:02.220 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:02.262 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:02.262 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:02.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:02.342+0000 7f78f4fce980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:02.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:02.342+0000 7f78f4fce980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:02.378 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:02Z"} 2024-03-04T18:48:02.378 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:02Z"} 2024-03-04T18:48:02.378 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:48:02.378 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:48:02.378 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:48:02.378 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:48:07.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:07.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:07.380 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:07.425 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:07.425 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:07.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:07.502+0000 7f6fda780980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:07.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:07.502+0000 7f6fda780980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:07.532 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:07Z"} 2024-03-04T18:48:07.532 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:07Z"} 2024-03-04T18:48:07.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:48:07.532 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:48:07.533 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:48:07.533 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:48:09.119 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:48:09.120+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:48:12.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:12.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:12.534 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:12.590 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:12.590 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:12.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:12.662+0000 7faee5f88980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:12.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:12.662+0000 7faee5f88980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:12.695 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:12Z"} 2024-03-04T18:48:12.695 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:12Z"} 2024-03-04T18:48:12.695 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:48:12.696 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:48:12.696 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:48:12.696 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:48:17.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:17.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:17.698 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:17.741 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:17.741 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:17.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:17.814+0000 7ffaa374b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:17.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:17.814+0000 7ffaa374b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:17.843 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:17Z"} 2024-03-04T18:48:17.843 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:17Z"} 2024-03-04T18:48:17.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:48:17.844 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:48:17.844 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:48:17.844 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:48:22.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:22.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:22.845 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:22.887 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:22.887 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:22.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:22.957+0000 7fb671420980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:22.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:22.957+0000 7fb671420980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:22.993 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:22Z"} 2024-03-04T18:48:22.993 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:22Z"} 2024-03-04T18:48:22.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:48:22.994 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:48:22.994 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:48:22.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:48:27.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:27.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:27.995 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:28.037 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:28.038 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:28.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:28.097+0000 7f9d5e281980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:28.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:28.101+0000 7f9d5e281980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:28.126 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:28Z"} 2024-03-04T18:48:28.127 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:28Z"} 2024-03-04T18:48:28.127 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:48:28.127 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:48:28.127 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:48:28.127 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:48:33.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:33.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:33.128 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:33.171 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:33.171 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:33.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:33.233+0000 7fb992f4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:33.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:33.233+0000 7fb992f4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:33.262 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:33Z"} 2024-03-04T18:48:33.262 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:33Z"} 2024-03-04T18:48:33.263 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:48:33.263 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:48:33.263 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:48:33.263 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:48:38.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:38.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:38.265 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:38.318 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:38.318 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:38.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:38.377+0000 7fea846e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:38.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:38.377+0000 7fea846e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:38.404 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:38Z"} 2024-03-04T18:48:38.404 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:38Z"} 2024-03-04T18:48:38.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:48:38.404 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:48:38.404 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:48:38.405 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:48:43.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:43.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:43.406 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:43.448 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:43.448 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:43.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:43.509+0000 7fc20a1d7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:43.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:43.509+0000 7fc20a1d7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:43.538 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:43Z"} 2024-03-04T18:48:43.539 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:43Z"} 2024-03-04T18:48:43.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:48:43.539 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:48:43.539 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:48:43.539 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:48:48.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:48.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:48.541 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:48.582 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:48.583 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:48.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:48.649+0000 7f3017362980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:48.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:48.649+0000 7f3017362980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:48.677 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:48Z"} 2024-03-04T18:48:48.677 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:48Z"} 2024-03-04T18:48:48.677 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:48:48.677 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:48:48.678 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:48:48.678 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:48:53.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:53.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:53.680 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:53.722 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:53.722 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:53.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:53.789+0000 7f0f01ed9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:53.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:53.789+0000 7f0f01ed9980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:53.819 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:53Z"} 2024-03-04T18:48:53.819 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:53Z"} 2024-03-04T18:48:53.819 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:48:53.820 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:48:53.820 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:48:53.820 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:48:58.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:58.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:58.821 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:58.864 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:58.864 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:58.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:58.945+0000 7f72e604f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:58.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:48:58.945+0000 7f72e604f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:58.973 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:58Z"} 2024-03-04T18:48:58.973 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:58Z"} 2024-03-04T18:48:58.973 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:48:58.973 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:48:58.973 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:48:58.973 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:49:03.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:03.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:03.975 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:04.019 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:04.019 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:04.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:04.077+0000 7f3e83033980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:04.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:04.077+0000 7f3e83033980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:04.106 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:04Z"} 2024-03-04T18:49:04.106 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:04Z"} 2024-03-04T18:49:04.107 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:49:04.107 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:49:04.107 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:49:04.107 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:49:09.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:09.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:09.108 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:09.151 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:09.151 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:09.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:09.221+0000 7fe6b5046980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:09.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:09.221+0000 7fe6b5046980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:09.251 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:09Z"} 2024-03-04T18:49:09.252 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:09Z"} 2024-03-04T18:49:09.252 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:49:09.252 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:49:09.252 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:49:09.252 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:49:14.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:14.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:14.254 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:14.295 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:14.295 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:14.373+0000 7fd7c8d9f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:14.377+0000 7fd7c8d9f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:14.408 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:14Z"} 2024-03-04T18:49:14.408 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:14Z"} 2024-03-04T18:49:14.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:49:14.409 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:49:14.409 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:49:14.409 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:49:19.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:19.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:19.411 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:19.453 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:19.453 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:19.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:19.525+0000 7f1c843b1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:19.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:19.529+0000 7f1c843b1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:19.556 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:19Z"} 2024-03-04T18:49:19.556 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:19Z"} 2024-03-04T18:49:19.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:49:19.557 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:49:19.557 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:49:19.557 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:49:24.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:24.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:24.559 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:24.602 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:24.603 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:24.668+0000 7f8eb01f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:24.672+0000 7f8eb01f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:24.703 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:24Z"} 2024-03-04T18:49:24.703 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:24Z"} 2024-03-04T18:49:24.703 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:49:24.703 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:49:24.703 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:49:24.704 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:49:29.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:29.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:29.705 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:29.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:29.840+0000 7f9643ccb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:29.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:29.840+0000 7f9643ccb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:29.878 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:29Z"} 2024-03-04T18:49:29.878 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:29Z"} 2024-03-04T18:49:29.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:49:29.879 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:49:29.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:49:29.879 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:49:34.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:34.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:34.880 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:34.923 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:34.923 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:34.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:34.992+0000 7f7725c1d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:34.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:34.992+0000 7f7725c1d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:35.023 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:34Z"} 2024-03-04T18:49:35.024 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:34Z"} 2024-03-04T18:49:35.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:49:35.024 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:49:35.024 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:49:35.025 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:49:40.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:40.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:40.026 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:40.069 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:40.069 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:40.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:40.136+0000 7f30d780b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:40.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:40.136+0000 7f30d780b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:40.169 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:40Z"} 2024-03-04T18:49:40.169 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:40Z"} 2024-03-04T18:49:40.169 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:49:40.170 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:49:40.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:49:40.170 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:49:45.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:45.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:45.172 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:45.214 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:45.214 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:45.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:45.288+0000 7f8bcd7bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:45.288+0000 7f8bcd7bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:45.321 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.424470_313.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.424470Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:45Z"} 2024-03-04T18:49:45.322 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.424470_313.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.424470Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:45Z"} 2024-03-04T18:49:45.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.424470_313.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.424470Z', 'realm_epoch': 2}}] 2024-03-04T18:49:45.322 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:49:45.322 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.424470_313.1'} 2024-03-04T18:49:45.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.424470_313.1 2024-03-04T18:49:50.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:50.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:50.324 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:50.368 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:50.368 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:50.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:50.440+0000 7fd2675c2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:50.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:50.440+0000 7fd2675c2980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:50.469 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:50Z"} 2024-03-04T18:49:50.470 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:50Z"} 2024-03-04T18:49:50.470 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:49:50.470 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:49:50.470 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:49:50.470 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:49:55.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:55.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:55.472 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:55.517 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:55.517 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:55.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:55.576+0000 7feff27af980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:55.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:49:55.576+0000 7feff27af980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:55.606 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:55Z"} 2024-03-04T18:49:55.606 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:55Z"} 2024-03-04T18:49:55.606 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:49:55.607 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:49:55.607 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:49:55.607 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:50:00.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:00.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:00.608 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:00.650 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:00.650 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:00.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:00.712+0000 7f8024d37980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:00.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:00.712+0000 7f8024d37980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:00.740 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:00Z"} 2024-03-04T18:50:00.740 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:00Z"} 2024-03-04T18:50:00.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:50:00.740 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:50:00.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:50:00.740 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:50:05.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:05.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:05.743 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:05.787 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:05.787 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:05.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:05.864+0000 7f4cb7acd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:05.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:05.868+0000 7f4cb7acd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:05.902 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:05Z"} 2024-03-04T18:50:05.903 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:05Z"} 2024-03-04T18:50:05.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:50:05.903 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:50:05.903 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:50:05.903 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T18:50:10.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:10.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:10.905 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:11.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:11.012+0000 7f789091b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:11.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:11.016+0000 7f789091b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:11.048 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:11Z"} 2024-03-04T18:50:11.048 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:11Z"} 2024-03-04T18:50:11.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:50:11.049 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:50:11.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:50:11.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:50:16.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:16.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:16.051 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:16.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:16.164+0000 7f9dbc73d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:16.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:16.164+0000 7f9dbc73d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:16.205 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:16Z"} 2024-03-04T18:50:16.205 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:16Z"} 2024-03-04T18:50:16.206 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:50:16.206 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:50:16.206 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:50:16.206 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:50:21.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:21.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:21.208 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:21.251 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:21.251 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:21.319+0000 7f41bcf95980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:21.319+0000 7f41bcf95980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:21.354 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:21Z"} 2024-03-04T18:50:21.354 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:21Z"} 2024-03-04T18:50:21.354 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:50:21.355 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:50:21.355 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:50:21.355 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:50:26.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:26.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:26.356 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:26.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:26.487+0000 7fb8a467d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:26.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:26.487+0000 7fb8a467d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:26.517 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:26Z"} 2024-03-04T18:50:26.518 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:26Z"} 2024-03-04T18:50:26.518 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:50:26.518 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:50:26.518 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:50:26.518 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:50:31.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:31.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:31.520 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:31.564 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:31.564 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:31.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:31.627+0000 7f731fca3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:31.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:31.627+0000 7f731fca3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:31.656 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:31Z"} 2024-03-04T18:50:31.656 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:31Z"} 2024-03-04T18:50:31.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:50:31.657 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T18:50:31.657 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:50:31.657 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T18:50:36.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... FAIL 2024-03-04T18:50:36.662 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-68 2024-03-04T18:50:36.662 DEBUG:boto:path=/ppfahn-68/ 2024-03-04T18:50:36.662 DEBUG:boto:auth_path=/ppfahn-68/ 2024-03-04T18:50:36.662 DEBUG:boto:Method: PUT 2024-03-04T18:50:36.663 DEBUG:boto:Path: /ppfahn-68/ 2024-03-04T18:50:36.663 DEBUG:boto:Data: 2024-03-04T18:50:36.663 DEBUG:boto:Headers: {} 2024-03-04T18:50:36.663 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:50:36.663 DEBUG:boto:Port: 8000 2024-03-04T18:50:36.663 DEBUG:boto:Params: {} 2024-03-04T18:50:36.664 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:50:36.664 DEBUG:boto:Token: None 2024-03-04T18:50:36.664 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:50:36 GMT /ppfahn-68/ 2024-03-04T18:50:36.665 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:mwb6PUqwiT/z7X7aNOHMfkP5Fz4= 2024-03-04T18:50:36.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mwb6PUqwiT/z7X7aNOHMfkP5Fz4=', 'Content-Length': '0'} 2024-03-04T18:50:36.714 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c1fc30bf1fdb2bad-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:50:36.714 DEBUG:boto:path=/ppfahn-68/ 2024-03-04T18:50:36.714 DEBUG:boto:auth_path=/ppfahn-68/ 2024-03-04T18:50:36.714 DEBUG:boto:Method: HEAD 2024-03-04T18:50:36.714 DEBUG:boto:Path: /ppfahn-68/ 2024-03-04T18:50:36.714 DEBUG:boto:Data: 2024-03-04T18:50:36.715 DEBUG:boto:Headers: {} 2024-03-04T18:50:36.715 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:50:36.715 DEBUG:boto:Port: 8000 2024-03-04T18:50:36.715 DEBUG:boto:Params: {} 2024-03-04T18:50:36.715 DEBUG:boto:Token: None 2024-03-04T18:50:36.715 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:50:36 GMT /ppfahn-68/ 2024-03-04T18:50:36.715 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM= 2024-03-04T18:50:36.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM=', 'Content-Length': '0'} 2024-03-04T18:50:36.721 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004598b08dc07cbf62-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:50:36.721 DEBUG:boto:path=/ppfahn-68/a 2024-03-04T18:50:36.721 DEBUG:boto:auth_path=/ppfahn-68/a 2024-03-04T18:50:36.721 DEBUG:boto:Method: PUT 2024-03-04T18:50:36.722 DEBUG:boto:Path: /ppfahn-68/a 2024-03-04T18:50:36.722 DEBUG:boto:Data: 2024-03-04T18:50:36.722 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T18:50:36.722 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:50:36.722 DEBUG:boto:Port: 8000 2024-03-04T18:50:36.722 DEBUG:boto:Params: {} 2024-03-04T18:50:36.722 DEBUG:boto:Token: None 2024-03-04T18:50:36.723 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 18:50:36 GMT /ppfahn-68/a 2024-03-04T18:50:36.723 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:d0taOZ22dCQWa/fDwGf8feucCwY= 2024-03-04T18:50:36.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:d0taOZ22dCQWa/fDwGf8feucCwY='} 2024-03-04T18:50:36.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005af2cf190a2b0c1c-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:50:36.738 DEBUG:boto:path=/ppfahn-68/ 2024-03-04T18:50:36.738 DEBUG:boto:auth_path=/ppfahn-68/ 2024-03-04T18:50:36.738 DEBUG:boto:Method: HEAD 2024-03-04T18:50:36.738 DEBUG:boto:Path: /ppfahn-68/ 2024-03-04T18:50:36.739 DEBUG:boto:Data: 2024-03-04T18:50:36.739 DEBUG:boto:Headers: {} 2024-03-04T18:50:36.739 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:50:36.739 DEBUG:boto:Port: 8000 2024-03-04T18:50:36.739 DEBUG:boto:Params: {} 2024-03-04T18:50:36.739 DEBUG:boto:Token: None 2024-03-04T18:50:36.739 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:50:36 GMT /ppfahn-68/ 2024-03-04T18:50:36.739 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM= 2024-03-04T18:50:36.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM=', 'Content-Length': '0'} 2024-03-04T18:50:36.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a79932c357452e77-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:50:36.744 DEBUG:boto:path=/ppfahn-68/b 2024-03-04T18:50:36.744 DEBUG:boto:auth_path=/ppfahn-68/b 2024-03-04T18:50:36.745 DEBUG:boto:Method: PUT 2024-03-04T18:50:36.745 DEBUG:boto:Path: /ppfahn-68/b 2024-03-04T18:50:36.745 DEBUG:boto:Data: 2024-03-04T18:50:36.745 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T18:50:36.745 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:50:36.745 DEBUG:boto:Port: 8000 2024-03-04T18:50:36.745 DEBUG:boto:Params: {} 2024-03-04T18:50:36.745 DEBUG:boto:Token: None 2024-03-04T18:50:36.746 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 18:50:36 GMT /ppfahn-68/b 2024-03-04T18:50:36.746 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:vn7WgJ2H+/v0ikR51ydC8eDdNWg= 2024-03-04T18:50:36.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vn7WgJ2H+/v0ikR51ydC8eDdNWg='} 2024-03-04T18:50:36.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002865ed5d55ede09d-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:50:36.792 DEBUG:boto:path=/ppfahn-68/ 2024-03-04T18:50:36.793 DEBUG:boto:auth_path=/ppfahn-68/ 2024-03-04T18:50:36.793 DEBUG:boto:Method: HEAD 2024-03-04T18:50:36.793 DEBUG:boto:Path: /ppfahn-68/ 2024-03-04T18:50:36.793 DEBUG:boto:Data: 2024-03-04T18:50:36.793 DEBUG:boto:Headers: {} 2024-03-04T18:50:36.793 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:50:36.793 DEBUG:boto:Port: 8000 2024-03-04T18:50:36.793 DEBUG:boto:Params: {} 2024-03-04T18:50:36.793 DEBUG:boto:Token: None 2024-03-04T18:50:36.794 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:50:36 GMT /ppfahn-68/ 2024-03-04T18:50:36.794 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM= 2024-03-04T18:50:36.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM=', 'Content-Length': '0'} 2024-03-04T18:50:36.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a10c3ca9d09d611-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:50:36.799 DEBUG:boto:path=/ppfahn-68/c 2024-03-04T18:50:36.799 DEBUG:boto:auth_path=/ppfahn-68/c 2024-03-04T18:50:36.799 DEBUG:boto:Method: PUT 2024-03-04T18:50:36.799 DEBUG:boto:Path: /ppfahn-68/c 2024-03-04T18:50:36.799 DEBUG:boto:Data: 2024-03-04T18:50:36.799 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T18:50:36.799 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:50:36.799 DEBUG:boto:Port: 8000 2024-03-04T18:50:36.799 DEBUG:boto:Params: {} 2024-03-04T18:50:36.800 DEBUG:boto:Token: None 2024-03-04T18:50:36.800 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 18:50:36 GMT /ppfahn-68/c 2024-03-04T18:50:36.800 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:CAFmecd/Si+5RKT/OdcVfv4Gz8w= 2024-03-04T18:50:36.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CAFmecd/Si+5RKT/OdcVfv4Gz8w='} 2024-03-04T18:50:36.815 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001fe82e2188b91e35-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:50:36.816 DEBUG:boto:path=/ppfahn-68/ 2024-03-04T18:50:36.816 DEBUG:boto:auth_path=/ppfahn-68/ 2024-03-04T18:50:36.816 DEBUG:boto:Method: HEAD 2024-03-04T18:50:36.816 DEBUG:boto:Path: /ppfahn-68/ 2024-03-04T18:50:36.816 DEBUG:boto:Data: 2024-03-04T18:50:36.816 DEBUG:boto:Headers: {} 2024-03-04T18:50:36.816 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:50:36.817 DEBUG:boto:Port: 8000 2024-03-04T18:50:36.817 DEBUG:boto:Params: {} 2024-03-04T18:50:36.817 DEBUG:boto:Token: None 2024-03-04T18:50:36.817 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:50:36 GMT /ppfahn-68/ 2024-03-04T18:50:36.817 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM= 2024-03-04T18:50:36.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM=', 'Content-Length': '0'} 2024-03-04T18:50:36.822 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aaca58dd69f9c4e7-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:50:36.822 DEBUG:boto:path=/ppfahn-68/d 2024-03-04T18:50:36.822 DEBUG:boto:auth_path=/ppfahn-68/d 2024-03-04T18:50:36.822 DEBUG:boto:Method: PUT 2024-03-04T18:50:36.822 DEBUG:boto:Path: /ppfahn-68/d 2024-03-04T18:50:36.822 DEBUG:boto:Data: 2024-03-04T18:50:36.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T18:50:36.823 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:50:36.823 DEBUG:boto:Port: 8000 2024-03-04T18:50:36.823 DEBUG:boto:Params: {} 2024-03-04T18:50:36.823 DEBUG:boto:Token: None 2024-03-04T18:50:36.823 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 18:50:36 GMT /ppfahn-68/d 2024-03-04T18:50:36.823 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:gMIz82FDScZg4Ol5/AyHd4Pc5PY= 2024-03-04T18:50:36.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gMIz82FDScZg4Ol5/AyHd4Pc5PY='} 2024-03-04T18:50:36.868 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d8546a99bb18633-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:50:36.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:36.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:36.869 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:36.929 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:36.929 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:36.940 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:50:36.937+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:50:36.944 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:50:36.941+0000 7f5c2310a640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T18:50:36.945 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:50:36.941+0000 7f5c2a919640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T18:50:37.107 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T18:50:37.140 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709578236.710082_611.1","last_update":"2024-03-04T18:50:36.710082Z"}] 2024-03-04T18:50:37.140 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:50:37.140 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:50:37.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:37.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:37.141 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:37.184 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:37.184 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:37.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:37.267+0000 7fda6d96a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:37.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:37.267+0000 7fda6d96a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:37.299 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:37Z"} 2024-03-04T18:50:37.299 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:37Z"} 2024-03-04T18:50:37.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:50:37.300 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:50:37.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:50:37.300 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:50:42.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:42.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:42.302 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:42.347 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:42.347 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:42.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:42.447+0000 7febbe2c3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:42.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:42.451+0000 7febbe2c3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:42.488 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:42Z"} 2024-03-04T18:50:42.488 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:42Z"} 2024-03-04T18:50:42.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:50:42.488 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:50:42.489 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:50:42.489 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T18:50:47.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:47.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:47.490 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:47.534 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:47.534 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:47.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:47.591+0000 7f30b7545980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:47.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:47.595+0000 7f30b7545980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:47.623 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:47Z"} 2024-03-04T18:50:47.624 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:47Z"} 2024-03-04T18:50:47.624 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:50:47.624 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:50:47.624 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:50:47.624 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T18:50:52.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:52.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:52.625 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:52.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:52.799+0000 7fd97be9f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:52.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:52.799+0000 7fd97be9f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:52Z"} 2024-03-04T18:50:52.835 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:52Z"} 2024-03-04T18:50:52.836 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:50:52.836 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:50:52.836 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:50:52.836 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:50:57.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:57.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:57.837 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:57.881 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:57.881 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:57.939+0000 7fd625dfa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:50:57.939+0000 7fd625dfa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:57.971 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:57Z"} 2024-03-04T18:50:57.971 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:57Z"} 2024-03-04T18:50:57.972 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:50:57.972 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:50:57.972 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:50:57.972 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:51:02.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:02.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:02.974 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:03.017 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:03.017 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:03.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:03.083+0000 7f7e43c6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:03.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:03.083+0000 7f7e43c6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:03.118 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:03Z"} 2024-03-04T18:51:03.119 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:03Z"} 2024-03-04T18:51:03.119 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:51:03.119 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:51:03.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:51:03.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:51:08.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:08.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:08.121 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:08.164 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:08.164 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:08.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:08.227+0000 7fa635182980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:08.227+0000 7fa635182980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:08.254 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:08Z"} 2024-03-04T18:51:08.255 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:08Z"} 2024-03-04T18:51:08.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:51:08.255 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:51:08.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:51:08.255 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:51:13.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:13.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:13.257 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:13.300 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:13.300 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:13.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:13.375+0000 7fa39ba51980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:13.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:13.375+0000 7fa39ba51980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:13.403 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:13Z"} 2024-03-04T18:51:13.403 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:13Z"} 2024-03-04T18:51:13.403 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:51:13.404 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:51:13.404 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:51:13.404 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:51:18.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:18.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:18.406 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:18.450 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:18.450 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:18.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:18.518+0000 7fbbecd25980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:18.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:18.522+0000 7fbbecd25980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:18.555 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.480321_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.480321Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:18Z"} 2024-03-04T18:51:18.555 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.480321_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.480321Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:18Z"} 2024-03-04T18:51:18.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.480321_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.480321Z', 'realm_epoch': 2}}] 2024-03-04T18:51:18.556 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:51:18.556 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.480321_315.1'} 2024-03-04T18:51:18.556 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.480321_315.1 2024-03-04T18:51:23.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:23.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:23.558 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:23.601 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:23.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:23.682+0000 7f8721281980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:23.682+0000 7f8721281980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:23.707 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:23Z"} 2024-03-04T18:51:23.707 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:23Z"} 2024-03-04T18:51:23.708 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:51:23.708 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:51:23.708 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:51:23.708 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:51:26.486 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:51:26.484+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:51:28.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:28.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:28.709 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:28.752 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:28.753 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:28.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:28.818+0000 7fa3ac5fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:28.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:28.818+0000 7fa3ac5fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:28.844 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:28Z"} 2024-03-04T18:51:28.844 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:28Z"} 2024-03-04T18:51:28.845 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:51:28.845 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:51:28.845 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:51:28.845 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.310269_305.1 2024-03-04T18:51:33.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:33.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:33.847 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:33.888 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:33.888 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:33.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:33.966+0000 7fb404344980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:33.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:33.970+0000 7fb404344980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:33.998 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:33Z"} 2024-03-04T18:51:33.998 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:33Z"} 2024-03-04T18:51:33.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:51:33.998 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:51:33.999 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:51:33.999 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T18:51:38.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:39.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:39.000 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:39.043 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:39.043 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:39.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:39.114+0000 7fa3cb345980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:39.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:39.118+0000 7fa3cb345980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:39.150 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:39Z"} 2024-03-04T18:51:39.151 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:39Z"} 2024-03-04T18:51:39.151 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:51:39.151 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:51:39.151 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:51:39.151 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:51:44.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:44.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:44.153 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:44.196 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:44.196 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:44.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:44.266+0000 7f959aecc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:44.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:44.266+0000 7f959aecc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:44.296 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:44Z"} 2024-03-04T18:51:44.297 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:44Z"} 2024-03-04T18:51:44.297 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:51:44.297 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:51:44.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:51:44.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.310269_305.1 2024-03-04T18:51:49.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:49.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:49.298 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:49.340 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:49.340 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:49.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:49.410+0000 7fe0b5afb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:49.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:49.410+0000 7fe0b5afb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:49.440 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:49Z"} 2024-03-04T18:51:49.440 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:49Z"} 2024-03-04T18:51:49.440 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:51:49.440 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:51:49.440 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:51:49.440 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.310269_305.1 2024-03-04T18:51:54.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:54.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:54.442 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:54.484 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:54.484 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:54.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:54.546+0000 7f07e0764980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:54.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:54.550+0000 7f07e0764980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:54.580 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:54Z"} 2024-03-04T18:51:54.581 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:54Z"} 2024-03-04T18:51:54.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:51:54.581 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:51:54.581 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:51:54.581 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:51:59.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:59.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:59.584 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:59.626 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:59.626 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:59.686+0000 7f41e4ada980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:51:59.690+0000 7f41e4ada980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:59.717 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:59Z"} 2024-03-04T18:51:59.717 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:59Z"} 2024-03-04T18:51:59.717 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T18:51:59.718 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:51:59.718 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T18:51:59.718 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.380518_307.1 2024-03-04T18:52:04.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:04.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:04.719 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:04.762 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:04.762 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:04.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:04.830+0000 7ff1ee252980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:04.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:04.834+0000 7ff1ee252980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:04.861 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:04Z"} 2024-03-04T18:52:04.861 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:04Z"} 2024-03-04T18:52:04.861 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:52:04.861 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:52:04.861 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:52:04.861 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:52:09.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:09.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:09.863 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:09.906 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:09.906 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:09.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:09.974+0000 7f4b5c8c8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:09.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:09.974+0000 7f4b5c8c8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:10.000 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:09Z"} 2024-03-04T18:52:10.000 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:09Z"} 2024-03-04T18:52:10.000 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:52:10.001 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:52:10.001 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:52:10.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:52:15.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:15.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:15.003 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:15.046 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:15.047 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:15.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:15.110+0000 7f5b3c613980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:15.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:15.110+0000 7f5b3c613980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:15.138 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:15Z"} 2024-03-04T18:52:15.138 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:15Z"} 2024-03-04T18:52:15.138 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:52:15.139 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:52:15.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:52:15.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:52:20.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:20.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:20.140 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:20.184 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:20.185 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:20.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:20.261+0000 7f79eeede980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:20.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:20.261+0000 7f79eeede980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:20.296 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:20Z"} 2024-03-04T18:52:20.296 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:20Z"} 2024-03-04T18:52:20.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:52:20.296 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:52:20.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:52:20.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T18:52:25.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:25.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:25.298 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:25.348 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:25.348 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:25.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:25.409+0000 7f60c82d5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:25.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:25.413+0000 7f60c82d5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:25.438 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:25Z"} 2024-03-04T18:52:25.438 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:25Z"} 2024-03-04T18:52:25.439 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:52:25.439 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:52:25.439 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:52:25.439 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T18:52:30.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:30.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:30.440 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:30.485 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:30.485 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:30.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:30.557+0000 7f49106fd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:30.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:30.557+0000 7f49106fd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:30.586 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.815737_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.815737Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:30Z"} 2024-03-04T18:52:30.586 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.815737_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.815737Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:30Z"} 2024-03-04T18:52:30.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.815737_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.815737Z', 'realm_epoch': 2}}] 2024-03-04T18:52:30.586 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:52:30.586 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.815737_334.1'} 2024-03-04T18:52:30.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575077.815737_334.1 2024-03-04T18:52:35.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:35.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:35.588 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:35.630 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:35.630 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:35.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:35.689+0000 7fb5c7c6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:35.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:35.689+0000 7fb5c7c6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:35.718 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:35Z"} 2024-03-04T18:52:35.718 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:35Z"} 2024-03-04T18:52:35.719 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:52:35.719 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:52:35.719 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:52:35.719 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:52:40.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:40.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:40.721 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:40.763 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:40.764 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:40.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:40.837+0000 7fbf48b4d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:40.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:40.841+0000 7fbf48b4d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:40.874 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:40Z"} 2024-03-04T18:52:40.874 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:40Z"} 2024-03-04T18:52:40.874 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:52:40.874 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:52:40.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:52:40.875 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:52:45.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:45.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:45.876 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:45.919 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:45.919 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:45.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:45.985+0000 7f4d87e15980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:45.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:45.985+0000 7f4d87e15980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:46.012 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:45Z"} 2024-03-04T18:52:46.012 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:45Z"} 2024-03-04T18:52:46.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:52:46.012 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:52:46.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:52:46.013 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:52:51.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:51.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:51.014 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:51.056 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:51.056 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:51.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:51.129+0000 7f5cb7e61980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:51.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:51.129+0000 7f5cb7e61980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:51.170 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:51Z"} 2024-03-04T18:52:51.170 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:51Z"} 2024-03-04T18:52:51.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:52:51.171 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:52:51.171 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:52:51.171 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:52:56.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:56.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:56.172 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:56.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:56.281+0000 7f14a410b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:56.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:52:56.281+0000 7f14a410b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:56.317 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:56Z"} 2024-03-04T18:52:56.317 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:56Z"} 2024-03-04T18:52:56.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:52:56.318 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:52:56.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:52:56.318 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:53:01.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:01.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:01.320 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:01.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:01.441+0000 7fa7e9e8e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:01.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:01.441+0000 7fa7e9e8e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:01.477 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:01Z"} 2024-03-04T18:53:01.478 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:01Z"} 2024-03-04T18:53:01.478 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:53:01.478 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:53:01.478 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:53:01.478 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:53:06.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:06.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:06.480 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:06.526 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:06.526 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:06.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:06.613+0000 7f9bcc5aa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:06.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:06.613+0000 7f9bcc5aa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:06.683 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:06Z"} 2024-03-04T18:53:06.683 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:06Z"} 2024-03-04T18:53:06.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:53:06.683 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:53:06.683 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:53:06.683 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:53:10.036 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:53:10.034+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:53:11.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:11.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:11.685 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:11.727 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:11.727 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:11.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:11.797+0000 7f32e812a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:11.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:11.797+0000 7f32e812a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:11.824 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:11Z"} 2024-03-04T18:53:11.825 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:11Z"} 2024-03-04T18:53:11.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:53:11.825 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:53:11.825 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:53:11.825 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:53:16.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:16.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:16.826 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:16.870 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:16.870 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:16.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:16.933+0000 7f4da96b5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:16.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:16.933+0000 7f4da96b5980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:16.964 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:16Z"} 2024-03-04T18:53:16.964 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:16Z"} 2024-03-04T18:53:16.964 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:53:16.964 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:53:16.965 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:53:16.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:53:21.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:21.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:21.990 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:22.050 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:22.051 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:22.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:22.144+0000 7f4863d9e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:22.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:22.144+0000 7f4863d9e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:22.191 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:22Z"} 2024-03-04T18:53:22.193 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:22Z"} 2024-03-04T18:53:22.193 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:53:22.193 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:53:22.193 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:53:22.193 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:53:27.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:27.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:27.203 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:27.250 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:27.316+0000 7f00f925d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:27.316+0000 7f00f925d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:27.345 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:27Z"} 2024-03-04T18:53:27.345 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:27Z"} 2024-03-04T18:53:27.345 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:53:27.345 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:53:27.345 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:53:27.346 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:53:32.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:32.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:32.347 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:32.389 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:32.389 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:32.464+0000 7f7c01e8a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:32.464+0000 7f7c01e8a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:32.498 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:32Z"} 2024-03-04T18:53:32.498 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:32Z"} 2024-03-04T18:53:32.498 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:53:32.498 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:53:32.499 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:53:32.499 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:53:37.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:37.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:37.500 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:37.543 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:37.543 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:37.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:37.612+0000 7fbfee031980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:37.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:37.616+0000 7fbfee031980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:37.643 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:37Z"} 2024-03-04T18:53:37.643 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:37Z"} 2024-03-04T18:53:37.643 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:53:37.643 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:53:37.643 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:53:37.644 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:53:42.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:42.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:42.644 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:42.688 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:42.688 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:42.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:42.748+0000 7fb6abe7a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:42.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:42.748+0000 7fb6abe7a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:42.777 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:42Z"} 2024-03-04T18:53:42.777 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:42Z"} 2024-03-04T18:53:42.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.388355_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.388355Z', 'realm_epoch': 2}}] 2024-03-04T18:53:42.777 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:53:42.777 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.388355_309.1'} 2024-03-04T18:53:42.777 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.388355_309.1 2024-03-04T18:53:47.778 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:47.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:47.778 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:47.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:47.884+0000 7f589dd2c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:47.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:47.888+0000 7f589dd2c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:47.915 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:47Z"} 2024-03-04T18:53:47.915 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:47Z"} 2024-03-04T18:53:47.915 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:53:47.915 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:53:47.915 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:53:47.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:53:52.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:52.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:52.917 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:52.959 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:52.959 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:53.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:53.032+0000 7f696c812980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:53.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:53.032+0000 7f696c812980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:53.064 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:53Z"} 2024-03-04T18:53:53.064 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:53Z"} 2024-03-04T18:53:53.064 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:53:53.065 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:53:53.065 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:53:53.065 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:53:58.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:58.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:58.066 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:58.110 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:58.110 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:58.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:58.184+0000 7f7257aec980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:58.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:53:58.184+0000 7f7257aec980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:58.215 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:58Z"} 2024-03-04T18:53:58.216 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:58Z"} 2024-03-04T18:53:58.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:53:58.216 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:53:58.216 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:53:58.216 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:54:03.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:03.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:03.218 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:03.259 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:03.259 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:03.336+0000 7f3339372980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:03.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:03.336+0000 7f3339372980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:03.364 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:03Z"} 2024-03-04T18:54:03.364 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:03Z"} 2024-03-04T18:54:03.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:54:03.365 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:54:03.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:54:03.365 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:54:08.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:08.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:08.366 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:08.411 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:08.411 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:08.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:08.472+0000 7fbc80ee7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:08.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:08.472+0000 7fbc80ee7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:08.500 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:08Z"} 2024-03-04T18:54:08.501 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:08Z"} 2024-03-04T18:54:08.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:54:08.501 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:54:08.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:54:08.501 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:54:13.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:13.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:13.503 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:13.545 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:13.545 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:13.604+0000 7f7f46070980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:13.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:13.604+0000 7f7f46070980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:13.635 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:13Z"} 2024-03-04T18:54:13.635 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:13Z"} 2024-03-04T18:54:13.635 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:54:13.635 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:54:13.636 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:54:13.636 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:54:18.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:18.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:18.637 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:18.690 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:18.690 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:18.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:18.751+0000 7f615268a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:18.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:18.755+0000 7f615268a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:18.783 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:18Z"} 2024-03-04T18:54:18.783 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:18Z"} 2024-03-04T18:54:18.783 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:54:18.783 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:54:18.783 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:54:18.783 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T18:54:23.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:23.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:23.785 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:23.828 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:23.828 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:23.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:23.899+0000 7f5c2169e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:23.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:23.899+0000 7f5c2169e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:23.930 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:23Z"} 2024-03-04T18:54:23.930 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:23Z"} 2024-03-04T18:54:23.930 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.315397_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.315397Z', 'realm_epoch': 2}}] 2024-03-04T18:54:23.930 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:54:23.931 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.315397_306.1'} 2024-03-04T18:54:23.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.315397_306.1 2024-03-04T18:54:28.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:28.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:28.932 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:28.974 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:28.975 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:29.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:29.087+0000 7f827d637980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:29.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:29.091+0000 7f827d637980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:29.119 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:29Z"} 2024-03-04T18:54:29.119 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:29Z"} 2024-03-04T18:54:29.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:54:29.120 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:54:29.120 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:54:29.120 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T18:54:34.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:34.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:34.121 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:34.164 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:34.165 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:34.231+0000 7fe0da3d7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:34.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:34.235+0000 7fe0da3d7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:34.266 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:34Z"} 2024-03-04T18:54:34.266 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:34Z"} 2024-03-04T18:54:34.266 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:54:34.266 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:54:34.266 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:54:34.266 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T18:54:39.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:39.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:39.268 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:39.311 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:39.311 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:39.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:39.375+0000 7fbcdc130980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:39.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:39.379+0000 7fbcdc130980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:39.423 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:39Z"} 2024-03-04T18:54:39.423 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:39Z"} 2024-03-04T18:54:39.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:54:39.423 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:54:39.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:54:39.424 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.310269_305.1 2024-03-04T18:54:44.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:44.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:44.426 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:44.468 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:44.468 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:44.539+0000 7fba83e16980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:44.539+0000 7fba83e16980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:44.568 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:44Z"} 2024-03-04T18:54:44.568 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:44Z"} 2024-03-04T18:54:44.569 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:54:44.569 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:54:44.569 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:54:44.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:54:49.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:49.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:49.571 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:49.614 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:49.614 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:49.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:49.683+0000 7f6006596980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:49.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:49.683+0000 7f6006596980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:49.713 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:49Z"} 2024-03-04T18:54:49.713 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:49Z"} 2024-03-04T18:54:49.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:54:49.713 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:54:49.713 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:54:49.713 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:54:54.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:54.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:54.714 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:54.835+0000 7f8b68baa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:54.839+0000 7f8b68baa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:54.873 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:54Z"} 2024-03-04T18:54:54.873 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:54Z"} 2024-03-04T18:54:54.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:54:54.873 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:54:54.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:54:54.873 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:54:59.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:59.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:59.875 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:59.917 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:59.918 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:59.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:59.983+0000 7f8576e3f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:59.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:54:59.983+0000 7f8576e3f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:00.015 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:59Z"} 2024-03-04T18:55:00.015 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:59Z"} 2024-03-04T18:55:00.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.495723_318.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.495723Z', 'realm_epoch': 2}}] 2024-03-04T18:55:00.016 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:55:00.016 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.495723_318.1'} 2024-03-04T18:55:00.016 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.495723_318.1 2024-03-04T18:55:05.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:05.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:05.018 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:05.072 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:05.072 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:05.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:05.159+0000 7f23814dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:05.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:05.159+0000 7f23814dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:05.184 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:05Z"} 2024-03-04T18:55:05.184 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:05Z"} 2024-03-04T18:55:05.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.495723_318.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.495723Z', 'realm_epoch': 2}}] 2024-03-04T18:55:05.184 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:55:05.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.495723_318.1'} 2024-03-04T18:55:05.185 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.495723_318.1 2024-03-04T18:55:10.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:10.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:10.186 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:10.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:10.303+0000 7fb95eef0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:10.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:10.307+0000 7fb95eef0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:10.336 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:10Z"} 2024-03-04T18:55:10.337 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:10Z"} 2024-03-04T18:55:10.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:55:10.337 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:55:10.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:55:10.337 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.310269_305.1 2024-03-04T18:55:15.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:15.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:15.338 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:15.387 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:15.387 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:15.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:15.467+0000 7fdce1a22980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:15.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:15.467+0000 7fdce1a22980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:15.497 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:15Z"} 2024-03-04T18:55:15.497 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:15Z"} 2024-03-04T18:55:15.497 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:55:15.497 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:55:15.497 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:55:15.497 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:55:20.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:20.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:20.499 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:20.541 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:20.541 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:20.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:20.610+0000 7ff12c91d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:20.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:20.610+0000 7ff12c91d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:20.639 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:20Z"} 2024-03-04T18:55:20.639 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:20Z"} 2024-03-04T18:55:20.639 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:55:20.639 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:55:20.640 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:55:20.640 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T18:55:25.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:25.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:25.641 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:25.683 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:25.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:25.766+0000 7f5547c6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:25.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:25.766+0000 7f5547c6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:25.795 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:25Z"} 2024-03-04T18:55:25.795 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:25Z"} 2024-03-04T18:55:25.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:55:25.795 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:55:25.795 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:55:25.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:55:30.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:30.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:30.797 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:30.841 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:30.841 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:30.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:30.906+0000 7f73cf444980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:30.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:30.906+0000 7f73cf444980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:30.933 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:30Z"} 2024-03-04T18:55:30.933 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:30Z"} 2024-03-04T18:55:30.933 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:55:30.933 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:55:30.933 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:55:30.933 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:55:35.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:35.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:35.935 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:35.979 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:35.979 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:36.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:36.066+0000 7f958812f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:36.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:36.066+0000 7f958812f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:36.107 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:36Z"} 2024-03-04T18:55:36.107 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:36Z"} 2024-03-04T18:55:36.107 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:55:36.108 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:55:36.108 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:55:36.108 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:55:41.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:41.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:41.109 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:41.152 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:41.152 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:41.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:41.226+0000 7f5b9fff7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:41.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:41.226+0000 7f5b9fff7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:41.260 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:41Z"} 2024-03-04T18:55:41.261 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:41Z"} 2024-03-04T18:55:41.261 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:55:41.261 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T18:55:41.261 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:55:41.261 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T18:55:46.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2024-03-04T18:55:46.265 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ppfahn-69 2024-03-04T18:55:46.266 DEBUG:boto:path=/ppfahn-69/ 2024-03-04T18:55:46.266 DEBUG:boto:auth_path=/ppfahn-69/ 2024-03-04T18:55:46.266 DEBUG:boto:Method: PUT 2024-03-04T18:55:46.266 DEBUG:boto:Path: /ppfahn-69/ 2024-03-04T18:55:46.266 DEBUG:boto:Data: 2024-03-04T18:55:46.266 DEBUG:boto:Headers: {} 2024-03-04T18:55:46.266 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:55:46.267 DEBUG:boto:Port: 8000 2024-03-04T18:55:46.267 DEBUG:boto:Params: {} 2024-03-04T18:55:46.267 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:55:46.267 DEBUG:boto:Token: None 2024-03-04T18:55:46.267 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:55:46 GMT /ppfahn-69/ 2024-03-04T18:55:46.268 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:8WQKpOZYRNx3NiC8PqYWg6cvHMk= 2024-03-04T18:55:46.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8WQKpOZYRNx3NiC8PqYWg6cvHMk=', 'Content-Length': '0'} 2024-03-04T18:55:46.310 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e2cbec3a68d7b188-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:55:46.310 DEBUG:boto:path=/ppfahn-69/ 2024-03-04T18:55:46.310 DEBUG:boto:auth_path=/ppfahn-69/ 2024-03-04T18:55:46.310 DEBUG:boto:Method: HEAD 2024-03-04T18:55:46.310 DEBUG:boto:Path: /ppfahn-69/ 2024-03-04T18:55:46.311 DEBUG:boto:Data: 2024-03-04T18:55:46.311 DEBUG:boto:Headers: {} 2024-03-04T18:55:46.311 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:55:46.311 DEBUG:boto:Port: 8000 2024-03-04T18:55:46.311 DEBUG:boto:Params: {} 2024-03-04T18:55:46.311 DEBUG:boto:Token: None 2024-03-04T18:55:46.311 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:55:46 GMT /ppfahn-69/ 2024-03-04T18:55:46.312 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw= 2024-03-04T18:55:46.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw=', 'Content-Length': '0'} 2024-03-04T18:55:46.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038f809046450a688-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:55:46.314 DEBUG:boto:path=/ppfahn-69/a 2024-03-04T18:55:46.314 DEBUG:boto:auth_path=/ppfahn-69/a 2024-03-04T18:55:46.314 DEBUG:boto:Method: PUT 2024-03-04T18:55:46.314 DEBUG:boto:Path: /ppfahn-69/a 2024-03-04T18:55:46.315 DEBUG:boto:Data: 2024-03-04T18:55:46.315 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T18:55:46.315 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:55:46.315 DEBUG:boto:Port: 8000 2024-03-04T18:55:46.315 DEBUG:boto:Params: {} 2024-03-04T18:55:46.315 DEBUG:boto:Token: None 2024-03-04T18:55:46.315 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 18:55:46 GMT /ppfahn-69/a 2024-03-04T18:55:46.315 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:9bcaI/Chx84lFhFBIk4WB6QGQ+U= 2024-03-04T18:55:46.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9bcaI/Chx84lFhFBIk4WB6QGQ+U='} 2024-03-04T18:55:46.325 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000364316d504145386-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:55:46.325 DEBUG:boto:path=/ppfahn-69/ 2024-03-04T18:55:46.326 DEBUG:boto:auth_path=/ppfahn-69/ 2024-03-04T18:55:46.326 DEBUG:boto:Method: HEAD 2024-03-04T18:55:46.326 DEBUG:boto:Path: /ppfahn-69/ 2024-03-04T18:55:46.326 DEBUG:boto:Data: 2024-03-04T18:55:46.326 DEBUG:boto:Headers: {} 2024-03-04T18:55:46.326 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:55:46.326 DEBUG:boto:Port: 8000 2024-03-04T18:55:46.326 DEBUG:boto:Params: {} 2024-03-04T18:55:46.326 DEBUG:boto:Token: None 2024-03-04T18:55:46.327 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:55:46 GMT /ppfahn-69/ 2024-03-04T18:55:46.327 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw= 2024-03-04T18:55:46.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw=', 'Content-Length': '0'} 2024-03-04T18:55:46.329 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f91bcdd13c80db0-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:55:46.329 DEBUG:boto:path=/ppfahn-69/b 2024-03-04T18:55:46.329 DEBUG:boto:auth_path=/ppfahn-69/b 2024-03-04T18:55:46.329 DEBUG:boto:Method: PUT 2024-03-04T18:55:46.330 DEBUG:boto:Path: /ppfahn-69/b 2024-03-04T18:55:46.330 DEBUG:boto:Data: 2024-03-04T18:55:46.330 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T18:55:46.330 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:55:46.330 DEBUG:boto:Port: 8000 2024-03-04T18:55:46.330 DEBUG:boto:Params: {} 2024-03-04T18:55:46.330 DEBUG:boto:Token: None 2024-03-04T18:55:46.330 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 18:55:46 GMT /ppfahn-69/b 2024-03-04T18:55:46.330 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ITJCtU7GlSkhhj4Fq/nClFfNlOk= 2024-03-04T18:55:46.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ITJCtU7GlSkhhj4Fq/nClFfNlOk='} 2024-03-04T18:55:46.372 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b63e02fce9771c41-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:55:46.372 DEBUG:boto:path=/ppfahn-69/ 2024-03-04T18:55:46.372 DEBUG:boto:auth_path=/ppfahn-69/ 2024-03-04T18:55:46.373 DEBUG:boto:Method: HEAD 2024-03-04T18:55:46.373 DEBUG:boto:Path: /ppfahn-69/ 2024-03-04T18:55:46.373 DEBUG:boto:Data: 2024-03-04T18:55:46.373 DEBUG:boto:Headers: {} 2024-03-04T18:55:46.373 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:55:46.373 DEBUG:boto:Port: 8000 2024-03-04T18:55:46.373 DEBUG:boto:Params: {} 2024-03-04T18:55:46.373 DEBUG:boto:Token: None 2024-03-04T18:55:46.373 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:55:46 GMT /ppfahn-69/ 2024-03-04T18:55:46.373 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw= 2024-03-04T18:55:46.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw=', 'Content-Length': '0'} 2024-03-04T18:55:46.377 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a0e7b32313b32f2-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:55:46.377 DEBUG:boto:path=/ppfahn-69/c 2024-03-04T18:55:46.377 DEBUG:boto:auth_path=/ppfahn-69/c 2024-03-04T18:55:46.378 DEBUG:boto:Method: PUT 2024-03-04T18:55:46.378 DEBUG:boto:Path: /ppfahn-69/c 2024-03-04T18:55:46.378 DEBUG:boto:Data: 2024-03-04T18:55:46.378 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T18:55:46.378 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:55:46.378 DEBUG:boto:Port: 8000 2024-03-04T18:55:46.378 DEBUG:boto:Params: {} 2024-03-04T18:55:46.378 DEBUG:boto:Token: None 2024-03-04T18:55:46.378 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 18:55:46 GMT /ppfahn-69/c 2024-03-04T18:55:46.378 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:ptd7ofyGw0FFbHOXwMjSH9j9G+M= 2024-03-04T18:55:46.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ptd7ofyGw0FFbHOXwMjSH9j9G+M='} 2024-03-04T18:55:46.387 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c4e53722359d069-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:55:46.388 DEBUG:boto:path=/ppfahn-69/ 2024-03-04T18:55:46.388 DEBUG:boto:auth_path=/ppfahn-69/ 2024-03-04T18:55:46.388 DEBUG:boto:Method: HEAD 2024-03-04T18:55:46.388 DEBUG:boto:Path: /ppfahn-69/ 2024-03-04T18:55:46.388 DEBUG:boto:Data: 2024-03-04T18:55:46.388 DEBUG:boto:Headers: {} 2024-03-04T18:55:46.388 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:55:46.388 DEBUG:boto:Port: 8000 2024-03-04T18:55:46.388 DEBUG:boto:Params: {} 2024-03-04T18:55:46.389 DEBUG:boto:Token: None 2024-03-04T18:55:46.389 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 18:55:46 GMT /ppfahn-69/ 2024-03-04T18:55:46.389 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw= 2024-03-04T18:55:46.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw=', 'Content-Length': '0'} 2024-03-04T18:55:46.393 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f97bffdc91c99d47-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:55:46.393 DEBUG:boto:path=/ppfahn-69/d 2024-03-04T18:55:46.393 DEBUG:boto:auth_path=/ppfahn-69/d 2024-03-04T18:55:46.393 DEBUG:boto:Method: PUT 2024-03-04T18:55:46.393 DEBUG:boto:Path: /ppfahn-69/d 2024-03-04T18:55:46.393 DEBUG:boto:Data: 2024-03-04T18:55:46.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T18:55:46.394 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T18:55:46.394 DEBUG:boto:Port: 8000 2024-03-04T18:55:46.394 DEBUG:boto:Params: {} 2024-03-04T18:55:46.394 DEBUG:boto:Token: None 2024-03-04T18:55:46.394 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 18:55:46 GMT /ppfahn-69/d 2024-03-04T18:55:46.394 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:y1rK82aw6Y8Dm+aRJNhUZZxmi/0= 2024-03-04T18:55:46.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:y1rK82aw6Y8Dm+aRJNhUZZxmi/0='} 2024-03-04T18:55:46.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034a6e775a0b0c1c4-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:55:46.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:46.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:46.437 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:46.485 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:46.485 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:46.697 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T18:55:46.726 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709578546.310901_628.1","last_update":"2024-03-04T18:55:46.310901Z"}] 2024-03-04T18:55:46.727 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:55:46.727 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:55:46.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:46.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:46.727 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:46.770 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:46.770 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:46.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:46.830+0000 7f88504aa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:46.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:46.834+0000 7f88504aa980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:46.860 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:46Z"} 2024-03-04T18:55:46.860 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:46Z"} 2024-03-04T18:55:46.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:55:46.861 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:55:46.861 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:55:46.861 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:55:48.872 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:55:48.870+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:55:48.872 INFO:tasks.rgw.c1.client.1.smithi136.stdout:2024-03-04T18:55:48.870+0000 7f5c22108640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T18:55:51.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:51.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:51.862 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:51.905 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:51.905 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:51.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:51.982+0000 7fe81348a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:51.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:51.982+0000 7fe81348a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:52.016 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:51Z"} 2024-03-04T18:55:52.016 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:51Z"} 2024-03-04T18:55:52.017 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:55:52.017 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:55:52.017 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:55:52.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T18:55:57.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:57.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:57.019 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:57.061 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:57.061 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:57.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:57.134+0000 7f8b4b5bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:57.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:55:57.134+0000 7f8b4b5bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:57.161 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:57Z"} 2024-03-04T18:55:57.161 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:57Z"} 2024-03-04T18:55:57.161 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T18:55:57.162 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:55:57.162 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T18:55:57.162 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.380518_307.1 2024-03-04T18:56:02.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:02.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:02.163 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:02.205 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:02.205 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:02.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:02.278+0000 7f3173b7a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:02.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:02.278+0000 7f3173b7a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:02.313 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:02Z"} 2024-03-04T18:56:02.313 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:02Z"} 2024-03-04T18:56:02.313 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:56:02.313 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:56:02.314 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:56:02.314 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:56:07.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:07.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:07.315 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:07.359 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:07.359 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:07.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:07.430+0000 7fcb1d0b7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:07.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:07.434+0000 7fcb1d0b7980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:07.461 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:07Z"} 2024-03-04T18:56:07.461 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:07Z"} 2024-03-04T18:56:07.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:56:07.462 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:56:07.462 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:56:07.462 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:56:12.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:12.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:12.463 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:12.505 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:12.505 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:12.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:12.570+0000 7fb4b5e60980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:12.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:12.570+0000 7fb4b5e60980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:12.597 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:12Z"} 2024-03-04T18:56:12.597 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:12Z"} 2024-03-04T18:56:12.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:56:12.598 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:56:12.598 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:56:12.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:56:17.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:17.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:17.600 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:17.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:17.710+0000 7ff5ba602980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:17.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:17.710+0000 7ff5ba602980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:17.742 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:17Z"} 2024-03-04T18:56:17.742 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:17Z"} 2024-03-04T18:56:17.742 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:56:17.742 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:56:17.742 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:56:17.742 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T18:56:22.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:22.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:22.744 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:22.786 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:22.786 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:22.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:22.845+0000 7f0fd1af8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:22.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:22.849+0000 7f0fd1af8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:22.878 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:22Z"} 2024-03-04T18:56:22.878 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:22Z"} 2024-03-04T18:56:22.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:56:22.879 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:56:22.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:56:22.879 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:56:26.527 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:56:26.526+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:56:27.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:27.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:27.880 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:27.933 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:27.933 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:28.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:28.005+0000 7f15ce02e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:28.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:28.009+0000 7f15ce02e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:28.035 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:28Z"} 2024-03-04T18:56:28.035 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:28Z"} 2024-03-04T18:56:28.035 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:56:28.035 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:56:28.035 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:56:28.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:56:33.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:33.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:33.037 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:33.080 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:33.080 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:33.141+0000 7fc91f107980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:33.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:33.141+0000 7fc91f107980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:33.176 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:33Z"} 2024-03-04T18:56:33.176 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:33Z"} 2024-03-04T18:56:33.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:56:33.177 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:56:33.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:56:33.177 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:56:38.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:38.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:38.179 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:38.223 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:38.223 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:38.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:38.301+0000 7f13588b3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:38.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:38.301+0000 7f13588b3980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:38.331 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:38Z"} 2024-03-04T18:56:38.331 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:38Z"} 2024-03-04T18:56:38.331 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:56:38.331 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:56:38.331 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:56:38.331 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T18:56:43.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:43.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:43.333 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:43.377 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:43.377 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:43.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:43.449+0000 7f48ea65f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:43.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:43.449+0000 7f48ea65f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:43.478 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:43Z"} 2024-03-04T18:56:43.479 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:43Z"} 2024-03-04T18:56:43.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:56:43.479 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:56:43.479 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:56:43.479 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:56:48.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:48.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:48.480 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:48.524 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:48.524 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:48.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:48.589+0000 7f05345b0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:48.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:48.589+0000 7f05345b0980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:48.618 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:48Z"} 2024-03-04T18:56:48.619 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:48Z"} 2024-03-04T18:56:48.619 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:56:48.619 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:56:48.619 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:56:48.619 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:56:53.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:53.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:53.622 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:53.665 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:53.665 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:53.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:53.749+0000 7f1c9fcdb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:53.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:53.749+0000 7f1c9fcdb980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:53.777 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:53Z"} 2024-03-04T18:56:53.777 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:53Z"} 2024-03-04T18:56:53.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:56:53.777 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:56:53.777 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:56:53.777 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:56:58.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:58.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:58.781 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:58.824 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:58.825 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:58.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:58.897+0000 7f663f958980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:58.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:56:58.897+0000 7f663f958980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:58.926 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:58Z"} 2024-03-04T18:56:58.926 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:58Z"} 2024-03-04T18:56:58.926 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:56:58.927 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:56:58.927 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:56:58.927 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T18:57:03.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:03.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:03.929 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:03.971 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:03.971 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:04.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:04.033+0000 7f0407a04980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:04.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:04.037+0000 7f0407a04980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:04.070 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:04Z"} 2024-03-04T18:57:04.070 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:04Z"} 2024-03-04T18:57:04.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:57:04.070 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:57:04.071 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:57:04.071 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T18:57:09.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:09.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:09.072 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:09.114 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:09.114 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:09.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:09.185+0000 7f2fd0c87980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:09.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:09.185+0000 7f2fd0c87980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:09.212 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:09Z"} 2024-03-04T18:57:09.212 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:09Z"} 2024-03-04T18:57:09.212 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:57:09.213 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:57:09.213 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:57:09.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:57:14.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:14.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:14.214 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:14.257 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:14.257 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:14.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:14.341+0000 7fec65906980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:14.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:14.341+0000 7fec65906980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:14.369 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:14Z"} 2024-03-04T18:57:14.370 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:14Z"} 2024-03-04T18:57:14.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:57:14.370 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:57:14.370 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:57:14.370 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:57:19.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:19.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:19.372 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:19.414 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:19.414 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:19.489+0000 7fcb9c778980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:19.493+0000 7fcb9c778980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:19.526 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:19Z"} 2024-03-04T18:57:19.526 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:19Z"} 2024-03-04T18:57:19.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:57:19.527 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:57:19.527 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:57:19.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:57:24.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:24.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:24.528 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:24.571 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:24.571 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:24.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:24.640+0000 7f8dc8b32980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:24.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:24.640+0000 7f8dc8b32980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:24.671 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:24Z"} 2024-03-04T18:57:24.671 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:24Z"} 2024-03-04T18:57:24.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:57:24.671 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:57:24.671 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:57:24.671 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:57:29.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:29.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:29.673 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:29.719 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:29.719 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:29.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:29.788+0000 7f56f4427980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:29.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:29.792+0000 7f56f4427980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:29.828 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:29Z"} 2024-03-04T18:57:29.828 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:29Z"} 2024-03-04T18:57:29.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:57:29.829 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:57:29.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:57:29.829 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T18:57:34.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:34.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:34.831 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:34.874 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:34.874 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:34.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:34.948+0000 7fb8e7f91980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:34.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:34.948+0000 7fb8e7f91980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:34.976 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:34Z"} 2024-03-04T18:57:34.976 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:34Z"} 2024-03-04T18:57:34.976 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:57:34.977 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:57:34.977 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:57:34.977 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:57:39.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:39.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:39.978 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:40.020 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:40.020 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:40.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:40.080+0000 7f578cf3a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:40.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:40.080+0000 7f578cf3a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:40.109 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:40Z"} 2024-03-04T18:57:40.109 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:40Z"} 2024-03-04T18:57:40.110 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:57:40.110 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:57:40.110 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:57:40.110 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T18:57:45.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:45.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:45.111 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:45.155 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:45.156 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:45.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:45.236+0000 7f7cecada980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:45.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:45.236+0000 7f7cecada980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:45.273 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:45Z"} 2024-03-04T18:57:45.273 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:45Z"} 2024-03-04T18:57:45.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.315397_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.315397Z', 'realm_epoch': 2}}] 2024-03-04T18:57:45.273 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:57:45.273 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.315397_306.1'} 2024-03-04T18:57:45.273 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.315397_306.1 2024-03-04T18:57:50.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:50.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:50.275 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:50.319 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:50.319 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:50.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:50.412+0000 7f5eb344a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:50.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:50.416+0000 7f5eb344a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:50.452 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:50Z"} 2024-03-04T18:57:50.452 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:50Z"} 2024-03-04T18:57:50.452 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:57:50.452 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:57:50.452 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:57:50.453 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:57:55.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:55.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:55.454 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:55.497 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:55.497 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:55.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:55.576+0000 7f712a9f8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:55.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:57:55.580+0000 7f712a9f8980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:55.610 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:55Z"} 2024-03-04T18:57:55.610 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:55Z"} 2024-03-04T18:57:55.610 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:57:55.610 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:57:55.610 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:57:55.610 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:58:00.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:00.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:00.612 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:00.656 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:00.656 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:00.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:00.724+0000 7ff37ebba980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:00.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:00.724+0000 7ff37ebba980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:00.761 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:00Z"} 2024-03-04T18:58:00.761 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:00Z"} 2024-03-04T18:58:00.761 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:58:00.761 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:58:00.761 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:58:00.762 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:58:05.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:05.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:05.763 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:05.806 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:05.806 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:05.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:05.876+0000 7f9c5bf9a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:05.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:05.880+0000 7f9c5bf9a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:05.909 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:05Z"} 2024-03-04T18:58:05.910 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:05Z"} 2024-03-04T18:58:05.910 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:58:05.910 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:58:05.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:58:05.910 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:58:10.098 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T18:58:10.099+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T18:58:10.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:10.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:10.912 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:10.955 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:10.955 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:11.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:11.020+0000 7f150462d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:11.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:11.020+0000 7f150462d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:11.048 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:11Z"} 2024-03-04T18:58:11.048 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:11Z"} 2024-03-04T18:58:11.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575076.582500_321.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:56.582500Z', 'realm_epoch': 2}}] 2024-03-04T18:58:11.048 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:58:11.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575076.582500_321.1'} 2024-03-04T18:58:11.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575076.582500_321.1 2024-03-04T18:58:16.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:16.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:16.050 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:16.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:16.156+0000 7f4d9ef9c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:16.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:16.156+0000 7f4d9ef9c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:16.208 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:16Z"} 2024-03-04T18:58:16.209 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:16Z"} 2024-03-04T18:58:16.209 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575076.582500_321.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:56.582500Z', 'realm_epoch': 2}}] 2024-03-04T18:58:16.209 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:58:16.209 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575076.582500_321.1'} 2024-03-04T18:58:16.209 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575076.582500_321.1 2024-03-04T18:58:21.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:21.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:21.210 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:21.254 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:21.254 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:21.311+0000 7fb223213980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:21.311+0000 7fb223213980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:21.343 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:21Z"} 2024-03-04T18:58:21.343 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:21Z"} 2024-03-04T18:58:21.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:58:21.344 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:58:21.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:58:21.344 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:58:26.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:26.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:26.345 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:26.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:26.471+0000 7f070b3a1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:26.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:26.471+0000 7f070b3a1980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:26.505 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:26Z"} 2024-03-04T18:58:26.547 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:26Z"} 2024-03-04T18:58:26.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:58:26.547 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:58:26.547 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:58:26.547 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:58:31.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:31.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:31.548 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:31.591 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:31.591 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:31.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:31.655+0000 7f7a05446980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:31.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:31.659+0000 7f7a05446980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:31.697 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:31Z"} 2024-03-04T18:58:31.697 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:31Z"} 2024-03-04T18:58:31.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:58:31.697 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:58:31.697 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:58:31.698 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:58:36.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:36.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:36.699 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:36.741 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:36.741 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:36.807+0000 7fdde78e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:36.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:36.811+0000 7fdde78e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:36.840 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:36Z"} 2024-03-04T18:58:36.840 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:36Z"} 2024-03-04T18:58:36.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:58:36.840 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:58:36.840 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:58:36.840 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T18:58:41.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:41.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:41.842 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:41.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:41.959+0000 7f8583c38980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:41.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:41.959+0000 7f8583c38980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:41.988 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:41Z"} 2024-03-04T18:58:41.988 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:41Z"} 2024-03-04T18:58:41.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T18:58:41.989 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:58:41.989 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T18:58:41.989 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.380518_307.1 2024-03-04T18:58:46.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:46.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:46.990 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:47.033 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:47.033 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:47.091+0000 7fed6ed36980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:47.095+0000 7fed6ed36980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:47.133 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:47Z"} 2024-03-04T18:58:47.134 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:47Z"} 2024-03-04T18:58:47.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:58:47.134 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:58:47.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:58:47.134 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:58:52.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:52.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:52.136 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:52.179 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:52.179 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:52.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:52.255+0000 7fb4c7b4c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:52.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:52.255+0000 7fb4c7b4c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:52.284 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:52Z"} 2024-03-04T18:58:52.284 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:52Z"} 2024-03-04T18:58:52.285 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:58:52.285 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:58:52.285 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:58:52.285 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T18:58:57.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:57.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:57.296 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:57.340 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:57.340 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:57.407+0000 7f7da822b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:58:57.407+0000 7f7da822b980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:57.438 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:57Z"} 2024-03-04T18:58:57.439 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:57Z"} 2024-03-04T18:58:57.439 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575076.582500_321.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:56.582500Z', 'realm_epoch': 2}}] 2024-03-04T18:58:57.439 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:58:57.439 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575076.582500_321.1'} 2024-03-04T18:58:57.439 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575076.582500_321.1 2024-03-04T18:59:02.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:02.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:02.441 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:02.483 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:02.483 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:02.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:02.547+0000 7f1851dad980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:02.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:02.547+0000 7f1851dad980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:02.571 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:02Z"} 2024-03-04T18:59:02.572 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:02Z"} 2024-03-04T18:59:02.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:59:02.572 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:59:02.572 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:59:02.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:59:07.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:07.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:07.574 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:07.616 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:07.616 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:07.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:07.683+0000 7f93aed9e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:07.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:07.683+0000 7f93aed9e980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:07.708 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:07Z"} 2024-03-04T18:59:07.709 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:07Z"} 2024-03-04T18:59:07.709 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:59:07.709 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:59:07.709 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:59:07.709 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:59:12.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:12.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:12.711 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:12.758 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:12.758 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:12.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:12.838+0000 7fcd52f5f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:12.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:12.842+0000 7fcd52f5f980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:12.869 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:12Z"} 2024-03-04T18:59:12.869 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:12Z"} 2024-03-04T18:59:12.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:59:12.870 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:59:12.870 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:59:12.870 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:59:17.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:17.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:17.871 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:17.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:17.978+0000 7f2d2790d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:17.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:17.982+0000 7f2d2790d980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:18.008 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:17Z"} 2024-03-04T18:59:18.008 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:17Z"} 2024-03-04T18:59:18.009 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:59:18.009 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:59:18.009 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:59:18.009 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.310269_305.1 2024-03-04T18:59:23.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:23.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:23.011 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:23.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:23.134+0000 7f041d10c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:23.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:23.134+0000 7f041d10c980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:23.163 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:23Z"} 2024-03-04T18:59:23.164 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:23Z"} 2024-03-04T18:59:23.164 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:59:23.164 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:59:23.164 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:59:23.164 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.310269_305.1 2024-03-04T18:59:28.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:28.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:28.166 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:28.210 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:28.210 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:28.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:28.286+0000 7f8ec0c4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:28.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:28.286+0000 7f8ec0c4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:28.312 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:28Z"} 2024-03-04T18:59:28.312 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:28Z"} 2024-03-04T18:59:28.312 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:59:28.313 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:59:28.313 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:59:28.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.310269_305.1 2024-03-04T18:59:33.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:33.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:33.315 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:33.357 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:33.357 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:33.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:33.418+0000 7fdd6a625980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:33.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:33.422+0000 7fdd6a625980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:33.446 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:33Z"} 2024-03-04T18:59:33.446 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:33Z"} 2024-03-04T18:59:33.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T18:59:33.447 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:59:33.447 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T18:59:33.447 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.310269_305.1 2024-03-04T18:59:38.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:38.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:38.448 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:38.490 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:38.490 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:38.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:38.550+0000 7fc4422d4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:38.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:38.554+0000 7fc4422d4980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:38.583 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:38Z"} 2024-03-04T18:59:38.583 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:38Z"} 2024-03-04T18:59:38.584 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:59:38.584 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:59:38.584 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:59:38.584 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:59:43.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:43.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:43.585 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:43.627 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:43.627 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:43.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:43.702+0000 7f87d0f21980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:43.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:43.702+0000 7f87d0f21980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:43.728 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:43Z"} 2024-03-04T18:59:43.728 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:43Z"} 2024-03-04T18:59:43.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T18:59:43.728 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:59:43.728 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T18:59:43.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T18:59:48.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:48.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:48.730 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:48.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:48.846+0000 7f86843bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:48.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:48.850+0000 7f86843bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:48.876 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.120434_389.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.120434Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:48Z"} 2024-03-04T18:59:48.876 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.120434_389.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.120434Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:48Z"} 2024-03-04T18:59:48.877 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.120434_389.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.120434Z', 'realm_epoch': 2}}] 2024-03-04T18:59:48.877 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:59:48.877 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.120434_389.1'} 2024-03-04T18:59:48.877 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575694.120434_389.1 2024-03-04T18:59:53.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:53.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:53.878 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:53.920 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:53.921 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:53.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:53.982+0000 7ff660139980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:53.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:53.982+0000 7ff660139980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:54.010 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:53Z"} 2024-03-04T18:59:54.010 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:53Z"} 2024-03-04T18:59:54.010 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:59:54.010 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:59:54.010 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:59:54.010 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T18:59:59.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:59.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:59.013 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:59.056 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:59.057 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:59.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:59.121+0000 7f45c4da6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:59.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T18:59:59.121+0000 7f45c4da6980 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:59.153 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:59Z"} 2024-03-04T18:59:59.153 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:59Z"} 2024-03-04T18:59:59.153 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T18:59:59.153 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T18:59:59.153 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T18:59:59.153 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:00:04.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:04.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:04.155 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:04.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:04.269+0000 7f022ffff980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:04.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:04.273+0000 7f022ffff980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:04.304 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:04Z"} 2024-03-04T19:00:04.304 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:04Z"} 2024-03-04T19:00:04.304 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:00:04.304 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:00:04.304 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:00:04.305 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:00:09.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:09.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:09.306 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:09.349 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:09.349 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:09.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:09.425+0000 7fdce3dd0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:09.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:09.425+0000 7fdce3dd0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:09.454 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:09Z"} 2024-03-04T19:00:09.454 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:09Z"} 2024-03-04T19:00:09.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:00:09.454 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:00:09.454 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:00:09.455 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:00:14.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:14.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:14.456 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:14.499 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:14.499 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:14.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:14.653+0000 7fb6d6e17980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:14.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:14.657+0000 7fb6d6e17980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:14.682 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:14Z"} 2024-03-04T19:00:14.682 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:14Z"} 2024-03-04T19:00:14.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:00:14.683 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:00:14.683 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:00:14.683 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:00:19.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:19.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:19.685 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:19.728 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:19.728 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:19.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:19.801+0000 7f44b3c7c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:19.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:19.801+0000 7f44b3c7c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:19.829 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.480321_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.480321Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:19Z"} 2024-03-04T19:00:19.829 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.480321_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.480321Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:19Z"} 2024-03-04T19:00:19.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.480321_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.480321Z', 'realm_epoch': 2}}] 2024-03-04T19:00:19.829 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:00:19.830 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.480321_315.1'} 2024-03-04T19:00:19.830 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.480321_315.1 2024-03-04T19:00:24.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:24.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:24.832 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:24.873 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:24.874 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:24.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:24.945+0000 7eff047b6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:24.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:24.945+0000 7eff047b6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:24.979 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:24Z"} 2024-03-04T19:00:24.979 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:24Z"} 2024-03-04T19:00:24.979 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:00:24.979 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:00:24.979 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:00:24.980 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:00:29.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:29.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:29.981 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:30.024 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:30.024 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:30.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:30.085+0000 7fbb58ff8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:30.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:30.085+0000 7fbb58ff8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:30.115 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:30Z"} 2024-03-04T19:00:30.115 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:30Z"} 2024-03-04T19:00:30.115 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:00:30.115 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:00:30.115 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:00:30.116 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:00:35.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:35.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:35.117 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:35.157 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:35.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:35.233+0000 7fd7f92c0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:35.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:35.233+0000 7fd7f92c0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:35.264 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:35Z"} 2024-03-04T19:00:35.264 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:35Z"} 2024-03-04T19:00:35.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:00:35.265 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:00:35.265 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:00:35.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:00:40.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:40.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:40.267 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:40.308 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:40.308 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:40.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:40.369+0000 7fe9861c0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:40.369+0000 7fe9861c0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:40.399 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:40Z"} 2024-03-04T19:00:40.399 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:40Z"} 2024-03-04T19:00:40.399 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:00:40.400 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:00:40.400 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:00:40.400 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.310269_305.1 2024-03-04T19:00:45.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:45.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:45.401 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:45.444 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:45.444 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:45.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:45.505+0000 7fbb4f8fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:45.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:45.505+0000 7fbb4f8fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:45.537 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.393343_310.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.393343Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:45Z"} 2024-03-04T19:00:45.537 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.393343_310.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.393343Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:45Z"} 2024-03-04T19:00:45.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.393343_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.393343Z', 'realm_epoch': 2}}] 2024-03-04T19:00:45.537 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:00:45.537 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.393343_310.1'} 2024-03-04T19:00:45.537 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.393343_310.1 2024-03-04T19:00:50.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:50.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:50.538 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:50.580 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:50.580 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:50.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:50.648+0000 7f139e8d0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:50.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:50.652+0000 7f139e8d0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:50.680 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:50Z"} 2024-03-04T19:00:50.680 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:50Z"} 2024-03-04T19:00:50.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.292440_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.292440Z', 'realm_epoch': 2}}] 2024-03-04T19:00:50.680 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:00:50.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.292440_304.1'} 2024-03-04T19:00:50.681 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.292440_304.1 2024-03-04T19:00:55.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2024-03-04T19:00:55.684 DEBUG:boto:path=/ppfahn-70/ 2024-03-04T19:00:55.684 DEBUG:boto:auth_path=/ppfahn-70/ 2024-03-04T19:00:55.685 DEBUG:boto:Method: PUT 2024-03-04T19:00:55.685 DEBUG:boto:Path: /ppfahn-70/ 2024-03-04T19:00:55.685 DEBUG:boto:Data: 2024-03-04T19:00:55.685 DEBUG:boto:Headers: {} 2024-03-04T19:00:55.685 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:00:55.685 DEBUG:boto:Port: 8000 2024-03-04T19:00:55.685 DEBUG:boto:Params: {} 2024-03-04T19:00:55.686 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:00:55.686 DEBUG:boto:Token: None 2024-03-04T19:00:55.686 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:00:55 GMT /ppfahn-70/ 2024-03-04T19:00:55.686 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:BRTDvVQKsyjyU9Ul61r6hN4/kKM= 2024-03-04T19:00:55.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:00:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BRTDvVQKsyjyU9Ul61r6hN4/kKM=', 'Content-Length': '0'} 2024-03-04T19:00:55.733 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ff544f1af2dd86d7-0065e61a67-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 19:00:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:00:55.733 DEBUG:rgw_multi.tests:created bucket=ppfahn-70 2024-03-04T19:00:55.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:55.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:55.734 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:55.899 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:55.900 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:56.092 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:00:56.146 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709578855.731020_637.1","last_update":"2024-03-04T19:00:55.731020Z"}] 2024-03-04T19:00:56.147 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:00:56.147 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T19:00:56.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:56.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:56.148 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:56.189 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:56.189 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:56.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:56.268+0000 7f51c66b4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:56.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:00:56.268+0000 7f51c66b4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:56.338 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:56Z"} 2024-03-04T19:00:56.338 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:56Z"} 2024-03-04T19:00:56.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:00:56.339 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:00:56.339 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:00:56.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:01:01.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:01.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:01.341 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:01.383 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:01.383 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:01.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:01.468+0000 7fddf282e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:01.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:01.468+0000 7fddf282e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:01.510 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:01Z"} 2024-03-04T19:01:01.511 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:01Z"} 2024-03-04T19:01:01.511 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:01:01.511 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:01:01.511 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:01:01.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:01:06.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:06.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:06.513 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:06.556 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:06.557 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:06.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:06.616+0000 7fbeaa422980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:06.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:06.616+0000 7fbeaa422980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:06.649 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:06Z"} 2024-03-04T19:01:06.649 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:06Z"} 2024-03-04T19:01:06.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:01:06.649 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:01:06.649 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:01:06.650 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:01:11.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:11.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:11.652 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:11.706 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:11.707 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:11.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:11.776+0000 7ff7169de980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:11.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:11.776+0000 7ff7169de980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:11.803 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:11Z"} 2024-03-04T19:01:11.803 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:11Z"} 2024-03-04T19:01:11.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:01:11.803 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:01:11.804 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:01:11.804 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:01:16.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:16.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:16.805 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:16.846 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:16.846 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:16.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:16.904+0000 7fa3294c6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:16.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:16.908+0000 7fa3294c6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:16.940 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:16Z"} 2024-03-04T19:01:16.940 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:16Z"} 2024-03-04T19:01:16.940 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:01:16.940 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:01:16.941 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:01:16.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:01:21.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:21.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:21.943 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:21.986 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:21.986 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:22.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:22.056+0000 7f8693ace980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:22.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:22.056+0000 7f8693ace980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:22.091 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:22Z"} 2024-03-04T19:01:22.092 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:22Z"} 2024-03-04T19:01:22.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:01:22.092 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:01:22.092 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:01:22.092 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:01:27.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:27.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:27.095 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:27.144 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:27.144 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:27.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:27.204+0000 7f8924b55980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:27.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:27.204+0000 7f8924b55980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:27.233 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:27Z"} 2024-03-04T19:01:27.233 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:27Z"} 2024-03-04T19:01:27.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:01:27.233 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:01:27.234 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:01:27.234 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:01:32.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:32.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:32.236 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:32.278 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:32.278 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:32.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:32.348+0000 7fac3bce9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:32.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:32.352+0000 7fac3bce9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:32.381 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.424470_313.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.424470Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:32Z"} 2024-03-04T19:01:32.381 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.424470_313.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.424470Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:32Z"} 2024-03-04T19:01:32.382 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.424470_313.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.424470Z', 'realm_epoch': 2}}] 2024-03-04T19:01:32.382 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:01:32.382 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.424470_313.1'} 2024-03-04T19:01:32.382 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.424470_313.1 2024-03-04T19:01:37.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:37.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:37.383 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:37.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:37.496+0000 7fcc54234980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:37.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:37.496+0000 7fcc54234980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:37.524 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:37Z"} 2024-03-04T19:01:37.525 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:37Z"} 2024-03-04T19:01:37.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:01:37.525 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:01:37.525 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:01:37.525 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:01:42.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:42.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:42.526 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:42.569 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:42.569 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:42.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:42.631+0000 7f79c8a17980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:42.631+0000 7f79c8a17980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:42.667 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:42Z"} 2024-03-04T19:01:42.667 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:42Z"} 2024-03-04T19:01:42.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T19:01:42.667 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:01:42.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T19:01:42.668 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.380518_307.1 2024-03-04T19:01:46.454 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T19:01:46.451+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:01:47.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:47.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:47.669 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:47.712 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:47.712 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:47.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:47.791+0000 7f216e20c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:47.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:47.795+0000 7f216e20c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:47.830 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:47Z"} 2024-03-04T19:01:47.830 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:47Z"} 2024-03-04T19:01:47.830 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:01:47.831 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:01:47.831 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:01:47.831 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:01:52.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:52.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:52.832 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:52.875 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:52.875 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:52.943+0000 7f025bf6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:52.943+0000 7f025bf6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:52.974 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:52Z"} 2024-03-04T19:01:52.974 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:52Z"} 2024-03-04T19:01:52.974 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:01:52.974 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:01:52.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:01:52.975 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:01:57.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:57.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:57.976 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:58.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:58.095+0000 7f00ef21f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:58.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:01:58.099+0000 7f00ef21f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:58.132 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:58Z"} 2024-03-04T19:01:58.132 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:58Z"} 2024-03-04T19:01:58.132 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:01:58.133 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:01:58.133 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:01:58.133 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:02:03.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:03.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:03.134 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:03.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:03.235+0000 7faa1f9d9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:03.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:03.235+0000 7faa1f9d9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:03.265 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:03Z"} 2024-03-04T19:02:03.265 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:03Z"} 2024-03-04T19:02:03.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:02:03.265 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:02:03.265 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:02:03.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:02:08.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:08.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:08.267 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:08.311 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:08.312 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:08.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:08.371+0000 7f3da1236980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:08.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:08.375+0000 7f3da1236980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:08.405 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:08Z"} 2024-03-04T19:02:08.405 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:08Z"} 2024-03-04T19:02:08.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:02:08.406 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:02:08.406 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:02:08.406 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:02:13.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:13.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:13.407 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:13.450 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:13.450 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:13.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:13.523+0000 7f76e66f8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:13.523+0000 7f76e66f8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:13.554 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:13Z"} 2024-03-04T19:02:13.554 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:13Z"} 2024-03-04T19:02:13.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:02:13.555 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:02:13.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:02:13.555 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:02:18.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:18.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:18.556 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:18.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:18.659+0000 7f571409d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:18.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:18.663+0000 7f571409d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:18.691 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:18Z"} 2024-03-04T19:02:18.691 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:18Z"} 2024-03-04T19:02:18.692 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:02:18.692 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:02:18.692 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:02:18.692 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:02:23.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:23.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:23.694 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:23.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:23.815+0000 7f4700f0e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:23.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:23.815+0000 7f4700f0e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:23.841 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:23Z"} 2024-03-04T19:02:23.841 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:23Z"} 2024-03-04T19:02:23.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:02:23.842 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:02:23.842 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:02:23.842 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:02:28.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:28.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:28.844 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:28.888 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:28.888 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:28.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:28.955+0000 7f1de5c93980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:28.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:28.959+0000 7f1de5c93980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:28.991 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:28Z"} 2024-03-04T19:02:28.991 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:28Z"} 2024-03-04T19:02:28.991 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:02:28.991 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:02:28.991 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:02:28.991 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:02:33.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:33.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:33.993 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:34.035 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:34.036 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:34.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:34.110+0000 7f7f7769b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:34.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:34.110+0000 7f7f7769b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:34.145 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:34Z"} 2024-03-04T19:02:34.145 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:34Z"} 2024-03-04T19:02:34.146 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:02:34.146 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:02:34.146 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:02:34.146 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:02:39.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:39.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:39.147 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:39.189 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:39.189 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:39.262+0000 7fa41f153980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:39.262+0000 7fa41f153980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:39.298 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:39Z"} 2024-03-04T19:02:39.298 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:39Z"} 2024-03-04T19:02:39.298 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:02:39.298 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:02:39.299 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:02:39.299 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:02:44.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:44.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:44.300 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:44.345 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:44.345 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:44.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:44.422+0000 7f10876eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:44.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:44.422+0000 7f10876eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:44.452 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:44Z"} 2024-03-04T19:02:44.452 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:44Z"} 2024-03-04T19:02:44.452 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:02:44.452 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:02:44.453 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:02:44.453 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:02:49.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:49.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:49.454 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:49.497 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:49.497 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:49.558+0000 7f470925b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:49.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:49.558+0000 7f470925b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:49.586 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:49Z"} 2024-03-04T19:02:49.587 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:49Z"} 2024-03-04T19:02:49.587 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:02:49.587 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:02:49.587 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:02:49.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:02:54.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:54.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:54.588 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:54.631 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:54.631 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:54.714+0000 7f2345c46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:54.718+0000 7f2345c46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:54.757 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:54Z"} 2024-03-04T19:02:54.757 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:54Z"} 2024-03-04T19:02:54.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:02:54.758 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:02:54.758 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:02:54.758 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:02:59.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:59.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:59.759 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:59.803 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:59.803 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:59.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:59.870+0000 7f9e0330f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:59.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:02:59.870+0000 7f9e0330f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:59.898 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:59Z"} 2024-03-04T19:02:59.898 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:59Z"} 2024-03-04T19:02:59.898 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:02:59.898 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:02:59.898 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:02:59.899 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:03:04.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:04.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:04.900 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:04.945 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:04.945 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:05.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:05.014+0000 7f05ff090980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:05.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:05.018+0000 7f05ff090980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:05.046 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:05Z"} 2024-03-04T19:03:05.046 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:05Z"} 2024-03-04T19:03:05.046 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:03:05.046 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:03:05.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:03:05.047 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:03:10.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:10.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:10.048 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:10.090 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:10.090 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:10.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:10.154+0000 7fc70aa46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:10.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:10.158+0000 7fc70aa46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:10.187 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:10Z"} 2024-03-04T19:03:10.187 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:10Z"} 2024-03-04T19:03:10.187 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:03:10.188 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:03:10.188 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:03:10.188 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:03:15.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:15.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:15.190 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:15.232 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:15.232 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:15.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:15.346+0000 7fc6b7856980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:15.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:15.350+0000 7fc6b7856980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:15.393 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:15Z"} 2024-03-04T19:03:15.393 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:15Z"} 2024-03-04T19:03:15.393 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:03:15.394 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:03:15.394 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:03:15.394 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:03:20.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:20.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:20.395 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:20.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:20.502+0000 7f5e5c55a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:20.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:20.502+0000 7f5e5c55a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:20.531 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:20Z"} 2024-03-04T19:03:20.531 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:20Z"} 2024-03-04T19:03:20.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:03:20.531 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:03:20.531 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:03:20.531 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:03:25.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:25.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:25.533 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:25.576 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:25.576 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:25.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:25.666+0000 7f135a33c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:25.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:25.670+0000 7f135a33c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:25.711 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:25Z"} 2024-03-04T19:03:25.711 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:25Z"} 2024-03-04T19:03:25.711 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:03:25.711 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:03:25.711 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:03:25.712 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:03:28.904 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T19:03:28.904+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:03:30.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:30.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:30.713 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:30.756 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:30.756 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:30.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:30.817+0000 7fa56d028980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:30.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:30.817+0000 7fa56d028980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:30.852 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:30Z"} 2024-03-04T19:03:30.852 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:30Z"} 2024-03-04T19:03:30.852 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:03:30.852 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:03:30.852 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:03:30.853 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:03:35.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:35.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:35.854 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:35.895 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:35.895 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:35.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:35.961+0000 7fdde5fba980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:35.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:35.961+0000 7fdde5fba980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:35.988 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:35Z"} 2024-03-04T19:03:35.988 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:35Z"} 2024-03-04T19:03:35.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.388355_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.388355Z', 'realm_epoch': 2}}] 2024-03-04T19:03:35.989 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:03:35.989 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.388355_309.1'} 2024-03-04T19:03:35.989 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.388355_309.1 2024-03-04T19:03:40.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:40.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:40.990 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:41.034 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:41.034 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:41.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:41.109+0000 7efce25eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:41.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:41.109+0000 7efce25eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:41.138 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:41Z"} 2024-03-04T19:03:41.139 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:41Z"} 2024-03-04T19:03:41.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:03:41.139 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:03:41.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:03:41.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:03:46.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:46.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:46.140 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:46.183 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:46.184 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:46.253+0000 7f62a49c7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:46.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:46.257+0000 7f62a49c7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:46.284 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:46Z"} 2024-03-04T19:03:46.284 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:46Z"} 2024-03-04T19:03:46.284 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:03:46.284 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:03:46.284 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:03:46.285 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:03:51.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:51.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:51.287 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:51.330 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:51.330 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:51.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:51.397+0000 7f6fe0661980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:51.401+0000 7f6fe0661980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:51.428 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:51Z"} 2024-03-04T19:03:51.428 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:51Z"} 2024-03-04T19:03:51.428 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:03:51.428 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:03:51.428 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:03:51.428 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:03:56.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:56.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:56.429 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:56.473 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:56.473 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:56.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:56.513+0000 7fa4f67fc640 0 ms_deliver_dispatch: unhandled message 0x7fa60004de00 osd_map(42..42 src has 1..42) v4 from mon.0 v2:172.21.15.153:3300/0 2024-03-04T19:03:56.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:56.557+0000 7fa61337d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:03:56.557+0000 7fa61337d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:56.586 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:56Z"} 2024-03-04T19:03:56.587 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:56Z"} 2024-03-04T19:03:56.587 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:03:56.587 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:03:56.587 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:03:56.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:04:01.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:01.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:01.589 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:01.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:01.713+0000 7fe4cbfc6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:01.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:01.717+0000 7fe4cbfc6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:01.753 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.735016_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.735016Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:01Z"} 2024-03-04T19:04:01.754 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.735016_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.735016Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:01Z"} 2024-03-04T19:04:01.754 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.735016_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.735016Z', 'realm_epoch': 2}}] 2024-03-04T19:04:01.754 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:04:01.754 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.735016_329.1'} 2024-03-04T19:04:01.754 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575077.735016_329.1 2024-03-04T19:04:06.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:06.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:06.756 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:06.800 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:06.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:06.861+0000 7f7e191b5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:06.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:06.861+0000 7f7e191b5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:06.889 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:06Z"} 2024-03-04T19:04:06.890 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:06Z"} 2024-03-04T19:04:06.890 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:04:06.890 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:04:06.890 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:04:06.890 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:04:11.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:11.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:11.892 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:11.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:11.997+0000 7fae1261f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:12.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:12.001+0000 7fae1261f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:12.035 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:12Z"} 2024-03-04T19:04:12.035 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:12Z"} 2024-03-04T19:04:12.035 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:04:12.035 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:04:12.035 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:04:12.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:04:17.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:17.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:17.038 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:17.089 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:17.089 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:17.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:17.153+0000 7f19bbe8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:17.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:17.153+0000 7f19bbe8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:17.183 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:17Z"} 2024-03-04T19:04:17.183 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:17Z"} 2024-03-04T19:04:17.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:04:17.183 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:04:17.183 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:04:17.183 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:04:22.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:22.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:22.185 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:22.229 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:22.229 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:22.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:22.304+0000 7f5368f54980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:22.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:22.304+0000 7f5368f54980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:22.335 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:22Z"} 2024-03-04T19:04:22.335 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:22Z"} 2024-03-04T19:04:22.335 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:04:22.336 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:04:22.336 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:04:22.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:04:27.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:27.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:27.337 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:27.381 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:27.381 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:27.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:27.452+0000 7f14053a4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:27.452+0000 7f14053a4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:27.488 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:27Z"} 2024-03-04T19:04:27.488 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:27Z"} 2024-03-04T19:04:27.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.315397_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.315397Z', 'realm_epoch': 2}}] 2024-03-04T19:04:27.488 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:04:27.489 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.315397_306.1'} 2024-03-04T19:04:27.489 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.315397_306.1 2024-03-04T19:04:32.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:32.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:32.490 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:32.534 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:32.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:32.600+0000 7f4175f12980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:32.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:32.600+0000 7f4175f12980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:32.629 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:32Z"} 2024-03-04T19:04:32.629 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:32Z"} 2024-03-04T19:04:32.630 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:04:32.630 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:04:32.630 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:04:32.630 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:04:37.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:37.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:37.631 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:37.673 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:37.673 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:37.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:37.748+0000 7f6d45928980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:37.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:37.748+0000 7f6d45928980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:37.785 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:37Z"} 2024-03-04T19:04:37.785 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:37Z"} 2024-03-04T19:04:37.786 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:04:37.786 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:04:37.786 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:04:37.786 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:04:42.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:42.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:42.787 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:42.830 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:42.830 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:42.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:42.908+0000 7f23fb946980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:42.908+0000 7f23fb946980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:42.944 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:42Z"} 2024-03-04T19:04:42.945 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:42Z"} 2024-03-04T19:04:42.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:04:42.945 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:04:42.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:04:42.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:04:47.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:47.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:47.946 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:47.988 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:47.988 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:48.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:48.068+0000 7f66ef762980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:48.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:48.068+0000 7f66ef762980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:48.098 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:48Z"} 2024-03-04T19:04:48.099 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:48Z"} 2024-03-04T19:04:48.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:04:48.099 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:04:48.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:04:48.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:04:53.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:53.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:53.100 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:53.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:53.208+0000 7f2590434980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:53.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:53.208+0000 7f2590434980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:53.241 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:53Z"} 2024-03-04T19:04:53.242 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:53Z"} 2024-03-04T19:04:53.242 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:04:53.242 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:04:53.242 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:04:53.242 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:04:58.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:58.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:58.245 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:58.424+0000 7f2f2d4e5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:04:58.424+0000 7f2f2d4e5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:58.456 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:58Z"} 2024-03-04T19:04:58.456 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:58Z"} 2024-03-04T19:04:58.457 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:04:58.457 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:04:58.457 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:04:58.457 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:05:03.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:03.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:03.459 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:03.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:03.564+0000 7f3e2cd4e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:03.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:03.564+0000 7f3e2cd4e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:03.597 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:03Z"} 2024-03-04T19:05:03.597 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:03Z"} 2024-03-04T19:05:03.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:05:03.597 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:05:03.598 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:05:03.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:05:08.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:08.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:08.599 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:08.641 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:08.641 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:08.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:08.720+0000 7fad00536980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:08.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:08.720+0000 7fad00536980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:08.749 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:08Z"} 2024-03-04T19:05:08.750 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:08Z"} 2024-03-04T19:05:08.750 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:05:08.750 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:05:08.750 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:05:08.750 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:05:13.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:13.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:13.751 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:13.793 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:13.793 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:13.871+0000 7f19f45d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:13.871+0000 7f19f45d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:13.909 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:13Z"} 2024-03-04T19:05:13.909 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:13Z"} 2024-03-04T19:05:13.909 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:05:13.910 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:05:13.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:05:13.910 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:05:18.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:18.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:18.911 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:18.952 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:18.952 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:19.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:19.023+0000 7fabb0cc0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:19.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:19.027+0000 7fabb0cc0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:19.059 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:19Z"} 2024-03-04T19:05:19.059 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:19Z"} 2024-03-04T19:05:19.059 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:05:19.060 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:05:19.060 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:05:19.060 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:05:24.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:24.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:24.062 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:24.104 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:24.104 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:24.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:24.163+0000 7f0f1b155980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:24.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:24.163+0000 7f0f1b155980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:24.192 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:24Z"} 2024-03-04T19:05:24.192 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:24Z"} 2024-03-04T19:05:24.192 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:05:24.192 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:05:24.192 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:05:24.193 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:05:29.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:29.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:29.194 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:29.235 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:29.235 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:29.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:29.311+0000 7f25c5493980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:29.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:29.315+0000 7f25c5493980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:29.350 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:29Z"} 2024-03-04T19:05:29.350 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:29Z"} 2024-03-04T19:05:29.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:05:29.351 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:05:29.351 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:05:29.351 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:05:34.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:34.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:34.352 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:34.394 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:34.394 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:34.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:34.459+0000 7fb6341fb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:34.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:34.463+0000 7fb6341fb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:34.495 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.650819_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.650819Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:34Z"} 2024-03-04T19:05:34.495 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.650819_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.650819Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:34Z"} 2024-03-04T19:05:34.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.650819_323.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.650819Z', 'realm_epoch': 2}}] 2024-03-04T19:05:34.496 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:05:34.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.650819_323.1'} 2024-03-04T19:05:34.496 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575077.650819_323.1 2024-03-04T19:05:39.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:39.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:39.497 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:39.541 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:39.541 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:39.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:39.623+0000 7fd8b5012980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:39.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:39.623+0000 7fd8b5012980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:39.660 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:39Z"} 2024-03-04T19:05:39.661 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:39Z"} 2024-03-04T19:05:39.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:05:39.661 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:05:39.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:05:39.661 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:05:44.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:44.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:44.662 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:44.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:44.771+0000 7f28e5cef980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:44.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:44.771+0000 7f28e5cef980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:44.798 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:44Z"} 2024-03-04T19:05:44.798 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:44Z"} 2024-03-04T19:05:44.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:05:44.798 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:05:44.799 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:05:44.799 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:05:49.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:49.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:49.800 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:49.844 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:49.844 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:49.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:49.907+0000 7f0e1870c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:49.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:49.907+0000 7f0e1870c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:49.938 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.386285_308.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.386285Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:49Z"} 2024-03-04T19:05:49.939 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.386285_308.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.386285Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:49Z"} 2024-03-04T19:05:49.939 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.386285_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.386285Z', 'realm_epoch': 2}}] 2024-03-04T19:05:49.939 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:05:49.939 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.386285_308.1'} 2024-03-04T19:05:49.939 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.386285_308.1 2024-03-04T19:05:54.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:54.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:54.940 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:55.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:55.059+0000 7f5a60afb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:55.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:05:55.059+0000 7f5a60afb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:55.086 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:55Z"} 2024-03-04T19:05:55.086 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:55Z"} 2024-03-04T19:05:55.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:05:55.087 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:05:55.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:05:55.087 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:06:00.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:00.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:00.088 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:00.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:00.198+0000 7f3a32952980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:00.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:00.198+0000 7f3a32952980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:00.225 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:00Z"} 2024-03-04T19:06:00.225 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:00Z"} 2024-03-04T19:06:00.226 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:06:00.226 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:06:00.226 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:06:00.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:06:05.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... FAIL 2024-03-04T19:06:05.230 DEBUG:boto:path=/ppfahn-71/ 2024-03-04T19:06:05.230 DEBUG:boto:auth_path=/ppfahn-71/ 2024-03-04T19:06:05.231 DEBUG:boto:Method: PUT 2024-03-04T19:06:05.231 DEBUG:boto:Path: /ppfahn-71/ 2024-03-04T19:06:05.231 DEBUG:boto:Data: 2024-03-04T19:06:05.231 DEBUG:boto:Headers: {} 2024-03-04T19:06:05.231 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:06:05.231 DEBUG:boto:Port: 8000 2024-03-04T19:06:05.231 DEBUG:boto:Params: {} 2024-03-04T19:06:05.232 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:06:05.232 DEBUG:boto:Token: None 2024-03-04T19:06:05.232 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:06:05 GMT /ppfahn-71/ 2024-03-04T19:06:05.232 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:GDDnhuSJPo2m98Rcy9XPZ5HsE0g= 2024-03-04T19:06:05.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:06:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GDDnhuSJPo2m98Rcy9XPZ5HsE0g=', 'Content-Length': '0'} 2024-03-04T19:06:05.279 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000097f1b3349a94121-0065e61b9d-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:06:05.279 DEBUG:rgw_multi.tests:created bucket=ppfahn-71 2024-03-04T19:06:05.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:05.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:05.280 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:05.411 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:05.412 INFO:teuthology.orchestra.run.smithi136.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:05.618 INFO:teuthology.orchestra.run.smithi136.stderr:No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:06:05.696 INFO:teuthology.orchestra.run.smithi136.stdout:[{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709579165.276086_654.1","last_update":"2024-03-04T19:06:05.276086Z"}] 2024-03-04T19:06:05.697 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:06:05.697 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T19:06:05.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:05.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:05.697 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:05.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:05.806+0000 7ff2ebd97980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:05.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:05.806+0000 7ff2ebd97980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:05.843 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:05Z"} 2024-03-04T19:06:05.843 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:05Z"} 2024-03-04T19:06:05.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:06:05.843 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:06:05.843 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:06:05.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:06:10.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:10.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:10.845 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:10.888 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:10.888 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:10.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:10.950+0000 7f8979ca2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:10.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:10.954+0000 7f8979ca2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:10.979 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:10Z"} 2024-03-04T19:06:10.980 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:10Z"} 2024-03-04T19:06:10.980 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:06:10.980 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:06:10.980 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:06:10.980 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:06:15.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:15.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:15.982 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:16.024 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:16.024 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:16.094+0000 7f74d8ff7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:16.094+0000 7f74d8ff7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:16.129 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:16Z"} 2024-03-04T19:06:16.129 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:16Z"} 2024-03-04T19:06:16.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:06:16.129 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:06:16.130 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:06:16.130 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:06:21.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:21.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:21.131 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:21.172 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:21.172 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:21.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:21.230+0000 7f7963c46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:21.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:21.234+0000 7f7963c46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:21.259 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:21Z"} 2024-03-04T19:06:21.260 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:21Z"} 2024-03-04T19:06:21.260 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:06:21.260 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:06:21.260 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:06:21.260 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:06:26.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:26.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:26.262 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:26.305 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:26.305 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:26.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:26.366+0000 7fac5e984980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:26.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:26.366+0000 7fac5e984980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:26.398 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:26Z"} 2024-03-04T19:06:26.398 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:26Z"} 2024-03-04T19:06:26.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.292440_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.292440Z', 'realm_epoch': 2}}] 2024-03-04T19:06:26.398 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:06:26.399 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.292440_304.1'} 2024-03-04T19:06:26.399 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.292440_304.1 2024-03-04T19:06:31.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:31.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:31.400 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:31.443 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:31.443 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:31.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:31.506+0000 7f82afc73980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:31.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:31.506+0000 7f82afc73980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:31.532 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:31Z"} 2024-03-04T19:06:31.532 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:31Z"} 2024-03-04T19:06:31.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:06:31.533 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:06:31.533 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:06:31.533 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:06:36.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:36.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:36.534 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:36.580 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:36.581 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:36.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:36.638+0000 7f6b3b394980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:36.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:36.638+0000 7f6b3b394980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:36.667 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:36Z"} 2024-03-04T19:06:36.667 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:36Z"} 2024-03-04T19:06:36.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:06:36.668 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:06:36.668 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:06:36.668 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:06:41.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:41.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:41.670 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:41.714 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:41.714 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:41.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:41.790+0000 7f72febc9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:41.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:41.794+0000 7f72febc9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:41.827 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:41Z"} 2024-03-04T19:06:41.827 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:41Z"} 2024-03-04T19:06:41.827 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:06:41.827 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:06:41.827 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:06:41.827 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.271482_301.1 2024-03-04T19:06:46.705 INFO:tasks.rgw.c1.client.0.smithi136.stdout:2024-03-04T19:06:46.703+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:06:46.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:46.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:46.829 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:46.886 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:46.886 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:46.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:46.985+0000 7fef77f4b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:46.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:46.989+0000 7fef77f4b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:47.016 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:46Z"} 2024-03-04T19:06:47.016 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:46Z"} 2024-03-04T19:06:47.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:06:47.016 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:06:47.016 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:06:47.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:06:52.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:52.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:52.018 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:52.062 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:52.062 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:52.121+0000 7f66af4cd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:52.125+0000 7f66af4cd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:52.158 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:52Z"} 2024-03-04T19:06:52.159 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:52Z"} 2024-03-04T19:06:52.159 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:06:52.159 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:06:52.159 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:06:52.159 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:06:57.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:57.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:57.160 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:57.203 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:57.203 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:57.293+0000 7f4b582bf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:57.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:06:57.293+0000 7f4b582bf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:57.328 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.815737_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.815737Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:57Z"} 2024-03-04T19:06:57.328 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.815737_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.815737Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:57Z"} 2024-03-04T19:06:57.329 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.815737_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.815737Z', 'realm_epoch': 2}}] 2024-03-04T19:06:57.329 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:06:57.329 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.815737_334.1'} 2024-03-04T19:06:57.329 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575077.815737_334.1 2024-03-04T19:07:02.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:02.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:02.330 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:07:02.369 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:07:02.369 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:07:02.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:07:02.445+0000 7f1ce082c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:02.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:07:02.445+0000 7f1ce082c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:02.476 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:02Z"} 2024-03-04T19:07:02.476 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:02Z"} 2024-03-04T19:07:02.477 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:07:02.477 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:07:02.477 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:07:02.477 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.271482_301.1 2024-03-04T19:07:07.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:07.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:07.479 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:07:07.521 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:07:07.521 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:07:07.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:07:07.581+0000 7f4333a53980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:07.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:07:07.585+0000 7f4333a53980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:07.613 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:07Z"} 2024-03-04T19:07:07.613 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:07Z"} 2024-03-04T19:07:07.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:07:07.614 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:07:07.614 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:07:07.614 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.271482_301.1 2024-03-04T19:07:12.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:12.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:12.615 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:07:12.657 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:07:12.657 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:07:12.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:07:12.761+0000 7f1a895b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:12.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:07:12.761+0000 7f1a895b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:12.789 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:12Z"} 2024-03-04T19:07:12.790 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:12Z"} 2024-03-04T19:07:12.790 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:07:12.790 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:07:12.790 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:07:12.790 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.310269_305.1 2024-03-04T19:07:17.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:17.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:17.792 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:07:17.834 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:07:17.834 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:07:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:07:17.913+0000 7f5e4fe02980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:17.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:07:17.913+0000 7f5e4fe02980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:17.948 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:17Z"} 2024-03-04T19:07:17.948 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:17Z"} 2024-03-04T19:07:17.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:07:17.949 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:07:17.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:07:17.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.271482_301.1 2024-03-04T19:07:22.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:22.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:22.950 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:07:22.993 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:07:22.993 INFO:teuthology.orchestra.run.smithi153.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:07:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:07:23.061+0000 7fc36ac10980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-04T19:07:23.061+0000 7fc36ac10980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:23.092 INFO:teuthology.orchestra.run.smithi153.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:23Z"} 2024-03-04T19:07:23.092 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:23Z"} 2024-03-04T19:07:23.093 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:07:23.093 DEBUG:rgw_multi.tests:log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:07:23.093 DEBUG:rgw_multi.tests:sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:07:23.093 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:07:28.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:28.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:28.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:07:28.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:07:28.097 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:07:28.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~0s 2024-03-04T19:07:28.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~0s 2024-03-04T19:07:28.308 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~0s 2024-03-04T19:07:28.308 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~0s 2024-03-04T19:07:28.308 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-03-04T19:07:28.308 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-03-04T19:07:28.308 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~0s 2024-03-04T19:07:28.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~0s 2024-03-04T19:07:28.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~0s 2024-03-04T19:07:28.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~0s 2024-03-04T19:07:28.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~0s 2024-03-04T19:07:28.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-03-04T19:07:28.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-03-04T19:07:28.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~0s 2024-03-04T19:07:33.312 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~5s 2024-03-04T19:07:33.312 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~5s 2024-03-04T19:07:33.312 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~5s 2024-03-04T19:07:33.312 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~5s 2024-03-04T19:07:33.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~5s 2024-03-04T19:07:33.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~5s 2024-03-04T19:07:33.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~5s 2024-03-04T19:07:33.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~5s 2024-03-04T19:07:33.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~5s 2024-03-04T19:07:33.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~5s 2024-03-04T19:07:33.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~5s 2024-03-04T19:07:33.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~5s 2024-03-04T19:07:33.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~5s 2024-03-04T19:07:33.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~5s 2024-03-04T19:07:37.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:07:37.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:07:38.315 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~10s 2024-03-04T19:07:38.315 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~10s 2024-03-04T19:07:38.315 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~10s 2024-03-04T19:07:38.316 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~10s 2024-03-04T19:07:38.316 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~10s 2024-03-04T19:07:38.316 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~10s 2024-03-04T19:07:38.316 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~10s 2024-03-04T19:07:38.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~10s 2024-03-04T19:07:38.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~10s 2024-03-04T19:07:38.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~10s 2024-03-04T19:07:38.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~10s 2024-03-04T19:07:38.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~10s 2024-03-04T19:07:38.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~10s 2024-03-04T19:07:38.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~10s 2024-03-04T19:07:43.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~15s 2024-03-04T19:07:43.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~15s 2024-03-04T19:07:43.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~15s 2024-03-04T19:07:43.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~15s 2024-03-04T19:07:43.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~15s 2024-03-04T19:07:43.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~15s 2024-03-04T19:07:43.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~15s 2024-03-04T19:07:43.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~15s 2024-03-04T19:07:43.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~15s 2024-03-04T19:07:43.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~15s 2024-03-04T19:07:43.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~15s 2024-03-04T19:07:43.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~15s 2024-03-04T19:07:43.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~15s 2024-03-04T19:07:43.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~15s 2024-03-04T19:07:48.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~20s 2024-03-04T19:07:48.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~20s 2024-03-04T19:07:48.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~20s 2024-03-04T19:07:48.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~20s 2024-03-04T19:07:48.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~20s 2024-03-04T19:07:48.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~20s 2024-03-04T19:07:48.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~20s 2024-03-04T19:07:48.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~20s 2024-03-04T19:07:48.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~20s 2024-03-04T19:07:48.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~20s 2024-03-04T19:07:48.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~20s 2024-03-04T19:07:48.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~20s 2024-03-04T19:07:48.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~20s 2024-03-04T19:07:48.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~20s 2024-03-04T19:07:53.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~25s 2024-03-04T19:07:53.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~25s 2024-03-04T19:07:53.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~25s 2024-03-04T19:07:53.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~25s 2024-03-04T19:07:53.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~25s 2024-03-04T19:07:53.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~25s 2024-03-04T19:07:53.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~25s 2024-03-04T19:07:53.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~25s 2024-03-04T19:07:53.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~25s 2024-03-04T19:07:53.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~25s 2024-03-04T19:07:53.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~25s 2024-03-04T19:07:53.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~25s 2024-03-04T19:07:53.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~25s 2024-03-04T19:07:53.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~25s 2024-03-04T19:07:58.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~30s 2024-03-04T19:07:58.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~30s 2024-03-04T19:07:58.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~30s 2024-03-04T19:07:58.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~30s 2024-03-04T19:07:58.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~30s 2024-03-04T19:07:58.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~30s 2024-03-04T19:07:58.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~30s 2024-03-04T19:07:58.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~30s 2024-03-04T19:07:58.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~30s 2024-03-04T19:07:58.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~30s 2024-03-04T19:07:58.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~30s 2024-03-04T19:07:58.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~30s 2024-03-04T19:07:58.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~30s 2024-03-04T19:07:58.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~30s 2024-03-04T19:08:03.347 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~35s 2024-03-04T19:08:03.347 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~35s 2024-03-04T19:08:03.347 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~35s 2024-03-04T19:08:03.347 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~35s 2024-03-04T19:08:03.347 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~35s 2024-03-04T19:08:03.348 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~35s 2024-03-04T19:08:03.348 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~35s 2024-03-04T19:08:03.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~35s 2024-03-04T19:08:03.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~35s 2024-03-04T19:08:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~35s 2024-03-04T19:08:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~35s 2024-03-04T19:08:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~35s 2024-03-04T19:08:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~35s 2024-03-04T19:08:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~35s 2024-03-04T19:08:08.353 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~40s 2024-03-04T19:08:08.354 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~40s 2024-03-04T19:08:08.354 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~40s 2024-03-04T19:08:08.354 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~40s 2024-03-04T19:08:08.354 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~40s 2024-03-04T19:08:08.354 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~40s 2024-03-04T19:08:08.354 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~40s 2024-03-04T19:08:08.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~40s 2024-03-04T19:08:08.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~40s 2024-03-04T19:08:08.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~40s 2024-03-04T19:08:08.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~40s 2024-03-04T19:08:08.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~40s 2024-03-04T19:08:08.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~40s 2024-03-04T19:08:08.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~40s 2024-03-04T19:08:11.708 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:08:13.356 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~45s 2024-03-04T19:08:13.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~45s 2024-03-04T19:08:13.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~45s 2024-03-04T19:08:13.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~45s 2024-03-04T19:08:13.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~45s 2024-03-04T19:08:13.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~45s 2024-03-04T19:08:13.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~45s 2024-03-04T19:08:13.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~45s 2024-03-04T19:08:13.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~45s 2024-03-04T19:08:13.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~45s 2024-03-04T19:08:13.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~45s 2024-03-04T19:08:13.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~45s 2024-03-04T19:08:13.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~45s 2024-03-04T19:08:13.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~45s 2024-03-04T19:08:18.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~50s 2024-03-04T19:08:18.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~50s 2024-03-04T19:08:18.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~50s 2024-03-04T19:08:18.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~50s 2024-03-04T19:08:18.360 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~50s 2024-03-04T19:08:18.360 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~50s 2024-03-04T19:08:18.360 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~50s 2024-03-04T19:08:18.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~50s 2024-03-04T19:08:18.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~50s 2024-03-04T19:08:18.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~50s 2024-03-04T19:08:18.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~50s 2024-03-04T19:08:18.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~50s 2024-03-04T19:08:18.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~50s 2024-03-04T19:08:18.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~50s 2024-03-04T19:08:23.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~55s 2024-03-04T19:08:23.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~55s 2024-03-04T19:08:23.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~55s 2024-03-04T19:08:23.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~55s 2024-03-04T19:08:23.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~55s 2024-03-04T19:08:23.367 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~55s 2024-03-04T19:08:23.367 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~55s 2024-03-04T19:08:23.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~55s 2024-03-04T19:08:23.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~55s 2024-03-04T19:08:23.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~55s 2024-03-04T19:08:23.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~55s 2024-03-04T19:08:23.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~55s 2024-03-04T19:08:23.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~55s 2024-03-04T19:08:23.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~55s 2024-03-04T19:08:23.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:08:23.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:08:26.780 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:08:28.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~60s 2024-03-04T19:08:28.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~60s 2024-03-04T19:08:28.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~60s 2024-03-04T19:08:28.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~60s 2024-03-04T19:08:28.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~60s 2024-03-04T19:08:28.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~60s 2024-03-04T19:08:28.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~60s 2024-03-04T19:08:28.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~60s 2024-03-04T19:08:28.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~60s 2024-03-04T19:08:28.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~60s 2024-03-04T19:08:28.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~60s 2024-03-04T19:08:28.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~60s 2024-03-04T19:08:28.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~60s 2024-03-04T19:08:28.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~60s 2024-03-04T19:08:29.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:08:29.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:08:29.852 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:08:29.852 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:08:29.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ERROR 2024-03-04T19:08:29.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:08:29.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:08:29.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:08:29.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:08:33.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~65s 2024-03-04T19:08:33.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~65s 2024-03-04T19:08:33.376 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~65s 2024-03-04T19:08:33.376 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~65s 2024-03-04T19:08:33.376 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~65s 2024-03-04T19:08:33.376 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~65s 2024-03-04T19:08:33.376 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~65s 2024-03-04T19:08:33.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~65s 2024-03-04T19:08:33.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~65s 2024-03-04T19:08:33.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~65s 2024-03-04T19:08:33.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~65s 2024-03-04T19:08:33.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~65s 2024-03-04T19:08:33.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~65s 2024-03-04T19:08:33.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~65s 2024-03-04T19:08:38.382 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~70s 2024-03-04T19:08:38.382 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~70s 2024-03-04T19:08:38.382 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~70s 2024-03-04T19:08:38.382 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~70s 2024-03-04T19:08:38.382 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~70s 2024-03-04T19:08:38.382 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~70s 2024-03-04T19:08:38.383 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~70s 2024-03-04T19:08:38.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~70s 2024-03-04T19:08:38.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~70s 2024-03-04T19:08:38.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~70s 2024-03-04T19:08:38.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~70s 2024-03-04T19:08:38.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~70s 2024-03-04T19:08:38.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~70s 2024-03-04T19:08:38.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~70s 2024-03-04T19:08:43.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~75s 2024-03-04T19:08:43.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~75s 2024-03-04T19:08:43.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~75s 2024-03-04T19:08:43.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~75s 2024-03-04T19:08:43.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~75s 2024-03-04T19:08:43.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~75s 2024-03-04T19:08:43.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~75s 2024-03-04T19:08:43.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~75s 2024-03-04T19:08:43.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~75s 2024-03-04T19:08:43.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~75s 2024-03-04T19:08:43.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~75s 2024-03-04T19:08:43.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~75s 2024-03-04T19:08:43.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~75s 2024-03-04T19:08:43.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~75s 2024-03-04T19:08:48.316 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:08:48.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~80s 2024-03-04T19:08:48.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~80s 2024-03-04T19:08:48.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~80s 2024-03-04T19:08:48.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~80s 2024-03-04T19:08:48.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~80s 2024-03-04T19:08:48.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~80s 2024-03-04T19:08:48.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~80s 2024-03-04T19:08:48.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~80s 2024-03-04T19:08:48.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~80s 2024-03-04T19:08:48.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~80s 2024-03-04T19:08:48.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~80s 2024-03-04T19:08:48.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~80s 2024-03-04T19:08:48.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~80s 2024-03-04T19:08:48.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~80s 2024-03-04T19:08:53.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~85s 2024-03-04T19:08:53.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~85s 2024-03-04T19:08:53.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~85s 2024-03-04T19:08:53.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~85s 2024-03-04T19:08:53.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~85s 2024-03-04T19:08:53.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~85s 2024-03-04T19:08:53.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~85s 2024-03-04T19:08:53.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~85s 2024-03-04T19:08:53.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~85s 2024-03-04T19:08:53.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~85s 2024-03-04T19:08:53.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~85s 2024-03-04T19:08:53.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~85s 2024-03-04T19:08:53.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~85s 2024-03-04T19:08:53.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~85s 2024-03-04T19:08:57.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:08:57.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:08:58.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~90s 2024-03-04T19:08:58.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~90s 2024-03-04T19:08:58.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~90s 2024-03-04T19:08:58.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~90s 2024-03-04T19:08:58.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~90s 2024-03-04T19:08:58.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~90s 2024-03-04T19:08:58.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~90s 2024-03-04T19:08:58.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~90s 2024-03-04T19:08:58.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~90s 2024-03-04T19:08:58.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~90s 2024-03-04T19:08:58.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~90s 2024-03-04T19:08:58.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~90s 2024-03-04T19:08:58.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~90s 2024-03-04T19:08:58.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~90s 2024-03-04T19:09:00.380 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:09:03.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~95s 2024-03-04T19:09:03.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~95s 2024-03-04T19:09:03.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~95s 2024-03-04T19:09:03.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~95s 2024-03-04T19:09:03.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~95s 2024-03-04T19:09:03.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~95s 2024-03-04T19:09:03.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~95s 2024-03-04T19:09:03.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~95s 2024-03-04T19:09:03.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~95s 2024-03-04T19:09:03.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~95s 2024-03-04T19:09:03.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~95s 2024-03-04T19:09:03.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~95s 2024-03-04T19:09:03.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~95s 2024-03-04T19:09:03.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~95s 2024-03-04T19:09:08.409 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~100s 2024-03-04T19:09:08.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~100s 2024-03-04T19:09:08.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~100s 2024-03-04T19:09:08.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~100s 2024-03-04T19:09:08.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~100s 2024-03-04T19:09:08.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~100s 2024-03-04T19:09:08.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~100s 2024-03-04T19:09:08.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~100s 2024-03-04T19:09:08.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~100s 2024-03-04T19:09:08.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~100s 2024-03-04T19:09:08.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~100s 2024-03-04T19:09:08.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~100s 2024-03-04T19:09:08.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~100s 2024-03-04T19:09:08.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~100s 2024-03-04T19:09:12.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:09:12.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:13.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~105s 2024-03-04T19:09:13.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~105s 2024-03-04T19:09:13.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~105s 2024-03-04T19:09:13.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~105s 2024-03-04T19:09:13.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~105s 2024-03-04T19:09:13.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~105s 2024-03-04T19:09:13.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~105s 2024-03-04T19:09:13.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~105s 2024-03-04T19:09:13.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~105s 2024-03-04T19:09:13.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~105s 2024-03-04T19:09:13.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~105s 2024-03-04T19:09:13.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~105s 2024-03-04T19:09:13.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~105s 2024-03-04T19:09:13.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~105s 2024-03-04T19:09:15.452 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:09:18.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~110s 2024-03-04T19:09:18.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~110s 2024-03-04T19:09:18.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~110s 2024-03-04T19:09:18.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~110s 2024-03-04T19:09:18.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~110s 2024-03-04T19:09:18.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~110s 2024-03-04T19:09:18.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~110s 2024-03-04T19:09:18.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:09:18.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:18.524 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:09:18.524 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:09:18.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ERROR 2024-03-04T19:09:18.526 DEBUG:boto:path=/ppfahn-72/ 2024-03-04T19:09:18.526 DEBUG:boto:auth_path=/ppfahn-72/ 2024-03-04T19:09:18.527 DEBUG:boto:Method: PUT 2024-03-04T19:09:18.527 DEBUG:boto:Path: /ppfahn-72/ 2024-03-04T19:09:18.527 DEBUG:boto:Data: 2024-03-04T19:09:18.527 DEBUG:boto:Headers: {} 2024-03-04T19:09:18.527 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:09:18.527 DEBUG:boto:Port: 8000 2024-03-04T19:09:18.527 DEBUG:boto:Params: {} 2024-03-04T19:09:18.527 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:18.528 DEBUG:boto:Token: None 2024-03-04T19:09:18.528 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:09:18 GMT /ppfahn-72/ 2024-03-04T19:09:18.528 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:09:18.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:09:18.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~110s 2024-03-04T19:09:18.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~110s 2024-03-04T19:09:18.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~110s 2024-03-04T19:09:18.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~110s 2024-03-04T19:09:18.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~110s 2024-03-04T19:09:18.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~110s 2024-03-04T19:09:18.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~110s 2024-03-04T19:09:21.596 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:09:21.596 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:21.928 DEBUG:boto:Token: None 2024-03-04T19:09:21.962 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:09:18 GMT /ppfahn-72/ 2024-03-04T19:09:21.962 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:09:21.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:09:23.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~115s 2024-03-04T19:09:23.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~115s 2024-03-04T19:09:23.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~115s 2024-03-04T19:09:23.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~115s 2024-03-04T19:09:23.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~115s 2024-03-04T19:09:23.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~115s 2024-03-04T19:09:23.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~115s 2024-03-04T19:09:23.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~115s 2024-03-04T19:09:23.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~115s 2024-03-04T19:09:23.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~115s 2024-03-04T19:09:23.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~115s 2024-03-04T19:09:23.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~115s 2024-03-04T19:09:23.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~115s 2024-03-04T19:09:23.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~115s 2024-03-04T19:09:24.668 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:09:24.668 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:26.602 DEBUG:boto:Token: None 2024-03-04T19:09:26.603 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:09:18 GMT /ppfahn-72/ 2024-03-04T19:09:26.603 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:09:26.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:09:27.740 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:09:27.740 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:28.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~120s 2024-03-04T19:09:28.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~120s 2024-03-04T19:09:28.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~120s 2024-03-04T19:09:28.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~120s 2024-03-04T19:09:28.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~120s 2024-03-04T19:09:28.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~120s 2024-03-04T19:09:28.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~120s 2024-03-04T19:09:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~120s 2024-03-04T19:09:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~120s 2024-03-04T19:09:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~120s 2024-03-04T19:09:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~120s 2024-03-04T19:09:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~120s 2024-03-04T19:09:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~120s 2024-03-04T19:09:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~120s 2024-03-04T19:09:31.404 DEBUG:boto:Token: None 2024-03-04T19:09:31.404 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:09:18 GMT /ppfahn-72/ 2024-03-04T19:09:31.404 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:09:31.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:09:33.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~125s 2024-03-04T19:09:33.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~125s 2024-03-04T19:09:33.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~125s 2024-03-04T19:09:33.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~125s 2024-03-04T19:09:33.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~125s 2024-03-04T19:09:33.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~125s 2024-03-04T19:09:33.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~125s 2024-03-04T19:09:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~125s 2024-03-04T19:09:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~125s 2024-03-04T19:09:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~125s 2024-03-04T19:09:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~125s 2024-03-04T19:09:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~125s 2024-03-04T19:09:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~125s 2024-03-04T19:09:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~125s 2024-03-04T19:09:33.884 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:09:33.884 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:35.695 DEBUG:boto:Token: None 2024-03-04T19:09:35.696 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:09:18 GMT /ppfahn-72/ 2024-03-04T19:09:35.696 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:09:35.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:09:36.956 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:09:36.956 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:38.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~130s 2024-03-04T19:09:38.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~130s 2024-03-04T19:09:38.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~130s 2024-03-04T19:09:38.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~130s 2024-03-04T19:09:38.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~130s 2024-03-04T19:09:38.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~130s 2024-03-04T19:09:38.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~130s 2024-03-04T19:09:38.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~130s 2024-03-04T19:09:38.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~130s 2024-03-04T19:09:38.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~130s 2024-03-04T19:09:38.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~130s 2024-03-04T19:09:38.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~130s 2024-03-04T19:09:38.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~130s 2024-03-04T19:09:38.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~130s 2024-03-04T19:09:43.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~135s 2024-03-04T19:09:43.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~135s 2024-03-04T19:09:43.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~135s 2024-03-04T19:09:43.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~135s 2024-03-04T19:09:43.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~135s 2024-03-04T19:09:43.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~135s 2024-03-04T19:09:43.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~135s 2024-03-04T19:09:43.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~135s 2024-03-04T19:09:43.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~135s 2024-03-04T19:09:43.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~135s 2024-03-04T19:09:43.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~135s 2024-03-04T19:09:43.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~135s 2024-03-04T19:09:43.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~135s 2024-03-04T19:09:43.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~135s 2024-03-04T19:09:45.925 DEBUG:boto:Token: None 2024-03-04T19:09:45.925 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:09:18 GMT /ppfahn-72/ 2024-03-04T19:09:45.925 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:09:45.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:09:46.172 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:09:46.172 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:48.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~140s 2024-03-04T19:09:48.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~140s 2024-03-04T19:09:48.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~140s 2024-03-04T19:09:48.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~140s 2024-03-04T19:09:48.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~140s 2024-03-04T19:09:48.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~140s 2024-03-04T19:09:48.440 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~140s 2024-03-04T19:09:48.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~140s 2024-03-04T19:09:48.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~140s 2024-03-04T19:09:48.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~140s 2024-03-04T19:09:48.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~140s 2024-03-04T19:09:48.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~140s 2024-03-04T19:09:48.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~140s 2024-03-04T19:09:48.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~140s 2024-03-04T19:09:49.594 DEBUG:boto:Token: None 2024-03-04T19:09:49.595 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:09:18 GMT /ppfahn-72/ 2024-03-04T19:09:49.595 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:09:49.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:09:52.316 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:09:52.316 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:53.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~145s 2024-03-04T19:09:53.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~145s 2024-03-04T19:09:53.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~145s 2024-03-04T19:09:53.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~145s 2024-03-04T19:09:53.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~145s 2024-03-04T19:09:53.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~145s 2024-03-04T19:09:53.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~145s 2024-03-04T19:09:53.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~145s 2024-03-04T19:09:53.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~145s 2024-03-04T19:09:53.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~145s 2024-03-04T19:09:53.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~145s 2024-03-04T19:09:53.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~145s 2024-03-04T19:09:53.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~145s 2024-03-04T19:09:53.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~145s 2024-03-04T19:09:58.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~150s 2024-03-04T19:09:58.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~150s 2024-03-04T19:09:58.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~150s 2024-03-04T19:09:58.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~150s 2024-03-04T19:09:58.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~150s 2024-03-04T19:09:58.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~150s 2024-03-04T19:09:58.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~150s 2024-03-04T19:09:58.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~150s 2024-03-04T19:09:58.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~150s 2024-03-04T19:09:58.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~150s 2024-03-04T19:09:58.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~150s 2024-03-04T19:09:58.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~150s 2024-03-04T19:09:58.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~150s 2024-03-04T19:09:58.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~150s 2024-03-04T19:10:03.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~155s 2024-03-04T19:10:03.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~155s 2024-03-04T19:10:03.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~155s 2024-03-04T19:10:03.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~155s 2024-03-04T19:10:03.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~155s 2024-03-04T19:10:03.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~155s 2024-03-04T19:10:03.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~155s 2024-03-04T19:10:03.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~155s 2024-03-04T19:10:03.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~155s 2024-03-04T19:10:03.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~155s 2024-03-04T19:10:03.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~155s 2024-03-04T19:10:03.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~155s 2024-03-04T19:10:03.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~155s 2024-03-04T19:10:03.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~155s 2024-03-04T19:10:08.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~160s 2024-03-04T19:10:08.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~160s 2024-03-04T19:10:08.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~160s 2024-03-04T19:10:08.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~160s 2024-03-04T19:10:08.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~160s 2024-03-04T19:10:08.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~160s 2024-03-04T19:10:08.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~160s 2024-03-04T19:10:08.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~160s 2024-03-04T19:10:08.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~160s 2024-03-04T19:10:08.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~160s 2024-03-04T19:10:08.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~160s 2024-03-04T19:10:08.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~160s 2024-03-04T19:10:08.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~160s 2024-03-04T19:10:08.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~160s 2024-03-04T19:10:13.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~165s 2024-03-04T19:10:13.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~165s 2024-03-04T19:10:13.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~165s 2024-03-04T19:10:13.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~165s 2024-03-04T19:10:13.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~165s 2024-03-04T19:10:13.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~165s 2024-03-04T19:10:13.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~165s 2024-03-04T19:10:13.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~165s 2024-03-04T19:10:13.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~165s 2024-03-04T19:10:13.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~165s 2024-03-04T19:10:13.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~165s 2024-03-04T19:10:13.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~165s 2024-03-04T19:10:13.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~165s 2024-03-04T19:10:13.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~165s 2024-03-04T19:10:18.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~170s 2024-03-04T19:10:18.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~170s 2024-03-04T19:10:18.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~170s 2024-03-04T19:10:18.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~170s 2024-03-04T19:10:18.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~170s 2024-03-04T19:10:18.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~170s 2024-03-04T19:10:18.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~170s 2024-03-04T19:10:18.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~170s 2024-03-04T19:10:18.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~170s 2024-03-04T19:10:18.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~170s 2024-03-04T19:10:18.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~170s 2024-03-04T19:10:18.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~170s 2024-03-04T19:10:18.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~170s 2024-03-04T19:10:18.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~170s 2024-03-04T19:10:22.688 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ERROR 2024-03-04T19:10:22.689 DEBUG:boto:path=/ppfahn-73/ 2024-03-04T19:10:22.689 DEBUG:boto:auth_path=/ppfahn-73/ 2024-03-04T19:10:22.689 DEBUG:boto:Method: PUT 2024-03-04T19:10:22.689 DEBUG:boto:Path: /ppfahn-73/ 2024-03-04T19:10:22.689 DEBUG:boto:Data: 2024-03-04T19:10:22.689 DEBUG:boto:Headers: {} 2024-03-04T19:10:22.689 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:10:22.690 DEBUG:boto:Port: 8000 2024-03-04T19:10:22.690 DEBUG:boto:Params: {} 2024-03-04T19:10:22.690 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:10:22.690 DEBUG:boto:Token: None 2024-03-04T19:10:22.690 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:10:22 GMT /ppfahn-73/ 2024-03-04T19:10:22.690 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:10:22.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:10:23.036 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:10:23.036 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:10:23.104 DEBUG:boto:Token: None 2024-03-04T19:10:23.104 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:10:22 GMT /ppfahn-73/ 2024-03-04T19:10:23.104 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:10:23.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:10:23.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~175s 2024-03-04T19:10:23.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~175s 2024-03-04T19:10:23.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~175s 2024-03-04T19:10:23.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~175s 2024-03-04T19:10:23.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~175s 2024-03-04T19:10:23.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~175s 2024-03-04T19:10:23.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~175s 2024-03-04T19:10:23.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~175s 2024-03-04T19:10:23.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~175s 2024-03-04T19:10:23.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~175s 2024-03-04T19:10:23.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~175s 2024-03-04T19:10:23.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~175s 2024-03-04T19:10:23.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~175s 2024-03-04T19:10:23.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~175s 2024-03-04T19:10:26.108 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:10:26.108 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:10:27.043 DEBUG:boto:Token: None 2024-03-04T19:10:27.043 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:10:22 GMT /ppfahn-73/ 2024-03-04T19:10:27.043 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:10:27.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:10:28.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~180s 2024-03-04T19:10:28.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~180s 2024-03-04T19:10:28.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~180s 2024-03-04T19:10:28.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~180s 2024-03-04T19:10:28.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~180s 2024-03-04T19:10:28.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~180s 2024-03-04T19:10:28.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~180s 2024-03-04T19:10:28.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~180s 2024-03-04T19:10:28.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~180s 2024-03-04T19:10:28.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~180s 2024-03-04T19:10:28.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~180s 2024-03-04T19:10:28.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~180s 2024-03-04T19:10:28.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~180s 2024-03-04T19:10:28.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~180s 2024-03-04T19:10:29.180 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:10:29.180 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:10:32.125 DEBUG:boto:Token: None 2024-03-04T19:10:32.126 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:10:22 GMT /ppfahn-73/ 2024-03-04T19:10:32.126 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:10:32.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:10:32.252 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:10:32.252 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:10:33.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~185s 2024-03-04T19:10:33.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~185s 2024-03-04T19:10:33.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~185s 2024-03-04T19:10:33.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~185s 2024-03-04T19:10:33.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~185s 2024-03-04T19:10:33.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~185s 2024-03-04T19:10:33.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~185s 2024-03-04T19:10:33.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~185s 2024-03-04T19:10:33.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~185s 2024-03-04T19:10:33.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~185s 2024-03-04T19:10:33.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~185s 2024-03-04T19:10:33.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~185s 2024-03-04T19:10:33.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~185s 2024-03-04T19:10:33.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~185s 2024-03-04T19:10:33.597 DEBUG:boto:Token: None 2024-03-04T19:10:33.597 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:10:22 GMT /ppfahn-73/ 2024-03-04T19:10:33.597 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:10:33.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:10:35.324 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:10:35.324 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:10:38.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~190s 2024-03-04T19:10:38.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~190s 2024-03-04T19:10:38.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~190s 2024-03-04T19:10:38.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~190s 2024-03-04T19:10:38.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~190s 2024-03-04T19:10:38.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~190s 2024-03-04T19:10:38.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~190s 2024-03-04T19:10:38.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~190s 2024-03-04T19:10:38.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~190s 2024-03-04T19:10:38.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~190s 2024-03-04T19:10:38.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~190s 2024-03-04T19:10:38.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~190s 2024-03-04T19:10:38.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~190s 2024-03-04T19:10:38.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~190s 2024-03-04T19:10:41.102 DEBUG:boto:Token: None 2024-03-04T19:10:41.103 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:10:22 GMT /ppfahn-73/ 2024-03-04T19:10:41.103 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:10:41.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:10:41.468 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:10:41.468 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:10:43.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~195s 2024-03-04T19:10:43.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~195s 2024-03-04T19:10:43.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~195s 2024-03-04T19:10:43.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~195s 2024-03-04T19:10:43.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~195s 2024-03-04T19:10:43.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~195s 2024-03-04T19:10:43.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~195s 2024-03-04T19:10:43.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~195s 2024-03-04T19:10:43.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~195s 2024-03-04T19:10:43.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~195s 2024-03-04T19:10:43.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~195s 2024-03-04T19:10:43.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~195s 2024-03-04T19:10:43.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~195s 2024-03-04T19:10:43.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~195s 2024-03-04T19:10:48.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~200s 2024-03-04T19:10:48.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~200s 2024-03-04T19:10:48.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~200s 2024-03-04T19:10:48.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~200s 2024-03-04T19:10:48.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~200s 2024-03-04T19:10:48.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~200s 2024-03-04T19:10:48.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~200s 2024-03-04T19:10:48.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~200s 2024-03-04T19:10:48.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~200s 2024-03-04T19:10:48.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~200s 2024-03-04T19:10:48.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~200s 2024-03-04T19:10:48.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~200s 2024-03-04T19:10:48.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~200s 2024-03-04T19:10:48.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~200s 2024-03-04T19:10:53.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~205s 2024-03-04T19:10:53.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~205s 2024-03-04T19:10:53.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~205s 2024-03-04T19:10:53.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~205s 2024-03-04T19:10:53.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~205s 2024-03-04T19:10:53.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~205s 2024-03-04T19:10:53.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~205s 2024-03-04T19:10:53.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~205s 2024-03-04T19:10:53.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~205s 2024-03-04T19:10:53.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~205s 2024-03-04T19:10:53.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~205s 2024-03-04T19:10:53.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~205s 2024-03-04T19:10:53.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~205s 2024-03-04T19:10:53.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~205s 2024-03-04T19:10:58.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~210s 2024-03-04T19:10:58.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~210s 2024-03-04T19:10:58.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~210s 2024-03-04T19:10:58.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~210s 2024-03-04T19:10:58.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~210s 2024-03-04T19:10:58.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~210s 2024-03-04T19:10:58.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~210s 2024-03-04T19:10:58.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~210s 2024-03-04T19:10:58.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~210s 2024-03-04T19:10:58.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~210s 2024-03-04T19:10:58.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~210s 2024-03-04T19:10:58.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~210s 2024-03-04T19:10:58.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~210s 2024-03-04T19:10:58.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~210s 2024-03-04T19:10:59.074 DEBUG:boto:Token: None 2024-03-04T19:10:59.075 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:10:22 GMT /ppfahn-73/ 2024-03-04T19:10:59.075 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:10:59.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:10:59.900 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:10:59.900 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:11:03.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~215s 2024-03-04T19:11:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~215s 2024-03-04T19:11:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~215s 2024-03-04T19:11:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~215s 2024-03-04T19:11:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~215s 2024-03-04T19:11:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~215s 2024-03-04T19:11:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~215s 2024-03-04T19:11:03.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~215s 2024-03-04T19:11:03.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~215s 2024-03-04T19:11:03.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~215s 2024-03-04T19:11:03.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~215s 2024-03-04T19:11:03.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~215s 2024-03-04T19:11:03.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~215s 2024-03-04T19:11:03.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~215s 2024-03-04T19:11:08.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~220s 2024-03-04T19:11:08.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~220s 2024-03-04T19:11:08.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~220s 2024-03-04T19:11:08.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~220s 2024-03-04T19:11:08.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~220s 2024-03-04T19:11:08.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~220s 2024-03-04T19:11:08.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~220s 2024-03-04T19:11:08.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~220s 2024-03-04T19:11:08.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~220s 2024-03-04T19:11:08.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~220s 2024-03-04T19:11:08.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~220s 2024-03-04T19:11:08.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~220s 2024-03-04T19:11:08.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~220s 2024-03-04T19:11:08.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~220s 2024-03-04T19:11:13.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~225s 2024-03-04T19:11:13.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~225s 2024-03-04T19:11:13.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~225s 2024-03-04T19:11:13.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~225s 2024-03-04T19:11:13.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~225s 2024-03-04T19:11:13.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~225s 2024-03-04T19:11:13.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~225s 2024-03-04T19:11:13.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~225s 2024-03-04T19:11:13.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~225s 2024-03-04T19:11:13.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~225s 2024-03-04T19:11:13.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~225s 2024-03-04T19:11:13.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~225s 2024-03-04T19:11:13.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~225s 2024-03-04T19:11:13.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~225s 2024-03-04T19:11:18.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~230s 2024-03-04T19:11:18.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~230s 2024-03-04T19:11:18.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~230s 2024-03-04T19:11:18.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~230s 2024-03-04T19:11:18.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~230s 2024-03-04T19:11:18.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~230s 2024-03-04T19:11:18.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~230s 2024-03-04T19:11:18.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~230s 2024-03-04T19:11:18.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~230s 2024-03-04T19:11:18.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~230s 2024-03-04T19:11:18.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~230s 2024-03-04T19:11:18.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~230s 2024-03-04T19:11:18.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~230s 2024-03-04T19:11:18.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~230s 2024-03-04T19:11:23.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~235s 2024-03-04T19:11:23.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~235s 2024-03-04T19:11:23.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~235s 2024-03-04T19:11:23.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~235s 2024-03-04T19:11:23.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~235s 2024-03-04T19:11:23.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~235s 2024-03-04T19:11:23.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~235s 2024-03-04T19:11:23.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~235s 2024-03-04T19:11:23.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~235s 2024-03-04T19:11:23.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~235s 2024-03-04T19:11:23.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~235s 2024-03-04T19:11:23.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~235s 2024-03-04T19:11:23.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~235s 2024-03-04T19:11:23.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~235s 2024-03-04T19:11:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~240s 2024-03-04T19:11:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~240s 2024-03-04T19:11:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~240s 2024-03-04T19:11:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~240s 2024-03-04T19:11:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~240s 2024-03-04T19:11:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~240s 2024-03-04T19:11:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~240s 2024-03-04T19:11:28.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~240s 2024-03-04T19:11:28.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~240s 2024-03-04T19:11:28.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~240s 2024-03-04T19:11:28.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~240s 2024-03-04T19:11:28.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~240s 2024-03-04T19:11:28.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~240s 2024-03-04T19:11:28.582 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~240s 2024-03-04T19:11:30.803 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ERROR 2024-03-04T19:11:30.803 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-ppfahn-1 2024-03-04T19:11:30.804 DEBUG:boto:Method: POST 2024-03-04T19:11:30.804 DEBUG:boto:Path: / 2024-03-04T19:11:30.804 DEBUG:boto:Data: 2024-03-04T19:11:30.804 DEBUG:boto:Headers: {} 2024-03-04T19:11:30.804 DEBUG:boto:Host: smithi136.front.sepia.ceph.com 2024-03-04T19:11:30.804 DEBUG:boto:Port: 8000 2024-03-04T19:11:30.804 DEBUG:boto:Params: {'RoleName': 'roles-ppfahn-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-03-04T19:11:30.804 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:11:30.804 DEBUG:boto:Token: None 2024-03-04T19:11:30.805 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191130Z host;x-amz-date 48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:11:30.805 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191130Z 20240304/front/smithi136/aws4_request db650c0abe8b2146d80bcd55c651e6ae9b841d606e666f2a5bb2516cc2dce797 2024-03-04T19:11:30.805 DEBUG:boto:Signature: 2f8a888dd3f90732a08978bdc60b84f78475f3aa6bef37d9e20550f317c20a41 2024-03-04T19:11:30.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191130Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=2f8a888dd3f90732a08978bdc60b84f78475f3aa6bef37d9e20550f317c20a41', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:11:33.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~245s 2024-03-04T19:11:33.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~245s 2024-03-04T19:11:33.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~245s 2024-03-04T19:11:33.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~245s 2024-03-04T19:11:33.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~245s 2024-03-04T19:11:33.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~245s 2024-03-04T19:11:33.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~245s 2024-03-04T19:11:33.582 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~245s 2024-03-04T19:11:33.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~245s 2024-03-04T19:11:33.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~245s 2024-03-04T19:11:33.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~245s 2024-03-04T19:11:33.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~245s 2024-03-04T19:11:33.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~245s 2024-03-04T19:11:33.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~245s 2024-03-04T19:11:33.884 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:11:33.884 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:11:34.342 DEBUG:boto:Token: None 2024-03-04T19:11:34.342 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191134Z host;x-amz-date 48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:11:34.342 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191134Z 20240304/front/smithi136/aws4_request d13514ace2196259a7ef35dceea287696e7599dfb484e16451c17b66a0ee8ffc 2024-03-04T19:11:34.342 DEBUG:boto:Signature: 9cd841eda09c0f218bdc3ec20397e09591f9226a198ed336077d3e98c5a30104 2024-03-04T19:11:34.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191134Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=9cd841eda09c0f218bdc3ec20397e09591f9226a198ed336077d3e98c5a30104', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:11:36.956 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:11:36.956 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:11:37.096 DEBUG:boto:Token: None 2024-03-04T19:11:37.096 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191137Z host;x-amz-date 48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:11:37.096 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191137Z 20240304/front/smithi136/aws4_request 565affa7ae2bb72e3cb16c240d6140110bcea47faaa9e1a138bd607a56ef8cff 2024-03-04T19:11:37.096 DEBUG:boto:Signature: f13e9a7e21f5c11e24f09c74fac1417942e0a985f4a4a1b7dc9faa2400f2a539 2024-03-04T19:11:37.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191137Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=f13e9a7e21f5c11e24f09c74fac1417942e0a985f4a4a1b7dc9faa2400f2a539', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:11:38.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~250s 2024-03-04T19:11:38.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~250s 2024-03-04T19:11:38.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~250s 2024-03-04T19:11:38.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~250s 2024-03-04T19:11:38.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~250s 2024-03-04T19:11:38.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~250s 2024-03-04T19:11:38.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~250s 2024-03-04T19:11:38.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~250s 2024-03-04T19:11:38.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~250s 2024-03-04T19:11:38.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~250s 2024-03-04T19:11:38.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~250s 2024-03-04T19:11:38.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~250s 2024-03-04T19:11:38.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~250s 2024-03-04T19:11:38.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~250s 2024-03-04T19:11:40.028 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:11:40.028 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:11:43.465 DEBUG:boto:Token: None 2024-03-04T19:11:43.466 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191143Z host;x-amz-date 48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:11:43.466 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191143Z 20240304/front/smithi136/aws4_request 0afb53c28b0bb5d5d702d2b1f8103d8382ff518b1059889e250619145b08f17e 2024-03-04T19:11:43.466 DEBUG:boto:Signature: a90b43bc8ab2dfa25f987793b8f9c93f5e9966112f991ce365ed86569800eb6d 2024-03-04T19:11:43.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191143Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=a90b43bc8ab2dfa25f987793b8f9c93f5e9966112f991ce365ed86569800eb6d', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:11:43.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~255s 2024-03-04T19:11:43.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~255s 2024-03-04T19:11:43.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~255s 2024-03-04T19:11:43.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~255s 2024-03-04T19:11:43.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~255s 2024-03-04T19:11:43.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~255s 2024-03-04T19:11:43.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~255s 2024-03-04T19:11:43.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~255s 2024-03-04T19:11:43.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~255s 2024-03-04T19:11:43.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~255s 2024-03-04T19:11:43.586 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~255s 2024-03-04T19:11:43.586 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~255s 2024-03-04T19:11:43.586 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~255s 2024-03-04T19:11:43.586 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~255s 2024-03-04T19:11:46.524 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:11:46.524 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:11:48.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~260s 2024-03-04T19:11:48.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~260s 2024-03-04T19:11:48.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~260s 2024-03-04T19:11:48.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~260s 2024-03-04T19:11:48.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~260s 2024-03-04T19:11:48.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~260s 2024-03-04T19:11:48.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~260s 2024-03-04T19:11:48.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~260s 2024-03-04T19:11:48.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~260s 2024-03-04T19:11:48.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~260s 2024-03-04T19:11:48.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~260s 2024-03-04T19:11:48.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~260s 2024-03-04T19:11:48.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~260s 2024-03-04T19:11:48.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~260s 2024-03-04T19:11:48.875 DEBUG:boto:Token: None 2024-03-04T19:11:48.876 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191148Z host;x-amz-date 48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:11:48.876 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191148Z 20240304/front/smithi136/aws4_request 069eb1a406a513f6e53d33b9093cdda0753c5681997b6248679984584190b8e5 2024-03-04T19:11:48.876 DEBUG:boto:Signature: b3ed709718ef604fc9758af66d1d9948cfa930a460b8d60db0bf497e46b85541 2024-03-04T19:11:48.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191148Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=b3ed709718ef604fc9758af66d1d9948cfa930a460b8d60db0bf497e46b85541', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:11:49.596 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:11:49.596 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:11:50.212 DEBUG:boto:Token: None 2024-03-04T19:11:50.213 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191150Z host;x-amz-date 48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:11:50.213 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191150Z 20240304/front/smithi136/aws4_request bf780ebbf10fca08c76d4c0ea602d8f48a395467b00c9411c79e012695afa995 2024-03-04T19:11:50.213 DEBUG:boto:Signature: 0e598ab0a95a9f03bf0b9b29c80922938a93c83d6390368ee44373bb947b80a0 2024-03-04T19:11:50.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191150Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=0e598ab0a95a9f03bf0b9b29c80922938a93c83d6390368ee44373bb947b80a0', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:11:53.276 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:11:53.276 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:11:53.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~265s 2024-03-04T19:11:53.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~265s 2024-03-04T19:11:53.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~265s 2024-03-04T19:11:53.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~265s 2024-03-04T19:11:53.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~265s 2024-03-04T19:11:53.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~265s 2024-03-04T19:11:53.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~265s 2024-03-04T19:11:53.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~265s 2024-03-04T19:11:53.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~265s 2024-03-04T19:11:53.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~265s 2024-03-04T19:11:53.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~265s 2024-03-04T19:11:53.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~265s 2024-03-04T19:11:53.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~265s 2024-03-04T19:11:53.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~265s 2024-03-04T19:11:58.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~270s 2024-03-04T19:11:58.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~270s 2024-03-04T19:11:58.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~270s 2024-03-04T19:11:58.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~270s 2024-03-04T19:11:58.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~270s 2024-03-04T19:11:58.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~270s 2024-03-04T19:11:58.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~270s 2024-03-04T19:11:58.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~270s 2024-03-04T19:11:58.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~270s 2024-03-04T19:11:58.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~270s 2024-03-04T19:11:58.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~270s 2024-03-04T19:11:58.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~270s 2024-03-04T19:11:58.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~270s 2024-03-04T19:11:58.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~270s 2024-03-04T19:12:03.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~275s 2024-03-04T19:12:03.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~275s 2024-03-04T19:12:03.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~275s 2024-03-04T19:12:03.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~275s 2024-03-04T19:12:03.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~275s 2024-03-04T19:12:03.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~275s 2024-03-04T19:12:03.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~275s 2024-03-04T19:12:03.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~275s 2024-03-04T19:12:03.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~275s 2024-03-04T19:12:03.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~275s 2024-03-04T19:12:03.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~275s 2024-03-04T19:12:03.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~275s 2024-03-04T19:12:03.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~275s 2024-03-04T19:12:03.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~275s 2024-03-04T19:12:03.651 DEBUG:boto:Token: None 2024-03-04T19:12:03.651 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191203Z host;x-amz-date 48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:12:03.651 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191203Z 20240304/front/smithi136/aws4_request 2e529e361105babfd54b48f4f63ae35569db8922e25559c8892d911b5f254db7 2024-03-04T19:12:03.651 DEBUG:boto:Signature: 1312af3d3e7c933e3b9bc76ffa622fce89f1a47d3c400110fbcec61770675238 2024-03-04T19:12:03.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191203Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=1312af3d3e7c933e3b9bc76ffa622fce89f1a47d3c400110fbcec61770675238', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:12:06.716 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:12:06.716 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:12:08.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~280s 2024-03-04T19:12:08.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~280s 2024-03-04T19:12:08.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~280s 2024-03-04T19:12:08.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~280s 2024-03-04T19:12:08.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~280s 2024-03-04T19:12:08.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~280s 2024-03-04T19:12:08.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~280s 2024-03-04T19:12:08.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~280s 2024-03-04T19:12:08.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~280s 2024-03-04T19:12:08.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~280s 2024-03-04T19:12:08.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~280s 2024-03-04T19:12:08.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~280s 2024-03-04T19:12:08.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~280s 2024-03-04T19:12:08.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~280s 2024-03-04T19:12:13.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~285s 2024-03-04T19:12:13.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~285s 2024-03-04T19:12:13.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~285s 2024-03-04T19:12:13.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~285s 2024-03-04T19:12:13.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~285s 2024-03-04T19:12:13.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~285s 2024-03-04T19:12:13.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~285s 2024-03-04T19:12:13.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~285s 2024-03-04T19:12:13.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~285s 2024-03-04T19:12:13.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~285s 2024-03-04T19:12:13.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~285s 2024-03-04T19:12:13.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~285s 2024-03-04T19:12:13.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~285s 2024-03-04T19:12:13.595 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~285s 2024-03-04T19:12:18.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~290s 2024-03-04T19:12:18.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~290s 2024-03-04T19:12:18.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~290s 2024-03-04T19:12:18.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~290s 2024-03-04T19:12:18.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~290s 2024-03-04T19:12:18.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~290s 2024-03-04T19:12:18.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~290s 2024-03-04T19:12:18.595 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~290s 2024-03-04T19:12:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~290s 2024-03-04T19:12:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~290s 2024-03-04T19:12:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~290s 2024-03-04T19:12:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~290s 2024-03-04T19:12:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~290s 2024-03-04T19:12:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~290s 2024-03-04T19:12:23.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~295s 2024-03-04T19:12:23.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~295s 2024-03-04T19:12:23.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~295s 2024-03-04T19:12:23.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~295s 2024-03-04T19:12:23.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~295s 2024-03-04T19:12:23.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~295s 2024-03-04T19:12:23.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~295s 2024-03-04T19:12:23.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~295s 2024-03-04T19:12:23.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~295s 2024-03-04T19:12:23.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~295s 2024-03-04T19:12:23.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~295s 2024-03-04T19:12:23.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~295s 2024-03-04T19:12:23.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~295s 2024-03-04T19:12:23.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~295s 2024-03-04T19:12:28.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~300s 2024-03-04T19:12:28.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~300s 2024-03-04T19:12:28.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~300s 2024-03-04T19:12:28.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~300s 2024-03-04T19:12:28.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~300s 2024-03-04T19:12:28.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~300s 2024-03-04T19:12:28.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~300s 2024-03-04T19:12:28.601 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-03-04T19:12:28.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~300s 2024-03-04T19:12:28.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~300s 2024-03-04T19:12:28.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~300s 2024-03-04T19:12:28.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~300s 2024-03-04T19:12:28.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~300s 2024-03-04T19:12:28.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~300s 2024-03-04T19:12:28.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~300s 2024-03-04T19:12:28.602 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-03-04T19:12:48.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ERROR 2024-03-04T19:12:48.769 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-ppfahn-2 2024-03-04T19:12:48.769 DEBUG:boto:Method: POST 2024-03-04T19:12:48.769 DEBUG:boto:Path: / 2024-03-04T19:12:48.769 DEBUG:boto:Data: 2024-03-04T19:12:48.770 DEBUG:boto:Headers: {} 2024-03-04T19:12:48.770 DEBUG:boto:Host: smithi136.front.sepia.ceph.com 2024-03-04T19:12:48.770 DEBUG:boto:Port: 8000 2024-03-04T19:12:48.770 DEBUG:boto:Params: {'RoleName': 'roles-ppfahn-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-03-04T19:12:48.770 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:12:48.770 DEBUG:boto:Token: None 2024-03-04T19:12:48.770 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191248Z host;x-amz-date fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:12:48.771 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191248Z 20240304/front/smithi136/aws4_request ee3c9737efd5084c82e2f6eeffd42776b5acc202eef5a9b48c8320b2646bc0ec 2024-03-04T19:12:48.771 DEBUG:boto:Signature: 2c3467418782f009d34b48160aab250cbfe15511f8369f9d597f705e52be08d5 2024-03-04T19:12:48.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191248Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=2c3467418782f009d34b48160aab250cbfe15511f8369f9d597f705e52be08d5', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:12:51.836 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:12:51.836 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:12:52.586 DEBUG:boto:Token: None 2024-03-04T19:12:52.587 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191252Z host;x-amz-date fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:12:52.587 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191252Z 20240304/front/smithi136/aws4_request 536cb27699bed49c4b9f8a7ab1b31463548804b63c2411b1e5e95776bb121250 2024-03-04T19:12:52.587 DEBUG:boto:Signature: 54b25f4fdb77413c3fdf181ac7f31e775d17008d2db56ae7cd4419297c151035 2024-03-04T19:12:52.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191252Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=54b25f4fdb77413c3fdf181ac7f31e775d17008d2db56ae7cd4419297c151035', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:12:54.908 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:12:54.908 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:12:56.897 DEBUG:boto:Token: None 2024-03-04T19:12:56.897 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191256Z host;x-amz-date fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:12:56.897 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191256Z 20240304/front/smithi136/aws4_request 522ba95f941726b8f32c9974e15f456cafc2e2f35eb517b018e5ac26be8c149d 2024-03-04T19:12:56.897 DEBUG:boto:Signature: 77f854100519f42f590abc0eb1c45eaa55776e32d0eba60731dc4c4e605a16f5 2024-03-04T19:12:56.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191256Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=77f854100519f42f590abc0eb1c45eaa55776e32d0eba60731dc4c4e605a16f5', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:12:57.980 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:12:57.980 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:13:00.405 DEBUG:boto:Token: None 2024-03-04T19:13:00.405 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191300Z host;x-amz-date fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:13:00.406 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191300Z 20240304/front/smithi136/aws4_request b577d00d7a93a894458bb7702a53fe3ff58ddd3965c9503ab1f7a6583af4fb50 2024-03-04T19:13:00.406 DEBUG:boto:Signature: 9a22a13d7e143051c28eb16b9f235e8aa4d3f6859e857cedce3d63ba1778c1ef 2024-03-04T19:13:00.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191300Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=9a22a13d7e143051c28eb16b9f235e8aa4d3f6859e857cedce3d63ba1778c1ef', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:13:01.052 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:13:01.052 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:13:08.821 DEBUG:boto:Token: None 2024-03-04T19:13:08.822 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191308Z host;x-amz-date fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:13:08.822 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191308Z 20240304/front/smithi136/aws4_request d924dd08c5be0fe43ad607ad9a2578321db978280dbc17279222485bec7594b2 2024-03-04T19:13:08.822 DEBUG:boto:Signature: e16a5ccdab6c3d9e9e8c62afe44eaabdcc448c54de731bdb55c1f1ccfdcc7d7c 2024-03-04T19:13:08.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191308Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=e16a5ccdab6c3d9e9e8c62afe44eaabdcc448c54de731bdb55c1f1ccfdcc7d7c', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:13:11.900 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:13:11.900 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:13:15.299 DEBUG:boto:Token: None 2024-03-04T19:13:15.299 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191315Z host;x-amz-date fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:13:15.300 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191315Z 20240304/front/smithi136/aws4_request 15ce56a58cfbeab88fbd7a2ffea5a5a6cc4705322f347dfc5e4e5d1c5490379c 2024-03-04T19:13:15.300 DEBUG:boto:Signature: bd21d8b35cf033c2660dd2c48c56bec2e6a8eb222042bf5584b8927ea970b05f 2024-03-04T19:13:15.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191315Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=bd21d8b35cf033c2660dd2c48c56bec2e6a8eb222042bf5584b8927ea970b05f', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:13:18.364 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:13:18.364 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:13:29.356 DEBUG:boto:Token: None 2024-03-04T19:13:29.357 DEBUG:boto:CanonicalRequest: POST / host:smithi136.front.sepia.ceph.com:8000 x-amz-date:20240304T191329Z host;x-amz-date fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:13:29.357 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T191329Z 20240304/front/smithi136/aws4_request 8fa06c5bedd01fc61baf8e5a1b6230e55284b2b012552ea1a8d507bba1cf50cb 2024-03-04T19:13:29.357 DEBUG:boto:Signature: 060b053167a1781ea5ccee6dd8c2f18d3e10fe9ae52c5e67ff9f3e1ba36d4197 2024-03-04T19:13:29.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191329Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=060b053167a1781ea5ccee6dd8c2f18d3e10fe9ae52c5e67ff9f3e1ba36d4197', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:13:32.412 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:13:32.412 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:13:38.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_delete_sync ... ERROR 2024-03-04T19:13:38.284 DEBUG:boto:path=/ppfahn-74/ 2024-03-04T19:13:38.284 DEBUG:boto:auth_path=/ppfahn-74/ 2024-03-04T19:13:38.284 DEBUG:boto:Method: PUT 2024-03-04T19:13:38.284 DEBUG:boto:Path: /ppfahn-74/ 2024-03-04T19:13:38.285 DEBUG:boto:Data: 2024-03-04T19:13:38.285 DEBUG:boto:Headers: {} 2024-03-04T19:13:38.285 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:13:38.285 DEBUG:boto:Port: 8000 2024-03-04T19:13:38.285 DEBUG:boto:Params: {} 2024-03-04T19:13:38.285 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:13:38.285 DEBUG:boto:Token: None 2024-03-04T19:13:38.285 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:13:38 GMT /ppfahn-74/ 2024-03-04T19:13:38.285 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:13:38.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:13:41.340 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:13:41.340 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:13:41.774 DEBUG:boto:Token: None 2024-03-04T19:13:41.774 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:13:38 GMT /ppfahn-74/ 2024-03-04T19:13:41.774 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:13:41.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:13:44.413 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:13:44.413 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:13:45.632 DEBUG:boto:Token: None 2024-03-04T19:13:45.633 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:13:38 GMT /ppfahn-74/ 2024-03-04T19:13:45.633 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:13:45.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:13:47.484 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:13:47.484 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:13:50.264 DEBUG:boto:Token: None 2024-03-04T19:13:50.265 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:13:38 GMT /ppfahn-74/ 2024-03-04T19:13:50.265 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:13:50.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:13:50.556 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:13:50.556 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:13:56.275 DEBUG:boto:Token: None 2024-03-04T19:13:56.275 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:13:38 GMT /ppfahn-74/ 2024-03-04T19:13:56.275 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:13:56.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:13:59.356 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:13:59.356 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:14:04.492 DEBUG:boto:Token: None 2024-03-04T19:14:04.492 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:13:38 GMT /ppfahn-74/ 2024-03-04T19:14:04.492 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:14:04.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:14:07.548 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:14:07.548 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:14:08.704 DEBUG:boto:Token: None 2024-03-04T19:14:08.704 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:13:38 GMT /ppfahn-74/ 2024-03-04T19:14:08.704 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:14:08.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:14:10.620 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:14:10.620 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:14:39.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ERROR 2024-03-04T19:14:39.247 DEBUG:boto:path=/ppfahn-75/ 2024-03-04T19:14:39.248 DEBUG:boto:auth_path=/ppfahn-75/ 2024-03-04T19:14:39.248 DEBUG:boto:Method: PUT 2024-03-04T19:14:39.248 DEBUG:boto:Path: /ppfahn-75/ 2024-03-04T19:14:39.248 DEBUG:boto:Data: 2024-03-04T19:14:39.248 DEBUG:boto:Headers: {} 2024-03-04T19:14:39.248 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:14:39.248 DEBUG:boto:Port: 8000 2024-03-04T19:14:39.248 DEBUG:boto:Params: {} 2024-03-04T19:14:39.248 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:14:39.248 DEBUG:boto:Token: None 2024-03-04T19:14:39.249 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:14:39 GMT /ppfahn-75/ 2024-03-04T19:14:39.249 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:14:39.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:14:42.300 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:14:42.300 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:14:42.703 DEBUG:boto:Token: None 2024-03-04T19:14:42.703 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:14:39 GMT /ppfahn-75/ 2024-03-04T19:14:42.703 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:14:42.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:14:45.372 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:14:45.372 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:14:47.362 DEBUG:boto:Token: None 2024-03-04T19:14:47.362 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:14:39 GMT /ppfahn-75/ 2024-03-04T19:14:47.362 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:14:47.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:14:48.444 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:14:48.444 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:14:51.247 DEBUG:boto:Token: None 2024-03-04T19:14:51.248 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:14:39 GMT /ppfahn-75/ 2024-03-04T19:14:51.248 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:14:51.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:14:51.516 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:14:51.516 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:14:55.766 DEBUG:boto:Token: None 2024-03-04T19:14:55.767 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:14:39 GMT /ppfahn-75/ 2024-03-04T19:14:55.767 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:14:55.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:14:58.844 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:14:58.844 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:15:07.716 DEBUG:boto:Token: None 2024-03-04T19:15:07.716 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:14:39 GMT /ppfahn-75/ 2024-03-04T19:15:07.716 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:15:07.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:15:10.780 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:15:10.780 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:15:22.117 DEBUG:boto:Token: None 2024-03-04T19:15:22.118 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:14:39 GMT /ppfahn-75/ 2024-03-04T19:15:22.119 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:15:22.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:15:25.180 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:15:25.180 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:15:52.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ERROR 2024-03-04T19:15:52.320 DEBUG:boto:path=/ppfahn-76/ 2024-03-04T19:15:52.321 DEBUG:boto:auth_path=/ppfahn-76/ 2024-03-04T19:15:52.321 DEBUG:boto:Method: PUT 2024-03-04T19:15:52.321 DEBUG:boto:Path: /ppfahn-76/ 2024-03-04T19:15:52.321 DEBUG:boto:Data: 2024-03-04T19:15:52.321 DEBUG:boto:Headers: {} 2024-03-04T19:15:52.321 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:15:52.321 DEBUG:boto:Port: 8000 2024-03-04T19:15:52.321 DEBUG:boto:Params: {} 2024-03-04T19:15:52.321 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:15:52.321 DEBUG:boto:Token: None 2024-03-04T19:15:52.322 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:15:52 GMT /ppfahn-76/ 2024-03-04T19:15:52.322 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:15:52.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:15:55.388 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:15:55.388 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:15:55.620 DEBUG:boto:Token: None 2024-03-04T19:15:55.620 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:15:52 GMT /ppfahn-76/ 2024-03-04T19:15:55.620 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:15:55.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:15:58.460 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:15:58.460 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:15:59.149 DEBUG:boto:Token: None 2024-03-04T19:15:59.150 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:15:52 GMT /ppfahn-76/ 2024-03-04T19:15:59.150 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:15:59.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:16:01.532 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:16:01.532 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:16:02.860 DEBUG:boto:Token: None 2024-03-04T19:16:02.861 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:15:52 GMT /ppfahn-76/ 2024-03-04T19:16:02.861 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:16:02.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:16:04.604 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:16:04.604 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:16:05.562 DEBUG:boto:Token: None 2024-03-04T19:16:05.562 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:15:52 GMT /ppfahn-76/ 2024-03-04T19:16:05.563 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:16:05.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:16:07.676 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:16:07.676 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:16:12.024 DEBUG:boto:Token: None 2024-03-04T19:16:12.024 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:15:52 GMT /ppfahn-76/ 2024-03-04T19:16:12.025 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:16:12.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:16:15.100 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:16:15.100 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:16:29.683 DEBUG:boto:Token: None 2024-03-04T19:16:29.684 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:15:52 GMT /ppfahn-76/ 2024-03-04T19:16:29.684 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:16:29.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:16:32.764 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:16:32.764 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:16:48.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ERROR 2024-03-04T19:16:48.476 DEBUG:boto:path=/ppfahn-77/ 2024-03-04T19:16:48.476 DEBUG:boto:auth_path=/ppfahn-77/ 2024-03-04T19:16:48.476 DEBUG:boto:Method: PUT 2024-03-04T19:16:48.477 DEBUG:boto:Path: /ppfahn-77/ 2024-03-04T19:16:48.477 DEBUG:boto:Data: 2024-03-04T19:16:48.477 DEBUG:boto:Headers: {} 2024-03-04T19:16:48.477 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:16:48.477 DEBUG:boto:Port: 8000 2024-03-04T19:16:48.477 DEBUG:boto:Params: {} 2024-03-04T19:16:48.477 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:16:48.477 DEBUG:boto:Token: None 2024-03-04T19:16:48.477 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:16:48 GMT /ppfahn-77/ 2024-03-04T19:16:48.478 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:16:48.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:16:51.548 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:16:51.549 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:16:51.708 DEBUG:boto:Token: None 2024-03-04T19:16:51.708 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:16:48 GMT /ppfahn-77/ 2024-03-04T19:16:51.708 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:16:51.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:16:54.620 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:16:54.620 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:16:54.959 DEBUG:boto:Token: None 2024-03-04T19:16:54.959 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:16:48 GMT /ppfahn-77/ 2024-03-04T19:16:54.959 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:16:54.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:16:57.692 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:16:57.692 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:16:59.277 DEBUG:boto:Token: None 2024-03-04T19:16:59.278 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:16:48 GMT /ppfahn-77/ 2024-03-04T19:16:59.278 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:16:59.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:17:00.764 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:17:00.765 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:17:06.288 DEBUG:boto:Token: None 2024-03-04T19:17:06.288 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:16:48 GMT /ppfahn-77/ 2024-03-04T19:17:06.289 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:17:06.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:17:09.340 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:17:09.340 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:17:22.764 DEBUG:boto:Token: None 2024-03-04T19:17:22.764 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:16:48 GMT /ppfahn-77/ 2024-03-04T19:17:22.764 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:17:22.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:17:25.820 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:17:25.820 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:17:42.527 DEBUG:boto:Token: None 2024-03-04T19:17:42.527 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:16:48 GMT /ppfahn-77/ 2024-03-04T19:17:42.528 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:17:42.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:17:45.596 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:17:45.596 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:18:24.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... ERROR 2024-03-04T19:18:24.249 DEBUG:boto:path=/ppfahn-78/ 2024-03-04T19:18:24.249 DEBUG:boto:auth_path=/ppfahn-78/ 2024-03-04T19:18:24.249 DEBUG:boto:Method: PUT 2024-03-04T19:18:24.249 DEBUG:boto:Path: /ppfahn-78/ 2024-03-04T19:18:24.249 DEBUG:boto:Data: 2024-03-04T19:18:24.249 DEBUG:boto:Headers: {} 2024-03-04T19:18:24.249 DEBUG:boto:Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:18:24.249 DEBUG:boto:Port: 8000 2024-03-04T19:18:24.249 DEBUG:boto:Params: {} 2024-03-04T19:18:24.250 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:18:24.250 DEBUG:boto:Token: None 2024-03-04T19:18:24.250 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:18:24 GMT /ppfahn-78/ 2024-03-04T19:18:24.250 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:18:24.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:18:27.324 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:18:27.325 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:18:28.122 DEBUG:boto:Token: None 2024-03-04T19:18:28.122 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:18:24 GMT /ppfahn-78/ 2024-03-04T19:18:28.122 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:18:28.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:18:30.396 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:18:30.396 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:18:31.088 DEBUG:boto:Token: None 2024-03-04T19:18:31.088 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:18:24 GMT /ppfahn-78/ 2024-03-04T19:18:31.088 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:18:31.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:18:33.468 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:18:33.468 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:18:36.044 DEBUG:boto:Token: None 2024-03-04T19:18:36.044 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:18:24 GMT /ppfahn-78/ 2024-03-04T19:18:36.045 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:18:36.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:18:36.540 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:18:36.540 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:18:37.019 DEBUG:boto:Token: None 2024-03-04T19:18:37.019 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:18:24 GMT /ppfahn-78/ 2024-03-04T19:18:37.019 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:18:37.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:18:40.092 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:18:40.092 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:18:52.174 DEBUG:boto:Token: None 2024-03-04T19:18:52.175 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:18:24 GMT /ppfahn-78/ 2024-03-04T19:18:52.175 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:18:52.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:18:55.228 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:18:55.228 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:18:57.416 DEBUG:boto:Token: None 2024-03-04T19:18:57.416 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 19:18:24 GMT /ppfahn-78/ 2024-03-04T19:18:57.416 DEBUG:boto:Signature: AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:18:57.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:18:58.300 DEBUG:boto:encountered OSError exception, reconnecting 2024-03-04T19:18:58.300 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:19:24.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... ERROR 2024-03-04T19:19:24.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:19:24.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:19:24.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:19:24.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:19:27.228 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:19:36.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:19:36.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:19:39.292 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:19:51.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:19:51.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:19:54.364 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:19:57.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:19:57.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:19:57.436 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:19:57.436 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:19:57.437 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ERROR 2024-03-04T19:19:57.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:19:57.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:19:57.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:19:57.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:20:00.508 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:20:09.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:20:09.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:20:12.572 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:20:24.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:20:24.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:20:27.644 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:20:30.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:20:30.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:20:30.716 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:20:30.716 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:20:30.717 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... ERROR 2024-03-04T19:20:30.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:20:30.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:20:30.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:20:30.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:20:33.788 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:20:42.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:20:42.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:20:45.852 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:20:57.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:20:57.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:21:00.924 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:21:03.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:21:03.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:21:03.996 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:21:03.996 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:21:03.997 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... ERROR 2024-03-04T19:21:03.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:21:03.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:21:03.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:21:03.998 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:21:07.068 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:21:16.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:21:16.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:21:19.132 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:21:31.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:21:31.145 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:21:34.204 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:21:37.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:21:37.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:21:37.276 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:21:37.276 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:21:37.277 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... ERROR 2024-03-04T19:21:37.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:21:37.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:21:37.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:21:37.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:21:40.352 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:21:49.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:21:49.364 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:21:52.444 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:22:04.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:22:04.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:22:07.516 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:22:10.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:22:10.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:22:10.588 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:22:10.588 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:22:10.589 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ERROR 2024-03-04T19:22:10.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:22:10.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:22:10.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:22:10.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:22:13.660 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:22:22.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:22:22.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:22:25.724 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:22:37.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:22:37.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:22:40.796 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:22:43.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:22:43.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:22:43.868 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:22:43.868 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:22:43.869 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ERROR 2024-03-04T19:22:43.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:22:43.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:22:43.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:22:43.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:22:46.940 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:22:55.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:22:55.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:22:59.004 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:11.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:11.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:14.076 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:17.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:17.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:17.148 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:17.148 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:23:17.149 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... ERROR 2024-03-04T19:23:17.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:17.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:17.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:17.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:20.220 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:29.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:29.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:32.284 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:44.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:44.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:47.356 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.428 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.428 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:23:50.429 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ERROR 2024-03-04T19:23:50.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create_rgw_down ... SKIP: test_bucket_create_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove_rgw_down ... SKIP: test_bucket_remove_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync_rgw_down ... SKIP: test_object_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete_rgw_down ... SKIP: test_object_delete_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync_rgw_down ... SKIP: test_concurrent_versioned_object_incremental_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync_rgw_down ... SKIP: test_suspended_delete_marker_full_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl_rgw_down ... SKIP: test_bucket_acl_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable_rgw_down ... SKIP: test_bucket_sync_enable_right_after_disable_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync_rgw_down ... SKIP: test_multipart_object_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_run_basic_incremental_rgw_down ... SKIP: test_bucket_sync_run_basic_incremental_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync_rgw_down ... SKIP: test_role_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init_rgw_down ... SKIP: test_bucket_full_sync_after_data_sync_init_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_sync_policy_config_zonegroup_rgw_down ... SKIP: test_sync_policy_config_zonegroup_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_sync_flow_symmetrical_zonegroup_all_rgw_down ... SKIP: test_sync_flow_symmetrical_zonegroup_all_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:23:50.434 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.435 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.435 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_reshard_full 2024-03-04T19:23:50.435 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.435 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.435 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.435 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.435 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1507, in test_bucket_reshard_full 2024-03-04T19:23:50.435 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:50.436 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:50.436 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:50.436 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 185, in zone_meta_checkpoint 2024-03-04T19:23:50.436 INFO:tasks.rgw_multisite_tests: global_status, period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2024-03-04T19:23:50.436 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 139, in meta_sync_status 2024-03-04T19:23:50.436 INFO:tasks.rgw_multisite_tests: meta_sync_status_json, retcode = zone.cluster.admin(cmd, check_retcode=False, read_only=True) 2024-03-04T19:23:50.436 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:23:50.436 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:23:50.436 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:23:50.436 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:23:50.436 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:23:50.436 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:23:50.436 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi153 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-71/ 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-71/ 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-71/ 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.437 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:06:05 GMT 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests:/ppfahn-71/ 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:GDDnhuSJPo2m98Rcy9XPZ5HsE0g= 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:06:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GDDnhuSJPo2m98Rcy9XPZ5HsE0g=', 'Content-Length': '0'} 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000097f1b3349a94121-0065e61b9d-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 19:06:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=ppfahn-71 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:50.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709579165.276086_654.1","last_update":"2024-03-04T19:06:05.276086Z"}] 2024-03-04T19:23:50.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:50.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:05.806+0000 7ff2ebd97980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:05.806+0000 7ff2ebd97980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:05Z"} 2024-03-04T19:23:50.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:05Z"} 2024-03-04T19:23:50.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:50.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:50.440 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.440 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:10.950+0000 7f8979ca2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:10.954+0000 7f8979ca2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:10Z"} 2024-03-04T19:23:50.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:10Z"} 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:16.094+0000 7f74d8ff7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:16.094+0000 7f74d8ff7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:16Z"} 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:16Z"} 2024-03-04T19:23:50.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:21.230+0000 7f7963c46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:21.234+0000 7f7963c46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:21Z"} 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:21Z"} 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:26.366+0000 7fac5e984980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:26.366+0000 7fac5e984980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:26Z"} 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:26Z"} 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.292440_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.292440Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.292440_304.1'} 2024-03-04T19:23:50.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.292440_304.1 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:31.506+0000 7f82afc73980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:31.506+0000 7f82afc73980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:31Z"} 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:31Z"} 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:50.444 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:36.638+0000 7f6b3b394980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:36.638+0000 7f6b3b394980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:36Z"} 2024-03-04T19:23:50.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:36Z"} 2024-03-04T19:23:50.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:50.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:50.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:41.790+0000 7f72febc9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:41.794+0000 7f72febc9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:41Z"} 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:41Z"} 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T19:06:46.703+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:50.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:46.985+0000 7fef77f4b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:46.989+0000 7fef77f4b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:46Z"} 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:46Z"} 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:52.121+0000 7f66af4cd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:52.125+0000 7f66af4cd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:52Z"} 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:52Z"} 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:57.293+0000 7f4b582bf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:57.293+0000 7f4b582bf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.815737_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.815737Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:57Z"} 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.815737_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.815737Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:57Z"} 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.815737_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.815737Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.815737_334.1'} 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575077.815737_334.1 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:07:02.445+0000 7f1ce082c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:07:02.445+0000 7f1ce082c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:02Z"} 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:02Z"} 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:07:07.581+0000 7f4333a53980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:07:07.585+0000 7f4333a53980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:07Z"} 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:07Z"} 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:07:12.761+0000 7f1a895b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:07:12.761+0000 7f1a895b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:12Z"} 2024-03-04T19:23:50.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:12Z"} 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:07:17.913+0000 7f5e4fe02980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:07:17.913+0000 7f5e4fe02980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:17Z"} 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:17Z"} 2024-03-04T19:23:50.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:07:23.061+0000 7fc36ac10980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:07:23.061+0000 7fc36ac10980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:23Z"} 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:23Z"} 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709579165.276086_654.1'} 2024-03-04T19:23:50.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709579165.276086_654.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~0s 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~0s 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~0s 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~0s 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~0s 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~0s 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~0s 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~0s 2024-03-04T19:23:50.454 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~0s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~0s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~0s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~0s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~0s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~5s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~5s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~5s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~5s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~5s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~5s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~5s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~5s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~5s 2024-03-04T19:23:50.455 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~5s 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~5s 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~5s 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~5s 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~5s 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~10s 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~10s 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~10s 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~10s 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~10s 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~10s 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~10s 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~10s 2024-03-04T19:23:50.456 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~10s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~10s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~10s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~10s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~10s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~10s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~15s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~15s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~15s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~15s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~15s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~15s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~15s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~15s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~15s 2024-03-04T19:23:50.457 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~15s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~15s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~15s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~15s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~15s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~20s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~20s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~20s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~20s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~20s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~20s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~20s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~20s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~20s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~20s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~20s 2024-03-04T19:23:50.458 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~20s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~20s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~20s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~25s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~25s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~25s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~25s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~25s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~25s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~25s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~25s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~25s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~25s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~25s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~25s 2024-03-04T19:23:50.459 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~25s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~25s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~30s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~30s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~30s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~30s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~30s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~30s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~30s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~30s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~30s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~30s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~30s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~30s 2024-03-04T19:23:50.460 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~30s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~30s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~35s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~35s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~35s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~35s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~35s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~35s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~35s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~35s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~35s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~35s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~35s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~35s 2024-03-04T19:23:50.461 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~35s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~35s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~40s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~40s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~40s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~40s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~40s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~40s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~40s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~40s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~40s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~40s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~40s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~40s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~40s 2024-03-04T19:23:50.462 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~40s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~45s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~45s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~45s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~45s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~45s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~45s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~45s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~45s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~45s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~45s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~45s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~45s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~45s 2024-03-04T19:23:50.463 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~45s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~50s 2024-03-04T19:23:50.464 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~55s 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:23:50.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~60s 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~60s 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~60s 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~60s 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~60s 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~60s 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~60s 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~60s 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~60s 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~60s 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~60s 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~60s 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~60s 2024-03-04T19:23:50.466 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~60s 2024-03-04T19:23:50.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:23:50.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:23:50.467 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:23:50.467 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.467 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.467 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.467 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_creation_time 2024-03-04T19:23:50.467 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.467 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.467 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.467 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.467 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1540, in test_bucket_creation_time 2024-03-04T19:23:50.468 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:50.468 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-03-04T19:23:50.468 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-03-04T19:23:50.468 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-03-04T19:23:50.468 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-03-04T19:23:50.468 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:23:50.468 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:23:50.468 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:23:50.468 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:23:50.468 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:23:50.468 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:23:50.468 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:23:50.468 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~65s 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~65s 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~65s 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~65s 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~65s 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~65s 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~65s 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~65s 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~65s 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~65s 2024-03-04T19:23:50.469 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~65s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~65s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~65s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~65s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~70s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~70s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~70s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~70s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~70s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~70s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~70s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~70s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~70s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~70s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~70s 2024-03-04T19:23:50.470 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~70s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~70s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~70s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~75s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~75s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~75s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~75s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~75s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~75s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~75s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~75s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~75s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~75s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~75s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~75s 2024-03-04T19:23:50.471 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~75s 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~75s 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~80s 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~80s 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~80s 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~80s 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~80s 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~80s 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~80s 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~80s 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~80s 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~80s 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~80s 2024-03-04T19:23:50.472 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~80s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~80s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~80s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~85s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~85s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~85s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~85s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~85s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~85s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~85s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~85s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~85s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~85s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~85s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~85s 2024-03-04T19:23:50.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~85s 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~85s 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~90s 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~90s 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~90s 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~90s 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~90s 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~90s 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~90s 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~90s 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~90s 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~90s 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~90s 2024-03-04T19:23:50.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~90s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~90s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~90s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~95s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~95s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~95s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~95s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~95s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~95s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~95s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~95s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~95s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~95s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~95s 2024-03-04T19:23:50.475 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~95s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~95s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~95s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~100s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~100s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~100s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~100s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~100s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~100s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~100s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~100s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~100s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~100s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~100s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~100s 2024-03-04T19:23:50.476 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~100s 2024-03-04T19:23:50.477 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~100s 2024-03-04T19:23:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.477 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~105s 2024-03-04T19:23:50.477 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~105s 2024-03-04T19:23:50.477 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~105s 2024-03-04T19:23:50.477 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~105s 2024-03-04T19:23:50.477 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~105s 2024-03-04T19:23:50.477 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~105s 2024-03-04T19:23:50.477 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~105s 2024-03-04T19:23:50.477 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~105s 2024-03-04T19:23:50.477 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~105s 2024-03-04T19:23:50.477 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~105s 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~105s 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~105s 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~105s 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~105s 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~110s 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~110s 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~110s 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~110s 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~110s 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~110s 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~110s 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.479 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:23:50.479 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.479 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.479 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.479 INFO:tasks.rgw_multisite_tests:ERROR: Create several generations of objects, then run bucket sync 2024-03-04T19:23:50.479 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.479 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.479 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.479 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.479 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1623, in test_bucket_sync_run_basic_incremental 2024-03-04T19:23:50.479 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-03-04T19:23:50.479 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-03-04T19:23:50.480 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-03-04T19:23:50.480 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-03-04T19:23:50.480 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-03-04T19:23:50.480 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-03-04T19:23:50.480 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-03-04T19:23:50.480 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-03-04T19:23:50.480 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-03-04T19:23:50.480 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-03-04T19:23:50.480 INFO:tasks.rgw_multisite_tests: raise ex 2024-03-04T19:23:50.480 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-03-04T19:23:50.480 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-03-04T19:23:50.480 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-03-04T19:23:50.480 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: self.connect() 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-03-04T19:23:50.481 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests:OSError: [Errno 113] No route to host 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-72/ 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-72/ 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-72/ 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:09:18 GMT 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests:/ppfahn-72/ 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~110s 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~110s 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~110s 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~110s 2024-03-04T19:23:50.483 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~110s 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~110s 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~110s 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:09:18 GMT 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests:/ppfahn-72/ 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:23:50.484 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~115s 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:09:18 GMT 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests:/ppfahn-72/ 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~120s 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~120s 2024-03-04T19:23:50.486 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~120s 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~120s 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~120s 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~120s 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~120s 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~120s 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~120s 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~120s 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~120s 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~120s 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~120s 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~120s 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.487 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:09:18 GMT 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:/ppfahn-72/ 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~125s 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~125s 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~125s 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~125s 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~125s 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~125s 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~125s 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~125s 2024-03-04T19:23:50.488 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~125s 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~125s 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~125s 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~125s 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~125s 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~125s 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:09:18 GMT 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests:/ppfahn-72/ 2024-03-04T19:23:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~130s 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~130s 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~130s 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~130s 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~130s 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~130s 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~130s 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~130s 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~130s 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~130s 2024-03-04T19:23:50.490 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~130s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~130s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~130s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~130s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~135s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~135s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~135s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~135s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~135s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~135s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~135s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~135s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~135s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~135s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~135s 2024-03-04T19:23:50.491 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~135s 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~135s 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~135s 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:09:18 GMT 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests:/ppfahn-72/ 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.492 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~140s 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:09:18 GMT 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests:/ppfahn-72/ 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E= 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:09:18 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:didWZqBkFpIrIKoUTg9Xv79Wv6E=', 'Content-Length': '0'} 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~145s 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~145s 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~145s 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~145s 2024-03-04T19:23:50.494 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~145s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~145s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~145s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~145s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~145s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~145s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~145s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~145s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~145s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~145s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~150s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~150s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~150s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~150s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~150s 2024-03-04T19:23:50.495 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~150s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~150s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~150s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~150s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~150s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~150s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~150s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~150s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~150s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~155s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~155s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~155s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~155s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~155s 2024-03-04T19:23:50.496 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~155s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~155s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~155s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~155s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~155s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~155s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~155s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~155s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~155s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~160s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~160s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~160s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~160s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~160s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~160s 2024-03-04T19:23:50.497 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~160s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~160s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~160s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~160s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~160s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~160s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~160s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~160s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~165s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~165s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~165s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~165s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~165s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~165s 2024-03-04T19:23:50.498 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~165s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~165s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~165s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~165s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~165s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~165s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~165s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~165s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~170s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~170s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~170s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~170s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~170s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~170s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~170s 2024-03-04T19:23:50.499 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~170s 2024-03-04T19:23:50.500 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~170s 2024-03-04T19:23:50.500 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~170s 2024-03-04T19:23:50.500 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~170s 2024-03-04T19:23:50.500 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~170s 2024-03-04T19:23:50.500 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~170s 2024-03-04T19:23:50.500 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~170s 2024-03-04T19:23:50.500 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.500 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.500 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.500 INFO:tasks.rgw_multisite_tests:ERROR: Create several generations of objects, trash them, then run bucket sync init 2024-03-04T19:23:50.500 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1686, in test_zap_init_bucket_sync_run 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-03-04T19:23:50.501 INFO:tasks.rgw_multisite_tests: raise ex 2024-03-04T19:23:50.502 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-03-04T19:23:50.502 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-03-04T19:23:50.502 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-03-04T19:23:50.502 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-03-04T19:23:50.502 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-03-04T19:23:50.502 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.502 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-03-04T19:23:50.502 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.502 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-03-04T19:23:50.502 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-03-04T19:23:50.502 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-03-04T19:23:50.502 INFO:tasks.rgw_multisite_tests: self.connect() 2024-03-04T19:23:50.502 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-03-04T19:23:50.503 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-03-04T19:23:50.503 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-03-04T19:23:50.503 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-03-04T19:23:50.503 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-03-04T19:23:50.503 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-03-04T19:23:50.503 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-03-04T19:23:50.503 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-03-04T19:23:50.503 INFO:tasks.rgw_multisite_tests:OSError: [Errno 113] No route to host 2024-03-04T19:23:50.503 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-73/ 2024-03-04T19:23:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-73/ 2024-03-04T19:23:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-73/ 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:10:22 GMT 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests:/ppfahn-73/ 2024-03-04T19:23:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:10:22 GMT 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests:/ppfahn-73/ 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:23:50.505 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~175s 2024-03-04T19:23:50.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:10:22 GMT 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests:/ppfahn-73/ 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~180s 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~180s 2024-03-04T19:23:50.507 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~180s 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~180s 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~180s 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~180s 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~180s 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~180s 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~180s 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~180s 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~180s 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~180s 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~180s 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~180s 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:10:22 GMT 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests:/ppfahn-73/ 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~185s 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~185s 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~185s 2024-03-04T19:23:50.509 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~185s 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~185s 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~185s 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~185s 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~185s 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~185s 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~185s 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~185s 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~185s 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~185s 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~185s 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.510 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:10:22 GMT 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:/ppfahn-73/ 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~190s 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~190s 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~190s 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~190s 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~190s 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~190s 2024-03-04T19:23:50.511 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~190s 2024-03-04T19:23:50.512 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~190s 2024-03-04T19:23:50.512 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~190s 2024-03-04T19:23:50.512 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~190s 2024-03-04T19:23:50.512 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~190s 2024-03-04T19:23:50.512 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~190s 2024-03-04T19:23:50.512 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~190s 2024-03-04T19:23:50.512 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~190s 2024-03-04T19:23:50.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.512 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.512 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.512 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.512 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:10:22 GMT 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:/ppfahn-73/ 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~195s 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~195s 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~195s 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~195s 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~195s 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~195s 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~195s 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~195s 2024-03-04T19:23:50.513 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~195s 2024-03-04T19:23:50.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~195s 2024-03-04T19:23:50.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~195s 2024-03-04T19:23:50.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~195s 2024-03-04T19:23:50.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~195s 2024-03-04T19:23:50.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~195s 2024-03-04T19:23:50.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~200s 2024-03-04T19:23:50.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~200s 2024-03-04T19:23:50.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~200s 2024-03-04T19:23:50.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~200s 2024-03-04T19:23:50.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~200s 2024-03-04T19:23:50.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~200s 2024-03-04T19:23:50.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~200s 2024-03-04T19:23:50.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~200s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~200s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~200s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~200s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~200s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~200s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~200s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~205s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~205s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~205s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~205s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~205s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~205s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~205s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~205s 2024-03-04T19:23:50.515 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~205s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~205s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~205s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~205s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~205s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~205s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~210s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~210s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~210s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~210s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~210s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~210s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~210s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~210s 2024-03-04T19:23:50.516 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~210s 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~210s 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~210s 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~210s 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~210s 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~210s 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:10:22 GMT 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:/ppfahn-73/ 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8= 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:10:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QaGZ1HQKAVQJ8AyBPlUDCQa4qv8=', 'Content-Length': '0'} 2024-03-04T19:23:50.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~215s 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~215s 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~215s 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~215s 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~215s 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~215s 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~215s 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~215s 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~215s 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~215s 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~215s 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~215s 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~215s 2024-03-04T19:23:50.518 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~215s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~220s 2024-03-04T19:23:50.519 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~225s 2024-03-04T19:23:50.520 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~230s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~235s 2024-03-04T19:23:50.521 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~235s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~235s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~235s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~235s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~235s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~235s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~235s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~235s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~235s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~235s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~235s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~235s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~235s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~240s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~240s 2024-03-04T19:23:50.522 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~240s 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~240s 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~240s 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~240s 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~240s 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~240s 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~240s 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~240s 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~240s 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~240s 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~240s 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~240s 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.523 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_role_sync 2024-03-04T19:23:50.524 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.524 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.524 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.524 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.524 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1733, in test_role_sync 2024-03-04T19:23:50.524 INFO:tasks.rgw_multisite_tests: roles, zone_role = create_role_per_zone(zonegroup_conns) 2024-03-04T19:23:50.524 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 513, in create_role_per_zone 2024-03-04T19:23:50.524 INFO:tasks.rgw_multisite_tests: role = zone.create_role("", role_name, policy_document, "") 2024-03-04T19:23:50.524 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multi/zone_rados.py", line 133, in create_role 2024-03-04T19:23:50.524 INFO:tasks.rgw_multisite_tests: return self.iam_conn.create_role(rolename, policy_document, path) 2024-03-04T19:23:50.524 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/iam/connection.py", line 1155, in create_role 2024-03-04T19:23:50.524 INFO:tasks.rgw_multisite_tests: return self.get_response('CreateRole', params) 2024-03-04T19:23:50.524 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/iam/connection.py", line 85, in get_response 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: response = self.make_request(action, params, path, verb) 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1116, in make_request 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request) 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: raise ex 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-03-04T19:23:50.525 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests: self.connect() 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests:OSError: [Errno 113] No route to host 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create role zone=test-zone1 name=roles-ppfahn-1 2024-03-04T19:23:50.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: / 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {'RoleName': 'roles-ppfahn-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191130Z 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.527 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:20240304T191130Z 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:db650c0abe8b2146d80bcd55c651e6ae9b841d606e666f2a5bb2516cc2dce797 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:2f8a888dd3f90732a08978bdc60b84f78475f3aa6bef37d9e20550f317c20a41 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191130Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=2f8a888dd3f90732a08978bdc60b84f78475f3aa6bef37d9e20550f317c20a41', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~245s 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~245s 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~245s 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~245s 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~245s 2024-03-04T19:23:50.528 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~245s 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~245s 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~245s 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~245s 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~245s 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~245s 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~245s 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~245s 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~245s 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.529 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191134Z 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests:48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests:20240304T191134Z 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests:d13514ace2196259a7ef35dceea287696e7599dfb484e16451c17b66a0ee8ffc 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests:9cd841eda09c0f218bdc3ec20397e09591f9226a198ed336077d3e98c5a30104 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191134Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=9cd841eda09c0f218bdc3ec20397e09591f9226a198ed336077d3e98c5a30104', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191137Z 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:20240304T191137Z 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.531 INFO:tasks.rgw_multisite_tests:565affa7ae2bb72e3cb16c240d6140110bcea47faaa9e1a138bd607a56ef8cff 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:f13e9a7e21f5c11e24f09c74fac1417942e0a985f4a4a1b7dc9faa2400f2a539 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191137Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=f13e9a7e21f5c11e24f09c74fac1417942e0a985f4a4a1b7dc9faa2400f2a539', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~250s 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~250s 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~250s 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~250s 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~250s 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~250s 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~250s 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~250s 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~250s 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~250s 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~250s 2024-03-04T19:23:50.532 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~250s 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~250s 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~250s 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191143Z 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests:48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:23:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:20240304T191143Z 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:0afb53c28b0bb5d5d702d2b1f8103d8382ff518b1059889e250619145b08f17e 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:a90b43bc8ab2dfa25f987793b8f9c93f5e9966112f991ce365ed86569800eb6d 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191143Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=a90b43bc8ab2dfa25f987793b8f9c93f5e9966112f991ce365ed86569800eb6d', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~255s 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~255s 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~255s 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~255s 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~255s 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~255s 2024-03-04T19:23:50.534 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~255s 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~255s 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~255s 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~255s 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~255s 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~255s 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~255s 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~255s 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~260s 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~260s 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~260s 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~260s 2024-03-04T19:23:50.535 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~260s 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~260s 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~260s 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~260s 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~260s 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~260s 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~260s 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~260s 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~260s 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~260s 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.536 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191148Z 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests:48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests:20240304T191148Z 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests:069eb1a406a513f6e53d33b9093cdda0753c5681997b6248679984584190b8e5 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests:b3ed709718ef604fc9758af66d1d9948cfa930a460b8d60db0bf497e46b85541 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191148Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=b3ed709718ef604fc9758af66d1d9948cfa930a460b8d60db0bf497e46b85541', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.538 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.538 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.538 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.538 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.538 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191150Z 2024-03-04T19:23:50.538 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.538 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.538 INFO:tasks.rgw_multisite_tests:48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:23:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.539 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.539 INFO:tasks.rgw_multisite_tests:20240304T191150Z 2024-03-04T19:23:50.539 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.539 INFO:tasks.rgw_multisite_tests:bf780ebbf10fca08c76d4c0ea602d8f48a395467b00c9411c79e012695afa995 2024-03-04T19:23:50.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.539 INFO:tasks.rgw_multisite_tests:0e598ab0a95a9f03bf0b9b29c80922938a93c83d6390368ee44373bb947b80a0 2024-03-04T19:23:50.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191150Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=0e598ab0a95a9f03bf0b9b29c80922938a93c83d6390368ee44373bb947b80a0', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.539 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~265s 2024-03-04T19:23:50.539 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~265s 2024-03-04T19:23:50.539 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~265s 2024-03-04T19:23:50.539 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~265s 2024-03-04T19:23:50.540 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~265s 2024-03-04T19:23:50.540 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~265s 2024-03-04T19:23:50.540 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~265s 2024-03-04T19:23:50.540 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~265s 2024-03-04T19:23:50.540 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~265s 2024-03-04T19:23:50.540 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~265s 2024-03-04T19:23:50.540 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~265s 2024-03-04T19:23:50.540 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~265s 2024-03-04T19:23:50.540 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~265s 2024-03-04T19:23:50.540 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~265s 2024-03-04T19:23:50.540 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~270s 2024-03-04T19:23:50.540 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~270s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~270s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~270s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~270s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~270s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~270s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~270s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~270s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~270s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~270s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~270s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~270s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~270s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~275s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~275s 2024-03-04T19:23:50.541 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~275s 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~275s 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~275s 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~275s 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~275s 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~275s 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~275s 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~275s 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~275s 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~275s 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~275s 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~275s 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.542 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191203Z 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests:48c92bfbaace0103878c20b0db8b285df513201c4850da85efbd9875a4ea2b64 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests:20240304T191203Z 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests:2e529e361105babfd54b48f4f63ae35569db8922e25559c8892d911b5f254db7 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests:1312af3d3e7c933e3b9bc76ffa622fce89f1a47d3c400110fbcec61770675238 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191203Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=1312af3d3e7c933e3b9bc76ffa622fce89f1a47d3c400110fbcec61770675238', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.544 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~280s 2024-03-04T19:23:50.544 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~280s 2024-03-04T19:23:50.544 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~280s 2024-03-04T19:23:50.544 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~280s 2024-03-04T19:23:50.544 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~280s 2024-03-04T19:23:50.544 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~280s 2024-03-04T19:23:50.544 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~280s 2024-03-04T19:23:50.544 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~280s 2024-03-04T19:23:50.544 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~280s 2024-03-04T19:23:50.544 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~280s 2024-03-04T19:23:50.544 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~280s 2024-03-04T19:23:50.544 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~280s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~280s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~280s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~285s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~285s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~285s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~285s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~285s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~285s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~285s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~285s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~285s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~285s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~285s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~285s 2024-03-04T19:23:50.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~285s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~285s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~290s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~290s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~290s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~290s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~290s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~290s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~290s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~290s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~290s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~290s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~290s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~290s 2024-03-04T19:23:50.546 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~290s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~290s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~295s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~295s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~295s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~295s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~295s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~295s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~295s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~295s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~295s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~295s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~295s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~295s 2024-03-04T19:23:50.547 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~295s 2024-03-04T19:23:50.548 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~295s 2024-03-04T19:23:50.548 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~300s 2024-03-04T19:23:50.548 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~300s 2024-03-04T19:23:50.548 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~300s 2024-03-04T19:23:50.548 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~300s 2024-03-04T19:23:50.548 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~300s 2024-03-04T19:23:50.548 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~300s 2024-03-04T19:23:50.548 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~300s 2024-03-04T19:23:50.548 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: BARK! unmounting mounts and killing all daemons 2024-03-04T19:23:50.548 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~300s 2024-03-04T19:23:50.548 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~300s 2024-03-04T19:23:50.548 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~300s 2024-03-04T19:23:50.548 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~300s 2024-03-04T19:23:50.549 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~300s 2024-03-04T19:23:50.549 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~300s 2024-03-04T19:23:50.549 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~300s 2024-03-04T19:23:50.549 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: BARK! unmounting mounts and killing all daemons 2024-03-04T19:23:50.549 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.549 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.549 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.549 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_role_delete_sync 2024-03-04T19:23:50.549 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.549 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.549 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.550 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1749, in test_role_delete_sync 2024-03-04T19:23:50.550 INFO:tasks.rgw_multisite_tests: zonegroup_conns.master_zone.create_role("", role_name, None, "") 2024-03-04T19:23:50.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multi/zone_rados.py", line 133, in create_role 2024-03-04T19:23:50.550 INFO:tasks.rgw_multisite_tests: return self.iam_conn.create_role(rolename, policy_document, path) 2024-03-04T19:23:50.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/iam/connection.py", line 1155, in create_role 2024-03-04T19:23:50.550 INFO:tasks.rgw_multisite_tests: return self.get_response('CreateRole', params) 2024-03-04T19:23:50.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/iam/connection.py", line 85, in get_response 2024-03-04T19:23:50.550 INFO:tasks.rgw_multisite_tests: response = self.make_request(action, params, path, verb) 2024-03-04T19:23:50.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1116, in make_request 2024-03-04T19:23:50.550 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request) 2024-03-04T19:23:50.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-03-04T19:23:50.550 INFO:tasks.rgw_multisite_tests: raise ex 2024-03-04T19:23:50.551 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-03-04T19:23:50.551 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-03-04T19:23:50.551 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-03-04T19:23:50.551 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-03-04T19:23:50.551 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-03-04T19:23:50.551 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.551 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-03-04T19:23:50.551 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.551 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-03-04T19:23:50.551 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-03-04T19:23:50.551 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-03-04T19:23:50.551 INFO:tasks.rgw_multisite_tests: self.connect() 2024-03-04T19:23:50.552 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-03-04T19:23:50.552 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-03-04T19:23:50.552 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-03-04T19:23:50.552 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-03-04T19:23:50.552 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-03-04T19:23:50.552 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-03-04T19:23:50.552 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-03-04T19:23:50.552 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-03-04T19:23:50.552 INFO:tasks.rgw_multisite_tests:OSError: [Errno 113] No route to host 2024-03-04T19:23:50.552 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create role zone=test-zone1 name=roles-ppfahn-2 2024-03-04T19:23:50.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: / 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {'RoleName': 'roles-ppfahn-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191248Z 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.553 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:20240304T191248Z 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:ee3c9737efd5084c82e2f6eeffd42776b5acc202eef5a9b48c8320b2646bc0ec 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:2c3467418782f009d34b48160aab250cbfe15511f8369f9d597f705e52be08d5 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191248Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=2c3467418782f009d34b48160aab250cbfe15511f8369f9d597f705e52be08d5', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.554 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191252Z 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests:fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests:20240304T191252Z 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests:536cb27699bed49c4b9f8a7ab1b31463548804b63c2411b1e5e95776bb121250 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests:54b25f4fdb77413c3fdf181ac7f31e775d17008d2db56ae7cd4419297c151035 2024-03-04T19:23:50.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191252Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=54b25f4fdb77413c3fdf181ac7f31e775d17008d2db56ae7cd4419297c151035', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191256Z 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests:fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.556 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:20240304T191256Z 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:522ba95f941726b8f32c9974e15f456cafc2e2f35eb517b018e5ac26be8c149d 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:77f854100519f42f590abc0eb1c45eaa55776e32d0eba60731dc4c4e605a16f5 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191256Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=77f854100519f42f590abc0eb1c45eaa55776e32d0eba60731dc4c4e605a16f5', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.557 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191300Z 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:20240304T191300Z 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:b577d00d7a93a894458bb7702a53fe3ff58ddd3965c9503ab1f7a6583af4fb50 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:9a22a13d7e143051c28eb16b9f235e8aa4d3f6859e857cedce3d63ba1778c1ef 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191300Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=9a22a13d7e143051c28eb16b9f235e8aa4d3f6859e857cedce3d63ba1778c1ef', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191308Z 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests:fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests:20240304T191308Z 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests:d924dd08c5be0fe43ad607ad9a2578321db978280dbc17279222485bec7594b2 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.559 INFO:tasks.rgw_multisite_tests:e16a5ccdab6c3d9e9e8c62afe44eaabdcc448c54de731bdb55c1f1ccfdcc7d7c 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191308Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=e16a5ccdab6c3d9e9e8c62afe44eaabdcc448c54de731bdb55c1f1ccfdcc7d7c', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191315Z 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests:fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:23:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:20240304T191315Z 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:15ce56a58cfbeab88fbd7a2ffea5a5a6cc4705322f347dfc5e4e5d1c5490379c 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:bd21d8b35cf033c2660dd2c48c56bec2e6a8eb222042bf5584b8927ea970b05f 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191315Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=bd21d8b35cf033c2660dd2c48c56bec2e6a8eb222042bf5584b8927ea970b05f', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.561 INFO:tasks.rgw_multisite_tests:host:smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T191329Z 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:fe7464cddeb69ed0465177e5ccf5c717626c7007d6c2cd94beb4fd64f895a52e 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:20240304T191329Z 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:20240304/front/smithi136/aws4_request 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:8fa06c5bedd01fc61baf8e5a1b6230e55284b2b012552ea1a8d507bba1cf50cb 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:060b053167a1781ea5ccee6dd8c2f18d3e10fe9ae52c5e67ff9f3e1ba36d4197 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T191329Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=HBI3JS1W1675M40GMG1A/20240304/front/smithi136/aws4_request,SignedHeaders=host;x-amz-date,Signature=060b053167a1781ea5ccee6dd8c2f18d3e10fe9ae52c5e67ff9f3e1ba36d4197', 'Host': 'smithi136.front.sepia.ceph.com:8000'} 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.562 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.563 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.563 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.563 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_full_sync_after_data_sync_init 2024-03-04T19:23:50.563 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.563 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.563 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.563 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.563 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1775, in test_bucket_full_sync_after_data_sync_init 2024-03-04T19:23:50.563 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-03-04T19:23:50.563 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-03-04T19:23:50.563 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-03-04T19:23:50.563 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-03-04T19:23:50.563 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-03-04T19:23:50.564 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-03-04T19:23:50.564 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-03-04T19:23:50.564 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-03-04T19:23:50.564 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-03-04T19:23:50.564 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-03-04T19:23:50.564 INFO:tasks.rgw_multisite_tests: raise ex 2024-03-04T19:23:50.564 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-03-04T19:23:50.564 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-03-04T19:23:50.564 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-03-04T19:23:50.564 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-03-04T19:23:50.564 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-03-04T19:23:50.564 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.564 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests: self.connect() 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests:OSError: [Errno 113] No route to host 2024-03-04T19:23:50.565 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-74/ 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-74/ 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-74/ 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.566 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:13:38 GMT 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:/ppfahn-74/ 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:13:38 GMT 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:/ppfahn-74/ 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.567 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:13:38 GMT 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests:/ppfahn-74/ 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:23:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:13:38 GMT 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests:/ppfahn-74/ 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:13:38 GMT 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests:/ppfahn-74/ 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.570 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.571 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:13:38 GMT 2024-03-04T19:23:50.571 INFO:tasks.rgw_multisite_tests:/ppfahn-74/ 2024-03-04T19:23:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.571 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:23:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:23:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.571 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.571 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.571 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.571 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:13:38 GMT 2024-03-04T19:23:50.572 INFO:tasks.rgw_multisite_tests:/ppfahn-74/ 2024-03-04T19:23:50.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.572 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw= 2024-03-04T19:23:50.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:13:38 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+9Xo7pdZ7097fLDQV1rvDwhZ3sw=', 'Content-Length': '0'} 2024-03-04T19:23:50.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.572 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.572 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.572 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.572 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init 2024-03-04T19:23:50.572 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.572 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.573 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.573 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.573 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1807, in test_resharded_bucket_full_sync_after_data_sync_init 2024-03-04T19:23:50.573 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-03-04T19:23:50.573 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-03-04T19:23:50.573 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-03-04T19:23:50.573 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-03-04T19:23:50.573 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-03-04T19:23:50.573 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-03-04T19:23:50.573 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-03-04T19:23:50.573 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-03-04T19:23:50.573 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-03-04T19:23:50.573 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: raise ex 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: self.connect() 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-03-04T19:23:50.574 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests:OSError: [Errno 113] No route to host 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-75/ 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-75/ 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-75/ 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:14:39 GMT 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests:/ppfahn-75/ 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:14:39 GMT 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests:/ppfahn-75/ 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.577 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:14:39 GMT 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests:/ppfahn-75/ 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:14:39 GMT 2024-03-04T19:23:50.578 INFO:tasks.rgw_multisite_tests:/ppfahn-75/ 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:14:39 GMT 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests:/ppfahn-75/ 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:23:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:14:39 GMT 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests:/ppfahn-75/ 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:14:39 GMT 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests:/ppfahn-75/ 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM= 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:14:39 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Sixp4SYDI5FGvicIve/0lHdqjrM=', 'Content-Length': '0'} 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.581 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.582 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init 2024-03-04T19:23:50.582 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.582 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.582 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.582 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.582 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1846, in test_bucket_incremental_sync_after_data_sync_init 2024-03-04T19:23:50.582 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-03-04T19:23:50.582 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-03-04T19:23:50.582 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-03-04T19:23:50.582 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-03-04T19:23:50.582 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-03-04T19:23:50.582 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-03-04T19:23:50.582 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: raise ex 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-03-04T19:23:50.583 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests: self.connect() 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests:OSError: [Errno 113] No route to host 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-76/ 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-76/ 2024-03-04T19:23:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-76/ 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:15:52 GMT 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests:/ppfahn-76/ 2024-03-04T19:23:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:15:52 GMT 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests:/ppfahn-76/ 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:23:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:15:52 GMT 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests:/ppfahn-76/ 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.588 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.588 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.588 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.588 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:15:52 GMT 2024-03-04T19:23:50.588 INFO:tasks.rgw_multisite_tests:/ppfahn-76/ 2024-03-04T19:23:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.588 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:23:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:23:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:15:52 GMT 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests:/ppfahn-76/ 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.589 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:15:52 GMT 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests:/ppfahn-76/ 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:15:52 GMT 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests:/ppfahn-76/ 2024-03-04T19:23:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.591 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM= 2024-03-04T19:23:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:15:52 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jMF+QfUs29lZ1jsGV1GsYInBzCM=', 'Content-Length': '0'} 2024-03-04T19:23:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.591 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.591 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.591 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.591 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-03-04T19:23:50.591 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.591 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.591 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.591 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.591 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1884, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: raise ex 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-03-04T19:23:50.592 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: self.connect() 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-03-04T19:23:50.593 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests:OSError: [Errno 113] No route to host 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-77/ 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-77/ 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-77/ 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:16:48 GMT 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests:/ppfahn-77/ 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.595 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:16:48 GMT 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests:/ppfahn-77/ 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.596 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:16:48 GMT 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests:/ppfahn-77/ 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:16:48 GMT 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests:/ppfahn-77/ 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.597 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:16:48 GMT 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests:/ppfahn-77/ 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:16:48 GMT 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests:/ppfahn-77/ 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.599 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.600 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.600 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.600 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:16:48 GMT 2024-03-04T19:23:50.600 INFO:tasks.rgw_multisite_tests:/ppfahn-77/ 2024-03-04T19:23:50.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.600 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8= 2024-03-04T19:23:50.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:16:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JjRKKkM8/566cFHAf4VTxXDO+m8=', 'Content-Length': '0'} 2024-03-04T19:23:50.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.600 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.600 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.600 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1932, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-03-04T19:23:50.601 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: raise ex 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: self.connect() 2024-03-04T19:23:50.602 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests:OSError: [Errno 113] No route to host 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-78/ 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-78/ 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-78/ 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:18:24 GMT 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests:/ppfahn-78/ 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:23:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:18:24 GMT 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests:/ppfahn-78/ 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:18:24 GMT 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests:/ppfahn-78/ 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.606 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:18:24 GMT 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests:/ppfahn-78/ 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.607 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:18:24 GMT 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests:/ppfahn-78/ 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:18:24 GMT 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests:/ppfahn-78/ 2024-03-04T19:23:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:18:24 GMT 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests:/ppfahn-78/ 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA= 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:18:24 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:99RmqJ2WxcmbrMNj0AsaMjo/XAA=', 'Content-Length': '0'} 2024-03-04T19:23:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered OSError exception, reconnecting 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests:ERROR: test_sync_policy_config_zonegroup: 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 2124, in test_sync_policy_config_zonegroup 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-03-04T19:23:50.610 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.612 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:23:50.612 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.612 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.612 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests:ERROR: test_sync_flow_symmetrical_zonegroup_all: 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 2196, in test_sync_flow_symmetrical_zonegroup_all 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:23:50.613 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:23:50.614 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:23:50.614 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:23:50.614 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:23:50.614 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.615 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:23:50.615 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.615 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.615 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.615 INFO:tasks.rgw_multisite_tests:ERROR: test_sync_flow_symmetrical_zonegroup_select: 2024-03-04T19:23:50.615 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.615 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.615 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 2259, in test_sync_flow_symmetrical_zonegroup_select 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:23:50.616 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests:ERROR: test_sync_flow_directional_zonegroup_select: 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 2329, in test_sync_flow_directional_zonegroup_select 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-03-04T19:23:50.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.620 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:23:50.620 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.620 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.620 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests:ERROR: test_sync_single_bucket: 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 2442, in test_sync_single_bucket 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:23:50.621 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.623 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:23:50.623 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.623 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.623 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.623 INFO:tasks.rgw_multisite_tests:ERROR: test_sync_different_buckets: 2024-03-04T19:23:50.623 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.623 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.623 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.623 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 2559, in test_sync_different_buckets 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:23:50.624 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.626 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:23:50.626 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.626 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.626 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.626 INFO:tasks.rgw_multisite_tests:ERROR: test_sync_multiple_buckets_to_single: 2024-03-04T19:23:50.626 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.626 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.626 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.626 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.626 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 2705, in test_sync_multiple_buckets_to_single 2024-03-04T19:23:50.626 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:50.626 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.628 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests:ERROR: test_sync_single_bucket_to_multiple: 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 2826, in test_sync_single_bucket_to_multiple 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:23:50.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:23:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:50.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:23:50.631 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:23:50.631 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:50.631 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.631 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:50.631 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_object_delete 2024-03-04T19:23:50.631 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:50.635 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:50.635 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:50.635 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 690, in test_multi_object_delete 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 389, in zone_bucket_checkpoint 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multisite.py", line 197, in admin 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=ppfahn-16 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:SZAEeLRPDDyXLmuFHftQL/jfxdE= 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SZAEeLRPDDyXLmuFHftQL/jfxdE=', 'Content-Length': '0'} 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ba544c897f8b488f-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=ppfahn-17 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.638 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:myYNfZl6maRqmkUu/nPs3WVZCGk= 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:myYNfZl6maRqmkUu/nPs3WVZCGk=', 'Content-Length': '0'} 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e33b23da7274a63a-0065e6079b-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=ppfahn-18 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:QWzbcXMDCWkAz/SXVm00zzHTNXs= 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QWzbcXMDCWkAz/SXVm00zzHTNXs=', 'Content-Length': '0'} 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000065dfcc1df8260b69-0065e6079b-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bfb44596a9d55c9-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj1 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj1 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj1 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.642 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj1 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:x06qphqdzidadB88XUxpVA6Dvlo= 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:x06qphqdzidadB88XUxpVA6Dvlo='} 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e89f7614d7ac46f-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014bb6c8666b6894b-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj2 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj2 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj2 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj2 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.645 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:lS3q+vs1zI0NXmZXY/UnICOIi7k= 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lS3q+vs1zI0NXmZXY/UnICOIi7k='} 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae60b74d65c464f5-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.646 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c4e0b659e66e95b-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj3 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj3 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj3 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.648 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.648 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.648 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.648 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.648 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj3 2024-03-04T19:23:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.648 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:CVWEtZAxF8hXlHA/gu06cp9VFlA= 2024-03-04T19:23:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CVWEtZAxF8hXlHA/gu06cp9VFlA='} 2024-03-04T19:23:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d858ee910bcc9751-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.649 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a26b20e4c4cf2356-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj4 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj4 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj4 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj4 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:IdId9DvdMYKcGG0PR5cavmxARFE= 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:IdId9DvdMYKcGG0PR5cavmxARFE='} 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030f6a38bdcfc9ffd-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000045ec365c162b89-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj5 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj5 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj5 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.653 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj5 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:tPxPb8+eWk6NBgy6zFn3J3bzACw= 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tPxPb8+eWk6NBgy6zFn3J3bzACw='} 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000719e5f8562aaa9e0-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fa05ce393e5a320-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj6 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj6 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj6 2024-03-04T19:23:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj6 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:VADglL/9xO9RUMHvdJ6wn1kQrq8= 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VADglL/9xO9RUMHvdJ6wn1kQrq8='} 2024-03-04T19:23:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e70c17c179aaa190-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.657 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a807b8ed05df750d-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj7 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj7 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj7 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj7 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:LB/E52PbEB1lZwlCoe8lll24Ekg= 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LB/E52PbEB1lZwlCoe8lll24Ekg='} 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae8607658889d40b-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097163db60e045f20-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj8 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj8 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj8 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.661 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj8 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:ZA4xG/plcG6CKkhy6Q9rIQ0dRjs= 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZA4xG/plcG6CKkhy6Q9rIQ0dRjs='} 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000700f9374dcf73a02-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd8181700cafd230-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj9 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj9 2024-03-04T19:23:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj9 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj9 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.664 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:B+W+7a1FThloZRIUDr70JxEOlYQ= 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:B+W+7a1FThloZRIUDr70JxEOlYQ='} 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef0daa0897f9474c-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.665 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b06ce9473add85ad-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj10 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj10 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj10 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj10 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:4ox8PXztuTPbHZo/XqJIPlsZE4Y= 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4ox8PXztuTPbHZo/XqJIPlsZE4Y='} 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0ad8071bae3e74f-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.668 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e36a61c02cc93ce7-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj11 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj11 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj11 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.669 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj11 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:7+t+WTj8YhJ+lwVisopKdCjdG54= 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7+t+WTj8YhJ+lwVisopKdCjdG54='} 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8e2cea3e15061d4-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006669ffd4d88f6237-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj12 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj12 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj12 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.672 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj12 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:jJGBJO4H2HvpPd0i2cAte2sfaCI= 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jJGBJO4H2HvpPd0i2cAte2sfaCI='} 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b02bafae4789d5d6-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9d9f506a166e962-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj13 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj13 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj13 2024-03-04T19:23:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj13 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:nLEdPd6nuql3/4YG1dW+tXVwtEQ= 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nLEdPd6nuql3/4YG1dW+tXVwtEQ='} 2024-03-04T19:23:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8eebd9480098b7d-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.676 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0a83cc007ba817b-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj14 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj14 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj14 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj14 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:HM2LOwczJG6m0MHjd7RwCGkMZng= 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HM2LOwczJG6m0MHjd7RwCGkMZng='} 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030a20ca2fb4fc54c-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000260c5b9b52737620-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj15 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj15 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj15 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.680 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj15 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:hVgmCZtO9/3iTYhN1p1PUDSsyIA= 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hVgmCZtO9/3iTYhN1p1PUDSsyIA='} 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005d04d2e5db520e7e-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d99646244b5863ba-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj16 2024-03-04T19:23:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj16 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj16 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj16 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.683 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:kiPpSN7eTkyfX0gibWc9tDxBAQg= 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kiPpSN7eTkyfX0gibWc9tDxBAQg='} 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b15796409537333d-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.684 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg= 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:OPD2YmnyaXcvVU8OZrmW/6+Y+xg=', 'Content-Length': '0'} 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002179f53e15c0438e-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj17 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj17 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj17 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.686 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.686 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.686 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.686 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:43 GMT 2024-03-04T19:23:50.686 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj17 2024-03-04T19:23:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.686 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:CZgw0ag+p01pvqzbz+7emVawm5s= 2024-03-04T19:23:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:43 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CZgw0ag+p01pvqzbz+7emVawm5s='} 2024-03-04T19:23:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf5ae0bef14a1964-0065e6079b-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.687 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e858e624a794995-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj18 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj18 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj18 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj18 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:eaLAyg0bVqT3QsLlSHSoC3pBcgw= 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eaLAyg0bVqT3QsLlSHSoC3pBcgw='} 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8d8f5157424aa1a-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.690 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.690 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.690 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.690 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.690 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000859adddbde4dc888-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj19 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj19 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj19 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj19 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:VS7Ti011Wuckv+ymtFAHjqUGhvg= 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VS7Ti011Wuckv+ymtFAHjqUGhvg='} 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021d31ecccfca329a-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000980143f4962a44-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj20 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj20 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj20 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.694 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj20 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:+siqT5DcDhL4Q7CoQef8yldtGJ0= 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+siqT5DcDhL4Q7CoQef8yldtGJ0='} 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e9665b84d883873-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a55d1de3f33906b7-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj21 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj21 2024-03-04T19:23:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj21 2024-03-04T19:23:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.697 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.697 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.697 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.697 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.697 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj21 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:n5DnqcyIED91/hUcYoKaQ7R4FhI= 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:n5DnqcyIED91/hUcYoKaQ7R4FhI='} 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c113331aa2d29f5-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001259bbe3b9165061-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj22 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj22 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj22 2024-03-04T19:23:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj22 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:XuIyssg8rA9AIpzHpZJnhT49yZ0= 2024-03-04T19:23:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XuIyssg8rA9AIpzHpZJnhT49yZ0='} 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4779adf9ed68d87-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.701 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054fb263a952a5a2c-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj23 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj23 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj23 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj23 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:3dv/bwaqyaYPFA6CHcGnieqGGkY= 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3dv/bwaqyaYPFA6CHcGnieqGGkY='} 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e0d9dd7dfb54f78c-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084bb20639fac9af3-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj24 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj24 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj24 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.705 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj24 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:dSUB0gVALfwS80dilGhmAHfflxU= 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dSUB0gVALfwS80dilGhmAHfflxU='} 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060e0e4e616258819-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1db9d018ad68eae-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj25 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj25 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj25 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj25 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.708 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:S+nf2CarJY+NQ2RfQKFV3MhUFQY= 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S+nf2CarJY+NQ2RfQKFV3MhUFQY='} 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000779a95cc2d58b58a-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.709 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebcf806dd8b72dd0-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj26 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj26 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj26 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj26 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:T5BNQVClvkd13Ru+HXdLq+pr/Ec= 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T5BNQVClvkd13Ru+HXdLq+pr/Ec='} 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000668befdd40331e4f-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.713 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e1c07362bcc70cb-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj27 2024-03-04T19:23:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj27 2024-03-04T19:23:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj27 2024-03-04T19:23:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj27 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:nbI4qVw0OR5c2Fieu0rjuZwjwG0= 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nbI4qVw0OR5c2Fieu0rjuZwjwG0='} 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1eb427ed10bfffe-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.715 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074b57f179c81833f-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj28 2024-03-04T19:23:50.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj28 2024-03-04T19:23:50.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj28 2024-03-04T19:23:50.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj28 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:LsqULEMt2kgpY/kacxHgPkW5oAY= 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LsqULEMt2kgpY/kacxHgPkW5oAY='} 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a06f8d62759ab614-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000194c642ee7595e2f-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj29 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj29 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj29 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.719 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj29 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:SJ/+hdfKGqA/5vSBdpNrS+p6nIc= 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SJ/+hdfKGqA/5vSBdpNrS+p6nIc='} 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f8af6c91a768fe1-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000592a51afee0a0eee-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj30 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj30 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj30 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.722 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj30 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:6ICKbvJCLUqrDzKNJINnTucUw84= 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6ICKbvJCLUqrDzKNJINnTucUw84='} 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d975bd9c1c1dd7d-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.724 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.724 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.724 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.724 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.724 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.724 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087d84d8374330965-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj31 2024-03-04T19:23:50.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj31 2024-03-04T19:23:50.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj31 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj31 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.725 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:jCbltS9+5qIt8fBTkyxEpefZvoE= 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jCbltS9+5qIt8fBTkyxEpefZvoE='} 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000232d8754f76a44f3-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.726 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040d2677de7c86bdc-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj32 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj32 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj32 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj32 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:ydlN8stGkpHawB3OthPLcTa9H9c= 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ydlN8stGkpHawB3OthPLcTa9H9c='} 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bbaf381cea5fdd16-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b514afcfee18efd-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj33 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj33 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj33 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.730 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj33 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:uUj/qdprHlPRAHk1VvP6CTGJ0KU= 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uUj/qdprHlPRAHk1VvP6CTGJ0KU='} 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062fbfd7c0f8f679e-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000966f14e7d3d128e9-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj34 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj34 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj34 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.733 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj34 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:RwCFGgm/6AY7XdwZTAFs2VILVgk= 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RwCFGgm/6AY7XdwZTAFs2VILVgk='} 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000aa7752c455daf4b-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d90522445c28727-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj35 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj35 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj35 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj35 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:S3qW0nA2K267pB4oBoS5lYCCras= 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S3qW0nA2K267pB4oBoS5lYCCras='} 2024-03-04T19:23:50.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c86aded11eecb97a-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.737 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090c9e17ba8bfc041-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj36 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj36 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj36 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj36 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:X73DsXkWcwu5tClBkoUIt5Yj2Cg= 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:X73DsXkWcwu5tClBkoUIt5Yj2Cg='} 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088765e8004952884-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000520fc915cc77bd21-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj37 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj37 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj37 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.741 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj37 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:zYkL3s09o5vcbkC8LWq+ZYFoZSE= 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zYkL3s09o5vcbkC8LWq+ZYFoZSE='} 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2a5007c12f45c46-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083e3ba9e28551e7f-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj38 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj38 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj38 2024-03-04T19:23:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj38 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:/s4PQsf28AiVpQGO/ykeMyZwAog= 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/s4PQsf28AiVpQGO/ykeMyZwAog='} 2024-03-04T19:23:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e421833a17da58e7-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.745 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d72aadd0894b5cf6-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj39 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj39 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj39 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj39 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:1pBFBV4pS/Z8AZ7vNMCjRBZj0iE= 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1pBFBV4pS/Z8AZ7vNMCjRBZj0iE='} 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b6af08c2602d27b-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3f503c205c2b591-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj40 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj40 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj40 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.749 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj40 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:L2rJVIEhBux35JbaASqva0GYu4Y= 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:L2rJVIEhBux35JbaASqva0GYu4Y='} 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f231862d5a10d63-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007fca701807e2bbe8-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj41 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj41 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj41 2024-03-04T19:23:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.752 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.752 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.752 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.752 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.752 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj41 2024-03-04T19:23:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:2NXoht6SMhjoNFvUXe7JYNk7G8M= 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2NXoht6SMhjoNFvUXe7JYNk7G8M='} 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f9b8800798e562f-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e163a0a022020793-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj42 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj42 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj42 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj42 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:gETzSWJuh2WQfQZTsscxxDrS8EQ= 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gETzSWJuh2WQfQZTsscxxDrS8EQ='} 2024-03-04T19:23:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2a77943b7794c0a-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.757 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.757 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.757 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.757 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.757 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.757 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e45f66b2d53b548-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj43 2024-03-04T19:23:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj43 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj43 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.758 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj43 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:vrs6qH3/EU7uxZwLffvkVOOjMGA= 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vrs6qH3/EU7uxZwLffvkVOOjMGA='} 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d951cac9ac89a11-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.760 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.760 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.760 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.760 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.760 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.760 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006cd35ac99c8f129d-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj44 2024-03-04T19:23:50.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj44 2024-03-04T19:23:50.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj44 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj44 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.761 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:dBOoV0sUyWiD5G+VhQmoAf7lMVA= 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dBOoV0sUyWiD5G+VhQmoAf7lMVA='} 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007edfd1a234fa8adb-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.762 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f59967247e4732e-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj45 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj45 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj45 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj45 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oJElWmmgEtgX9+HLhxzvySK0qOs= 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oJElWmmgEtgX9+HLhxzvySK0qOs='} 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055ef1ee3b3eef8a6-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.765 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.765 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.765 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.765 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d26daf4af46b7503-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj46 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj46 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj46 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.767 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.767 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.767 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.767 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.767 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj46 2024-03-04T19:23:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.767 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:cHKDrexfcbr5BT68bIAa8zsb2y8= 2024-03-04T19:23:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cHKDrexfcbr5BT68bIAa8zsb2y8='} 2024-03-04T19:23:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2c0bf932dc24d9f-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:44 GMT 2024-03-04T19:23:50.768 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o= 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:44 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TZWnxuFj9YBUuZ42syMTn+dc+0o=', 'Content-Length': '0'} 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae6158d3b89aa7b1-0065e6079c-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj47 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj47 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj47 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.770 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.770 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.770 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.770 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.770 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj47 2024-03-04T19:23:50.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.770 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:jKVhfyGcO3pDQrOzTqNiYiCL8tw= 2024-03-04T19:23:50.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jKVhfyGcO3pDQrOzTqNiYiCL8tw='} 2024-03-04T19:23:50.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007313120985bfd186-0065e6079d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.771 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:4saRfKQPUTEfX7GjBLAv4DeSJIw= 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4saRfKQPUTEfX7GjBLAv4DeSJIw=', 'Content-Length': '0'} 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078466d3a4e424606-0065e6079d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj48 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj48 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj48 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj48 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:1saF5tVTuc8a04rMDlDNhWyH8NY= 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:1saF5tVTuc8a04rMDlDNhWyH8NY='} 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f8ea052fb462931-0065e6079d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/ 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/ 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/ 2024-03-04T19:23:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.774 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.774 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.774 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.774 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.774 INFO:tasks.rgw_multisite_tests:/ppfahn-16/ 2024-03-04T19:23:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:4saRfKQPUTEfX7GjBLAv4DeSJIw= 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:4saRfKQPUTEfX7GjBLAv4DeSJIw=', 'Content-Length': '0'} 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000499e66335616f5d9-0065e6079d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-16/obj49 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-16/obj49 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-16/obj49 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:/ppfahn-16/obj49 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:EZtTBRz59vkkN9NY01AKz6AiPPg= 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:EZtTBRz59vkkN9NY01AKz6AiPPg='} 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3b3ecc4042d822c-0065e6079d-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5b56bfefcb80fff-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj1 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj1 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj1 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.778 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj1 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yDaLHN9NVF86OeyyypZaMMnSCUM= 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yDaLHN9NVF86OeyyypZaMMnSCUM='} 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c691aadcf08dd70d-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d5e0a1dbc5cb6db-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj2 2024-03-04T19:23:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj2 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj2 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.781 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj2 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:LSmslRrwbnLBGx60LRqeI5/Yzcg= 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:LSmslRrwbnLBGx60LRqeI5/Yzcg='} 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000641b903d8ac8d934-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e34a4de6b95b3367-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj3 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj3 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj3 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj3 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:SkZwVHAG6CGafHFnUcTfIbX4eBA= 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:SkZwVHAG6CGafHFnUcTfIbX4eBA='} 2024-03-04T19:23:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fee932b4845f1eee-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.785 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.786 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.786 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8eaa3b4459c1eaa-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj4 2024-03-04T19:23:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj4 2024-03-04T19:23:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj4 2024-03-04T19:23:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj4 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:aGFsD0am4GBu2d6pq4KjI+7RJNo= 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:aGFsD0am4GBu2d6pq4KjI+7RJNo='} 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000369ebfb905ffd1cc-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.788 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003da358e067b354ea-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj5 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj5 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj5 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.789 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj5 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:mHTg3x99jvaD5pqr5e+YR9bNILM= 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mHTg3x99jvaD5pqr5e+YR9bNILM='} 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6e7a8824a34fa8f-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000370ca0935fd2a0e6-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj6 2024-03-04T19:23:50.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj6 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj6 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj6 2024-03-04T19:23:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:RRD2rZGBdTtch233M4PbIlXn81s= 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RRD2rZGBdTtch233M4PbIlXn81s='} 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f38b8dd4d009344-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.793 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002207dd53dc885ad6-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj7 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj7 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj7 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj7 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TDNpslkjQNWD+EsXIjZjB55Iriw= 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TDNpslkjQNWD+EsXIjZjB55Iriw='} 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000714b5469cbfa6118-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f0136c27560b35b-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj8 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj8 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj8 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.797 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj8 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:32cUfduQM8/N3teYTZHcL676F/0= 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:32cUfduQM8/N3teYTZHcL676F/0='} 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cacf3606288e1040-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d3d0921af527010-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj9 2024-03-04T19:23:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj9 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj9 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.800 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj9 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:jhXwfSVKNRKCI0XCNFRLLkS68KA= 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jhXwfSVKNRKCI0XCNFRLLkS68KA='} 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000553843d86cfcc417-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc6462eb93a6cec6-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj10 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj10 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj10 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj10 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:2KhvwDFz4QOQc6Dc5jGi2tU3spo= 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2KhvwDFz4QOQc6Dc5jGi2tU3spo='} 2024-03-04T19:23:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc471d8dd0b1035c-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.804 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.804 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000824e170ba060605-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj11 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj11 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj11 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj11 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:nQDh7BLfBmYWB0iA7/y4huVGg3Y= 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nQDh7BLfBmYWB0iA7/y4huVGg3Y='} 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068313b9e4fbec8c9-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.808 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007eee396b18406604-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj12 2024-03-04T19:23:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj12 2024-03-04T19:23:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj12 2024-03-04T19:23:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj12 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:bpMBzJF2aShgS8tXNyR+k2FZrIk= 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bpMBzJF2aShgS8tXNyR+k2FZrIk='} 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9f58edc49189b1d-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cab407ce380c0bf4-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj13 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj13 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj13 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.811 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj13 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:5l38Nx3G/u/jTy85Q2ArhlFuJPA= 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5l38Nx3G/u/jTy85Q2ArhlFuJPA='} 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098c3a0d5c3dc6819-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db953d5fd78f4f79-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj14 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj14 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj14 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj14 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.814 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:+VhDsbFUjKxXfBsx3Mo3WInHMwY= 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+VhDsbFUjKxXfBsx3Mo3WInHMwY='} 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3259b56875eef73-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.815 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000395507de159b9e46-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj15 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj15 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj15 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj15 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:Cc6IsTmRLJd61TMTe2zOscRKrys= 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Cc6IsTmRLJd61TMTe2zOscRKrys='} 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000475518baaa41e52c-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.818 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f33a5baa2bca48c3-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj16 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj16 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj16 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.819 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.820 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.820 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.820 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.820 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj16 2024-03-04T19:23:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.820 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:MG3QM8vhvxmO+VMnpaEa/p/JNlg= 2024-03-04T19:23:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MG3QM8vhvxmO+VMnpaEa/p/JNlg='} 2024-03-04T19:23:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009440d29c86855a59-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004dcb5b0a8c6006ed-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj17 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj17 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj17 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.822 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj17 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:adKFllKXRZ5qCOL0eaFzg60hMgA= 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:adKFllKXRZ5qCOL0eaFzg60hMgA='} 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000032ab22713818c094-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016a40ca7fde5f96f-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj18 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj18 2024-03-04T19:23:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj18 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj18 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.825 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:ftJN7bclqZ2QLhisphT8nvEIBX0= 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ftJN7bclqZ2QLhisphT8nvEIBX0='} 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000350567b117c847-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.826 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f215bd851a620ba-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj19 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj19 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj19 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj19 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:rWCbZh6lK6Xg5Tlj/MgcPJ3YuHA= 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rWCbZh6lK6Xg5Tlj/MgcPJ3YuHA='} 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f82c6d3078e22cc-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.829 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061976106ac6a0bb9-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj20 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj20 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj20 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj20 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:wY89fp5GBX6ktR7Hp3IeFyW1Im8= 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wY89fp5GBX6ktR7Hp3IeFyW1Im8='} 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c0974a2316b9f5a-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbb8cc7f54142fb1-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj21 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj21 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj21 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.833 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj21 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:l6wv1l406h1V1Vkrip9nHDacyiQ= 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:l6wv1l406h1V1Vkrip9nHDacyiQ='} 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031ef01db3d0a9277-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005cd37d8e8e63de26-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj22 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj22 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj22 2024-03-04T19:23:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj22 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:xFuhqtKRRXtIMEKU2B562hoIjdY= 2024-03-04T19:23:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xFuhqtKRRXtIMEKU2B562hoIjdY='} 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009aab2421c8f84205-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.837 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1cda1ba5f36971a-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj23 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj23 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj23 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj23 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:qJr4lChYwcPmlvJ0RCCetCsSy5E= 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qJr4lChYwcPmlvJ0RCCetCsSy5E='} 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c473838de5b8c965-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001fa9793e600869b1-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj24 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj24 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj24 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.841 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj24 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:ofKfuGZmfzQKU9h2ZHsMueKnESU= 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ofKfuGZmfzQKU9h2ZHsMueKnESU='} 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b89513b5a44a43f-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000981b241e5b5673e1-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj25 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj25 2024-03-04T19:23:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj25 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj25 2024-03-04T19:23:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:3Cc04FwAcL+eFnDKqvkSZgauL50= 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3Cc04FwAcL+eFnDKqvkSZgauL50='} 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047bb45c010d4b9ac-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.845 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c320ec9cc09bd360-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj26 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj26 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj26 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj26 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:/56QYhPAnmvKfxjkZpikxqsiqIA= 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/56QYhPAnmvKfxjkZpikxqsiqIA='} 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ee56401707c6055-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4= 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yv8N+8ByddwY7O6Nu+2mm76cDi4=', 'Content-Length': '0'} 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eec9c8bc4810430f-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj27 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj27 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj27 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.849 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:45 GMT 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj27 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:lb4lbRs/vZokaRwNtxuSkIJFBEg= 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lb4lbRs/vZokaRwNtxuSkIJFBEg='} 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068aa4ee1e79a26ee-0065e6079d-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000819d21905f9ab41d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj28 2024-03-04T19:23:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj28 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj28 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj28 2024-03-04T19:23:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:P/bSpO5QB+IyU1FbDSjF598ZqrA= 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:P/bSpO5QB+IyU1FbDSjF598ZqrA='} 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cc292a4014a0735-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.853 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e91b402fd3bc9d78-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj29 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj29 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj29 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj29 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:9QDMa4H4+sIu37Mt3gjqwPOBy2c= 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9QDMa4H4+sIu37Mt3gjqwPOBy2c='} 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cbea81a5f22864b-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000506b5060a927f67a-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj30 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj30 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj30 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj30 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:vmRz5ivJZSmL4j7+xdssSAkVzyk= 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vmRz5ivJZSmL4j7+xdssSAkVzyk='} 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003652d86d18b31186-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000805106a68dd8829-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj31 2024-03-04T19:23:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj31 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj31 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.860 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj31 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:MikwRb715H1rZixXy5CX+wVxbvk= 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MikwRb715H1rZixXy5CX+wVxbvk='} 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e792898aece2343-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4336492e356181d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj32 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj32 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj32 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj32 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:qVUMhBkz8fKD1R1+DPok6bCvn/Q= 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qVUMhBkz8fKD1R1+DPok6bCvn/Q='} 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa71728a726bce8f-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f78c0791365d749d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj33 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj33 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj33 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.865 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj33 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:BebK0VpUAesGjabl17H5vnZTlRw= 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BebK0VpUAesGjabl17H5vnZTlRw='} 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b9585fb74c895cd9-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e9b883d0fad4bbc-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj34 2024-03-04T19:23:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj34 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj34 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj34 2024-03-04T19:23:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:D8h4QSWhQVhGEbCqxD64QS8ZVck= 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:D8h4QSWhQVhGEbCqxD64QS8ZVck='} 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000607601cec50adc60-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.870 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.870 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.870 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.870 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.870 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.870 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d86ce4b0488629d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj35 2024-03-04T19:23:50.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj35 2024-03-04T19:23:50.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj35 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj35 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.871 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:qXx7mdgy9thi8HKDW3IVOXQnkWg= 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qXx7mdgy9thi8HKDW3IVOXQnkWg='} 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9898341ffee892c-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.872 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.873 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.873 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.873 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.873 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de5a573780c2111b-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj36 2024-03-04T19:23:50.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj36 2024-03-04T19:23:50.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj36 2024-03-04T19:23:50.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj36 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oMujIWuffW5DXzRciETW+jxKn+0= 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oMujIWuffW5DXzRciETW+jxKn+0='} 2024-03-04T19:23:50.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057fdd23bd1f4da32-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.875 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d456efd45dde7b38-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj37 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj37 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj37 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj37 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:CCapDQ7R5YuC3D22A7Zt3EoAkkE= 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CCapDQ7R5YuC3D22A7Zt3EoAkkE='} 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5afdf9884447322-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000decc226cdca32f87-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj38 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj38 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj38 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.879 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj38 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:2PPCERf5TiOLFCQqKsY7swLppZM= 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2PPCERf5TiOLFCQqKsY7swLppZM='} 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005639fdb399978f6d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fdbd2235214a3be-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj39 2024-03-04T19:23:50.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj39 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj39 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj39 2024-03-04T19:23:50.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:p+pu7W3miBh2BIsEwrdf8qNIB0s= 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:p+pu7W3miBh2BIsEwrdf8qNIB0s='} 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c1e4e0122adaf6d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.883 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b06dadc81fa455f8-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj40 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj40 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj40 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj40 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:T+Orqd0GYxd4JomYHABUyozPpyo= 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T+Orqd0GYxd4JomYHABUyozPpyo='} 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e47a17a30566fa9c-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.886 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005232ebdd516be170-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj41 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj41 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj41 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj41 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8rMN8WbSJkBIMmifYmFa4omD8ig= 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8rMN8WbSJkBIMmifYmFa4omD8ig='} 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000aa8663fc100596f-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000283f6621939756c3-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj42 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj42 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj42 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.890 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj42 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:+st7WxInpo7TXAGR4vRWhxceaUE= 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+st7WxInpo7TXAGR4vRWhxceaUE='} 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3b402e08f847be9-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab239f6932e18c15-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj43 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj43 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj43 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj43 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:71WCl/b9KA7XM6dL+0TrE59i3yQ= 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:71WCl/b9KA7XM6dL+0TrE59i3yQ='} 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081292f3b69152814-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.894 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033d6ea807620431d-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj44 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj44 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj44 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.896 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.896 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.896 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.896 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.896 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj44 2024-03-04T19:23:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.896 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:CogpU7Py9ZE9IS0BR/aN3TNJfWk= 2024-03-04T19:23:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CogpU7Py9ZE9IS0BR/aN3TNJfWk='} 2024-03-04T19:23:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000369350ae28df5938-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055c13632a896e0a7-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj45 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj45 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj45 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj45 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:N+ezKFWyC1G/Jz19uyUgMgX0ikg= 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:N+ezKFWyC1G/Jz19uyUgMgX0ikg='} 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a38953808b6f9994-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.900 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.900 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.900 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.900 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.900 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.900 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3a00d1452f91175-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj46 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj46 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj46 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.901 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.902 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.902 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.902 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj46 2024-03-04T19:23:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.902 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:qRXLPLxbFyJphpxDBrrGeb+7Yfg= 2024-03-04T19:23:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qRXLPLxbFyJphpxDBrrGeb+7Yfg='} 2024-03-04T19:23:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c96630a68272a2b3-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000973ad9895456a4e9-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj47 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj47 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj47 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.904 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj47 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:szAHp1unAp3AdPyxu5zGqxfkDoE= 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:szAHp1unAp3AdPyxu5zGqxfkDoE='} 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031b0810b2f0ea471-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b62aa6767b99cc0-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj48 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj48 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj48 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj48 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:m1zKkweucbcEs/ygMm0rpLb00ws= 2024-03-04T19:23:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m1zKkweucbcEs/ygMm0rpLb00ws='} 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9b05bd30bfae35c-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/ 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/ 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/ 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.908 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:/ppfahn-17/ 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8= 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oRv8BEqF7na/mnGjkIKj3LaAev8=', 'Content-Length': '0'} 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f57c772f26fb4e86-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-17/obj49 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-17/obj49 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-17/obj49 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:/ppfahn-17/obj49 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:ALkrktUupXASaUtKYNpG0F7p4QE= 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ALkrktUupXASaUtKYNpG0F7p4QE='} 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000921874273e842b51-0065e6079e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.911 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a72310fb41dcbd0-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj1 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj1 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj1 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.912 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj1 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:tXXvBfu8xCaHPRuA0kLcEoID3n0= 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tXXvBfu8xCaHPRuA0kLcEoID3n0='} 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce81c5b799b371da-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e75a0cac7487aece-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj2 2024-03-04T19:23:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj2 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj2 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.915 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj2 2024-03-04T19:23:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.916 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:eHZF7PjUcl5Cljs6VenFAAlyMeM= 2024-03-04T19:23:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eHZF7PjUcl5Cljs6VenFAAlyMeM='} 2024-03-04T19:23:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2e4f46c9f4595ac-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d56b51579bc111c-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj3 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj3 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj3 2024-03-04T19:23:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.918 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.918 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.918 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.918 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.918 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj3 2024-03-04T19:23:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.919 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:QbpJ4/kruIpmTbxdVcxxIsvrIhQ= 2024-03-04T19:23:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:QbpJ4/kruIpmTbxdVcxxIsvrIhQ='} 2024-03-04T19:23:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f7da9cba83843fc-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9833eea2925af4c-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj4 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj4 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj4 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj4 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.921 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:NtXF9AOi9P86UBPvnSdkuKdn3Dg= 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NtXF9AOi9P86UBPvnSdkuKdn3Dg='} 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f16955d2af61501b-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.922 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020379bdec4d68e28-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj5 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj5 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj5 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.924 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.924 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.924 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.924 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.924 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj5 2024-03-04T19:23:50.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.924 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:n3sqCDrLv7P5eNNNzc1iFCSWQqw= 2024-03-04T19:23:50.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:n3sqCDrLv7P5eNNNzc1iFCSWQqw='} 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d3cebec60580091-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.925 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4fffd1f508a170c-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj6 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj6 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj6 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj6 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:XRNWc0Y33u90Fc/qyAxJfdgaOi0= 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XRNWc0Y33u90Fc/qyAxJfdgaOi0='} 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073db94e65aacbd36-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ceddd92152404cdd-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj7 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj7 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj7 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.929 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj7 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:nSvK9PMe02/tc0ktDkhmoIWMWrw= 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nSvK9PMe02/tc0ktDkhmoIWMWrw='} 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097ab9f90aa45ba08-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0= 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xKEeR2JpUtPhLtw4cz7E+Rja7p0=', 'Content-Length': '0'} 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000684b644cba205a2-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj8 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj8 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj8 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:46 GMT 2024-03-04T19:23:50.932 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj8 2024-03-04T19:23:50.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.933 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:7VnksVOus/Ex5YKJLKSjmIq80ko= 2024-03-04T19:23:50.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7VnksVOus/Ex5YKJLKSjmIq80ko='} 2024-03-04T19:23:50.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f941e02042bf3cb4-0065e6079e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad92d99651cda83c-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj9 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj9 2024-03-04T19:23:50.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj9 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj9 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.935 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:RySp7YRzKOe5z3gQbVfY9Ie9In8= 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RySp7YRzKOe5z3gQbVfY9Ie9In8='} 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9c4d784ed332dc7-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.936 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc8c8ffa8e523036-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj10 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj10 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj10 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj10 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:3Z6X5J6KVpezda4r60yLTOaO67U= 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3Z6X5J6KVpezda4r60yLTOaO67U='} 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000318e43f21fe53014-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.939 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dda5c040e34e49a1-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj11 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj11 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj11 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.940 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj11 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:S7uSTe+XfvOM2l0OCVsky9s3ISg= 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S7uSTe+XfvOM2l0OCVsky9s3ISg='} 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f92c9661d7dbb1b-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ada3ccd10332982-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj12 2024-03-04T19:23:50.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj12 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj12 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj12 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.943 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:BD+MTykFXSqp8LtTezj/wJL1yEU= 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BD+MTykFXSqp8LtTezj/wJL1yEU='} 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ce397ddf082381b-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.944 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfb0f63c972b8883-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj13 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj13 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj13 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj13 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:pLzw0cNJPkOs2aNWt+PyyOiGJu8= 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pLzw0cNJPkOs2aNWt+PyyOiGJu8='} 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015fdfe4f548e89b0-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.947 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080a82f0c40b30cfa-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj14 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj14 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj14 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.948 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj14 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:CPPeB/1RzwpgUrLK3X5jOwgqhm0= 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CPPeB/1RzwpgUrLK3X5jOwgqhm0='} 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d65629b60c47340-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c992ccbbe336c198-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj15 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj15 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj15 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.951 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj15 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:ZKtnkI4Vy/5hNP2L1WlIU1FXIxw= 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ZKtnkI4Vy/5hNP2L1WlIU1FXIxw='} 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074f23abec2167869-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc9e16202b0879ae-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj16 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj16 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj16 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj16 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:ekUhpFOqUF/04ijQyWjO9pnjgTU= 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ekUhpFOqUF/04ijQyWjO9pnjgTU='} 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdcc450764619a8a-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.955 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b26da1df2d88abab-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj17 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj17 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj17 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj17 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:S3ktK/Ofahw7lDp3YwaNMAiCL/8= 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:S3ktK/Ofahw7lDp3YwaNMAiCL/8='} 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081c49a539dcfa749-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d549a776d7ffb8aa-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj18 2024-03-04T19:23:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj18 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj18 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj18 2024-03-04T19:23:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:Y/QcqX8UBgGeVF/Ya9mZeoUixS0= 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Y/QcqX8UBgGeVF/Ya9mZeoUixS0='} 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a3486b5541ee207-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3aa9a361c3ceaad-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj19 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj19 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj19 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj19 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:tQxlhblPDnvYrBOTkX4WjLWgry8= 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tQxlhblPDnvYrBOTkX4WjLWgry8='} 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd5457ce8353704b-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d70a7ee907fd576a-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj20 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj20 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj20 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.964 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj20 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:NOE+dMQOq34WhzZjBlxQWLl8ESo= 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NOE+dMQOq34WhzZjBlxQWLl8ESo='} 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009673c75ebc7fa454-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c975c0b5fdaa35be-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj21 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj21 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj21 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.967 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj21 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TyjniXjn4XEx/BvPzKRYnClebPc= 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TyjniXjn4XEx/BvPzKRYnClebPc='} 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1005eaaba8f4f1e-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.969 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.969 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.969 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.969 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.969 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.969 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad9e7d6bca01447f-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj22 2024-03-04T19:23:50.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj22 2024-03-04T19:23:50.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj22 2024-03-04T19:23:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.970 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.970 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.970 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.970 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.970 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj22 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:XC3OehRPM+TEUkejR9803cm9InY= 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XC3OehRPM+TEUkejR9803cm9InY='} 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cf0a7c7236d0c82-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.972 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.972 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.972 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.972 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.972 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.972 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa4b5779478cefd7-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj23 2024-03-04T19:23:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj23 2024-03-04T19:23:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj23 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj23 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.973 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:m2BXxST4rhX4nEqZnBDlXXIfIFI= 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:m2BXxST4rhX4nEqZnBDlXXIfIFI='} 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000acf8fbf8e8ebff97-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.974 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a62081c2a7dfefc-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj24 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj24 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj24 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.976 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.976 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.976 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.976 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.976 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj24 2024-03-04T19:23:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.976 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:uLena+Q5ba/vzt8OqrTfmINjhKM= 2024-03-04T19:23:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uLena+Q5ba/vzt8OqrTfmINjhKM='} 2024-03-04T19:23:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c55cbebc826cf2eb-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.977 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba451b87b5135991-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj25 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj25 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj25 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj25 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:VPTjRg/bD/90Uo0BC4CzhPe3oS8= 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VPTjRg/bD/90Uo0BC4CzhPe3oS8='} 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000529dbdab99a8f169-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f7e947e00941a6f-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj26 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj26 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj26 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.981 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj26 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:JrMZQx8skL0IQs9+So6PGQxxKWY= 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JrMZQx8skL0IQs9+So6PGQxxKWY='} 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da8294b37bda4702-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009018a56643c8b64-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj27 2024-03-04T19:23:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj27 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj27 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.984 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj27 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:s9raxzbn77r1nM4gIjYCK1qqPYg= 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:s9raxzbn77r1nM4gIjYCK1qqPYg='} 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011aaa603ee485ced-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b57d00166d30515a-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj28 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj28 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj28 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj28 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:clzqdq4W+vu3fVfr/Pq134g0zfs= 2024-03-04T19:23:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:clzqdq4W+vu3fVfr/Pq134g0zfs='} 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000238ba022b950e81-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.988 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005de9df8d53016e54-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj29 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj29 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj29 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj29 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:E8vH3ChebXa/ZaeXS/OWVZFft7Y= 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:E8vH3ChebXa/ZaeXS/OWVZFft7Y='} 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbcccefd5722ffb9-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.991 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.992 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f3bc29a1bddbf09-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj30 2024-03-04T19:23:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj30 2024-03-04T19:23:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj30 2024-03-04T19:23:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj30 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:kXDFKkOvmAKbnnGegFAC2LKvhho= 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:kXDFKkOvmAKbnnGegFAC2LKvhho='} 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d434083ef746095d-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009df4e8957bd22947-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj31 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj31 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj31 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.995 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj31 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:fWpPIic20yYavqVs65Hn+tjaYpc= 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:fWpPIic20yYavqVs65Hn+tjaYpc='} 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000542169fceae67706-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ad5d6fe0ed6795f-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj32 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj32 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj32 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:50.998 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj32 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:UcPERRwoI9WY8cWh2FsnMzfeWgQ= 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:UcPERRwoI9WY8cWh2FsnMzfeWgQ='} 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003bb38c4ca2b2b5cb-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:50.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b049c66ab5f4ebb-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj33 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj33 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj33 2024-03-04T19:23:51.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj33 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:nhRRuPaUFFZqCtM5qkVSaThUvxM= 2024-03-04T19:23:51.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nhRRuPaUFFZqCtM5qkVSaThUvxM='} 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e73dc262f25698f2-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.002 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bc75abe0b11e040-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj34 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj34 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj34 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj34 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:VnZcd1IcyAH9U9jmoqr9SmCt6c4= 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VnZcd1IcyAH9U9jmoqr9SmCt6c4='} 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5435487d9bace6e-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:51.005 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.006 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029fcf73a15a2d321-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj35 2024-03-04T19:23:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj35 2024-03-04T19:23:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj35 2024-03-04T19:23:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj35 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:nsGNAIf9u1B3Cxukzrev7ST9pvU= 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:nsGNAIf9u1B3Cxukzrev7ST9pvU='} 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b5c25bceff2ecef-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047228b693d5456e9-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj36 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj36 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj36 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.009 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj36 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:2SE+4ckzwMwvEuzXAhozEVp+0LM= 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2SE+4ckzwMwvEuzXAhozEVp+0LM='} 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9e45acdb67cc77a-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010f450c1ebf2f11e-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj37 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj37 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj37 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.012 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj37 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:RarFwb+ENEl0Yc0a3AZ3+0+WnxE= 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:RarFwb+ENEl0Yc0a3AZ3+0+WnxE='} 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d295ef0998ac5dbd-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4= 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:US7vfQsv9bkmjwIPOMRRN1ApAV4=', 'Content-Length': '0'} 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d4c4e9aa4498083-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj38 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj38 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj38 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:47 GMT 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj38 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:/8IViy8HT8/bKsilVsPdPGBXhS4= 2024-03-04T19:23:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:47 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/8IViy8HT8/bKsilVsPdPGBXhS4='} 2024-03-04T19:23:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a64cd804e4c37691-0065e6079f-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.016 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d4ae951ad0c7557-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj39 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj39 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj39 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj39 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:C+xPRV2IGT7rT/HKMA5Bqd/cdkQ= 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:C+xPRV2IGT7rT/HKMA5Bqd/cdkQ='} 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cd4aeb34ae1d09d-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.019 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.020 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T19:23:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T19:23:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006baf7f28eece738d-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj40 2024-03-04T19:23:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj40 2024-03-04T19:23:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj40 2024-03-04T19:23:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj40 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:MXMIZcmN+H+QH8rIlqx7GIM2Afs= 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MXMIZcmN+H+QH8rIlqx7GIM2Afs='} 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a131d22b16f5f74-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc5424a2ff14ce7b-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj41 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj41 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj41 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.023 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj41 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:+t7tIFV5NGI1Hhe6+ECqyM7JZoU= 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+t7tIFV5NGI1Hhe6+ECqyM7JZoU='} 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c75eb7e11982ce9-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d2b8834f54a46f4-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj42 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj42 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj42 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.026 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj42 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:eFAn9RJrC2oNSl3u3Qhs0fExcls= 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eFAn9RJrC2oNSl3u3Qhs0fExcls='} 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6d9c88d4c9830be-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0b6259f29eaf597-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj43 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj43 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj43 2024-03-04T19:23:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj43 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:yxMj2iPqWdgj/I1P/2l8yw+5T9c= 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:yxMj2iPqWdgj/I1P/2l8yw+5T9c='} 2024-03-04T19:23:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c72e38993a560405-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.030 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011f0092d0415020e-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj44 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj44 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj44 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj44 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:N0C7V9T6qYdkftFJrxj7I7eymv4= 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:N0C7V9T6qYdkftFJrxj7I7eymv4='} 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046bf58e5ddce7200-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T19:23:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002120198180554935-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj45 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj45 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj45 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.034 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj45 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:xIYkjH30yELEE+6r30pyFu1Omw8= 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xIYkjH30yELEE+6r30pyFu1Omw8='} 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000159c3e8b332bcea4-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdabc9fbbc072696-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj46 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj46 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj46 2024-03-04T19:23:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj46 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:HP7rhD1GDQu7pT7yNIu497FM5EU= 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HP7rhD1GDQu7pT7yNIu497FM5EU='} 2024-03-04T19:23:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084ab3141f3b356b4-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.038 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093a3b1b57143a1f2-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj47 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj47 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj47 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj47 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:F321Zeirb4Qh1sZSv09Gl4V270o= 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F321Zeirb4Qh1sZSv09Gl4V270o='} 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000defd011a6cb65660-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T19:23:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a09371a9dddcfb2-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj48 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj48 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj48 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.042 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj48 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:rdJVEKBdVKD5QJl+Xc3BUtAobIA= 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:rdJVEKBdVKD5QJl+Xc3BUtAobIA='} 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eabc819ad0d34912-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/ 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/ 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/ 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests:/ppfahn-18/ 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY= 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8yIHipN9mEiUfIE9jYlzakwcUBY=', 'Content-Length': '0'} 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdb376f94bc5741b-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-18/obj49 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-18/obj49 2024-03-04T19:23:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-18/obj49 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:40:48 GMT 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:/ppfahn-18/obj49 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.045 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:/9VsweVnoUAw6iEnur47e9kogfU= 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:40:48 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:/9VsweVnoUAw6iEnur47e9kogfU='} 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b51d14cfbc113b17-0065e607a0-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:40:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709574043.443936_99.1","last_update":"2024-03-04T17:40:43.443936Z"}] 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574043.443936_99.1'} 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:40:48.598+0000 7f55d50db980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:40:48.602+0000 7f55d50db980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574043.443936_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:40:43.443936Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:48Z"} 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574043.443936_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:40:43.443936Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:48Z"} 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574043.443936_99.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:40:43.443936Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574043.443936_99.1'} 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574043.443936_99.1'} 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: 2024-03-04T17:40:48.723+0000 7fbbba3e4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: 2024-03-04T17:40:48.723+0000 7fbbba3e4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574043.443936_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:40:43.443936Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:48Z"} 2024-03-04T19:23:51.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709574043.443936_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:40:43.443936Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:48Z"} 2024-03-04T19:23:51.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709574043.443936_99.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:40:43.443936Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574043.443936_99.1'} 2024-03-04T19:23:51.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709574043.443936_99.1'} 2024-03-04T19:23:51.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-03-04T19:23:51.048 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T19:23:51.048 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ppfahn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T19:23:51.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ppfahn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T19:23:51.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:40:48.862+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:40:53.862+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:40:58.862+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:41:03.862+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:41:08.866+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:41:13.866+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:41:18.866+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:41:23.866+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:41:28.870+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:41:33.870+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:41:38.869+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:41:43.873+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:41:48.873+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:41:53.873+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:41:58.877+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:42:03.877+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:42:08.877+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:42:13.877+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:42:18.881+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:42:23.881+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi153.stdout: INFO: 2024-03-04T17:42:24.393+0000 7fa53962a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:42:28.881+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:42:33.885+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:42:38.885+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:42:43.885+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:42:48.885+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:42:53.889+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:42:58.888+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:43:03.892+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:43:08.892+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:43:13.892+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:43:18.892+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:43:23.892+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:43:28.896+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:43:33.896+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:43:38.900+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:43:43.900+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:43:48.900+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:43:53.900+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:43:58.904+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:44:03.904+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:44:08.904+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:44:13.908+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:44:18.908+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:44:23.911+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:44:28.907+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:44:33.911+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:44:38.911+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:44:43.911+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:44:48.915+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:44:53.919+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:44:58.915+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:45:03.919+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:45:08.923+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:45:13.919+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:45:18.923+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:45:23.927+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:45:28.927+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:45:33.931+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:45:38.931+0000 7f5debcbc980 1 waiting to reach incremental sync.. 2024-03-04T19:23:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:45:43.930+0000 7f5debcbc980 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-03-04T19:23:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:45:43.930+0000 7f5debcbc980 0 bucket sync checkpoint failed: (110) Connection timed out 2024-03-04T19:23:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:45:43.930+0000 7f5debcbc980 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-03-04T19:23:51.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-03-04T19:23:51.053 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.053 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.053 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:51.053 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_delete_notempty 2024-03-04T19:23:51.054 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.054 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 903, in test_bucket_delete_notempty 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=ppfahn-39 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-39/ 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-39/ 2024-03-04T19:23:51.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-39/ 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:57:57 GMT 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests:/ppfahn-39/ 2024-03-04T19:23:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:pOLIVhlha78u/MxFGMU3xnGMw6Y= 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:57 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pOLIVhlha78u/MxFGMU3xnGMw6Y=', 'Content-Length': '0'} 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e4f4a20d5da73bbd-0065e60ba5-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=ppfahn-40 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-40/ 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-40/ 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-40/ 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:57:57 GMT 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests:/ppfahn-40/ 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:illUyZQB/5xMr3RxrtGlJJ/Ya6Y= 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:57 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:illUyZQB/5xMr3RxrtGlJJ/Ya6Y=', 'Content-Length': '0'} 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000090cb67cd70ec89f7-0065e60ba5-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=ppfahn-41 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-41/ 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-41/ 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-41/ 2024-03-04T19:23:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:57:57 GMT 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests:/ppfahn-41/ 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:hoP2wBD9hudc1aKEgssHGx7sW1c= 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:57 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hoP2wBD9hudc1aKEgssHGx7sW1c=', 'Content-Length': '0'} 2024-03-04T19:23:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a42e7d8d3fb2275a-0065e60ba5-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709575077.825024_336.1","last_update":"2024-03-04T17:57:57.825024Z"}] 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:57:58.143+0000 7fb798edd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:57:58.143+0000 7fb798edd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:58Z"} 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:58Z"} 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:03.302+0000 7f7bc75dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:03.302+0000 7f7bc75dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:03Z"} 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:03Z"} 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:08.458+0000 7f640dc9e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:08.462+0000 7f640dc9e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:08Z"} 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:08Z"} 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:13.586+0000 7f6de1720980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:13.586+0000 7f6de1720980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:13Z"} 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:13Z"} 2024-03-04T19:23:51.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.861873_275.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.861873Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.861873_275.1'} 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.861873_275.1 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:18.710+0000 7f1d59043980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:18.710+0000 7f1d59043980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:18Z"} 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:18Z"} 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:23.834+0000 7fe89b4c8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:23.834+0000 7fe89b4c8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:23Z"} 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:23Z"} 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:28.974+0000 7fe70021f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:28.974+0000 7fe70021f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:28Z"} 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:28Z"} 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:34.098+0000 7fa1fe158980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:34.098+0000 7fa1fe158980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:34Z"} 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:34Z"} 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:39.230+0000 7f30fde7d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:39.234+0000 7f30fde7d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:39Z"} 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:39Z"} 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:44.349+0000 7ff7cfad2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:44.353+0000 7ff7cfad2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:44Z"} 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:44Z"} 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:49.473+0000 7fb9324a9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:49.473+0000 7fb9324a9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:49Z"} 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:49Z"} 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:54.597+0000 7fd9b4b3e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:54.597+0000 7fd9b4b3e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:54Z"} 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:54Z"} 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:59.725+0000 7ff5027c6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:58:59.725+0000 7ff5027c6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:59Z"} 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:59Z"} 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:04.869+0000 7f4dc672b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:04.869+0000 7f4dc672b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:04Z"} 2024-03-04T19:23:51.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:04Z"} 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:10.001+0000 7f43bc79b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:10.005+0000 7f43bc79b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:10Z"} 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:10Z"} 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.984537_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.984537Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.984537_285.1'} 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.984537_285.1 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:15.129+0000 7f4fd3fbd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:15.129+0000 7f4fd3fbd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:15Z"} 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:15Z"} 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.984537_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.984537Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.984537_285.1'} 2024-03-04T19:23:51.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.984537_285.1 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:20.268+0000 7f220e7cf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:20.268+0000 7f220e7cf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:20Z"} 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:20Z"} 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.984537_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.984537Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.984537_285.1'} 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.984537_285.1 2024-03-04T19:23:51.075 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:25.392+0000 7f8be327a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:25.396+0000 7f8be327a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:25Z"} 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:25Z"} 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:30.520+0000 7f3b2b833980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:30.520+0000 7f3b2b833980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:30Z"} 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:30Z"} 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:35.652+0000 7f03cc4a1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:35.652+0000 7f03cc4a1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:35Z"} 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:35Z"} 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:40.772+0000 7f078f2e5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:40.772+0000 7f078f2e5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:40Z"} 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:40Z"} 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:45.900+0000 7f39433b0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:45.900+0000 7f39433b0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:45Z"} 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:45Z"} 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:51.023+0000 7fa501a0e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:51.023+0000 7fa501a0e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:51Z"} 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:51Z"} 2024-03-04T19:23:51.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.081 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.081 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:56.143+0000 7f495086d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T17:59:56.143+0000 7f495086d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:56Z"} 2024-03-04T19:23:51.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:56Z"} 2024-03-04T19:23:51.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:01.267+0000 7f17dc209980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:01.267+0000 7f17dc209980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:01Z"} 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:01Z"} 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:06.399+0000 7f35b7ed8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:06.399+0000 7f35b7ed8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:06Z"} 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:06Z"} 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:11.523+0000 7faed07d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:11.523+0000 7faed07d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:11Z"} 2024-03-04T19:23:51.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:11Z"} 2024-03-04T19:23:51.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:16.651+0000 7fce752d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:16.651+0000 7fce752d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:16Z"} 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:16Z"} 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:21.771+0000 7f3789c3c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:21.775+0000 7f3789c3c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:21Z"} 2024-03-04T19:23:51.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:21Z"} 2024-03-04T19:23:51.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:26.898+0000 7f41d2cab980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:26.898+0000 7f41d2cab980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:26Z"} 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:26Z"} 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:32.018+0000 7fb259a3c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:32.022+0000 7fb259a3c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:32Z"} 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:32Z"} 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:37.142+0000 7f3e0dcc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:37.142+0000 7f3e0dcc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.019898_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.019898Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:37Z"} 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.019898_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.019898Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:37Z"} 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.019898_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.019898Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.019898_288.1'} 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575075.019898_288.1 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:42.274+0000 7f1d6e290980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:42.274+0000 7f1d6e290980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:42Z"} 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:42Z"} 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:47.398+0000 7fe0e2880980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:47.398+0000 7fe0e2880980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:47Z"} 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:47Z"} 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:52.562+0000 7fc8e90e1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:52.562+0000 7fc8e90e1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:52Z"} 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:52Z"} 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:57.690+0000 7f9d932a6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:00:57.690+0000 7f9d932a6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:57Z"} 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:57Z"} 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:02.813+0000 7f90b465e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:02.813+0000 7f90b465e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:02Z"} 2024-03-04T19:23:51.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:02Z"} 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:07.937+0000 7f0c191a6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:07.937+0000 7f0c191a6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:07Z"} 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:07Z"} 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:13.065+0000 7fe97ac82980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:13.065+0000 7fe97ac82980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:13Z"} 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:13Z"} 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:18.193+0000 7f87ac9ae980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:18.193+0000 7f87ac9ae980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:18Z"} 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:18Z"} 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:23.317+0000 7f9d587cf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:23.317+0000 7f9d587cf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:23Z"} 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:23Z"} 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.097 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:28.441+0000 7f1ee3bfd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:28.441+0000 7f1ee3bfd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:28Z"} 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:28Z"} 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:33.565+0000 7f2ef7fdc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:33.565+0000 7f2ef7fdc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.424470_313.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.424470Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:33Z"} 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.424470_313.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.424470Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:33Z"} 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.424470_313.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.424470Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.424470_313.1'} 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575075.424470_313.1 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:38.688+0000 7fccc0ec9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:38.688+0000 7fccc0ec9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:38Z"} 2024-03-04T19:23:51.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:38Z"} 2024-03-04T19:23:51.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:43.812+0000 7f5f8c63a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:43.812+0000 7f5f8c63a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:43Z"} 2024-03-04T19:23:51.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:43Z"} 2024-03-04T19:23:51.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.101 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.101 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:48.936+0000 7f95511a8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:48.940+0000 7f95511a8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:48Z"} 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:48Z"} 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:54.064+0000 7fc270bea980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:54.064+0000 7fc270bea980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:54Z"} 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:54Z"} 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:59.188+0000 7f592d269980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:01:59.188+0000 7f592d269980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:59Z"} 2024-03-04T19:23:51.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:59Z"} 2024-03-04T19:23:51.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T19:23:51.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.937123_282.1 2024-03-04T19:23:51.104 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.104 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:04.316+0000 7f66623de980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:04.316+0000 7f66623de980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:04Z"} 2024-03-04T19:23:51.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:04Z"} 2024-03-04T19:23:51.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:09.440+0000 7ff43f4fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:09.444+0000 7ff43f4fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:09Z"} 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:09Z"} 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:14.620+0000 7f7a48ef8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:14.620+0000 7f7a48ef8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:14Z"} 2024-03-04T19:23:51.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:14Z"} 2024-03-04T19:23:51.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:19.747+0000 7f7952ab1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:19.747+0000 7f7952ab1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:19Z"} 2024-03-04T19:23:51.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:19Z"} 2024-03-04T19:23:51.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T19:23:51.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.937123_282.1 2024-03-04T19:23:51.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:24.875+0000 7fd819909980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:24.875+0000 7fd819909980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:24Z"} 2024-03-04T19:23:51.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:24Z"} 2024-03-04T19:23:51.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:30.003+0000 7f98aeaa0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:30.007+0000 7f98aeaa0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:30Z"} 2024-03-04T19:23:51.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:30Z"} 2024-03-04T19:23:51.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.110 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.110 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:35.131+0000 7f188d3fc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:35.131+0000 7f188d3fc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:35Z"} 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:35Z"} 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi153.stdout: INFO: 2024-03-04T18:02:36.207+0000 7fa53962a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:40.259+0000 7fee128d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:40.259+0000 7fee128d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:40Z"} 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:40Z"} 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:45.387+0000 7f7eb87ec980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:45.387+0000 7f7eb87ec980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:45Z"} 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:45Z"} 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:50.511+0000 7f7ed19cd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:50.511+0000 7f7ed19cd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:50Z"} 2024-03-04T19:23:51.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:50Z"} 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.984537_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.984537Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.984537_285.1'} 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.984537_285.1 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:55.639+0000 7fccfc8fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:02:55.643+0000 7fccfc8fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:55Z"} 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:55Z"} 2024-03-04T19:23:51.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.984537_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.984537Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.984537_285.1'} 2024-03-04T19:23:51.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.984537_285.1 2024-03-04T19:23:51.115 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.115 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:00.767+0000 7f4136719980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:00.771+0000 7f4136719980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:00Z"} 2024-03-04T19:23:51.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:00Z"} 2024-03-04T19:23:51.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.825024_336.1'} 2024-03-04T19:23:51.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575077.825024_336.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.116 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.116 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.116 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:51.116 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_period_incremental_sync 2024-03-04T19:23:51.116 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.117 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:51.117 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:51.117 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:51.117 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 936, in test_multi_period_incremental_sync 2024-03-04T19:23:51.117 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:51.117 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:51.117 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:51.117 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:23:51.117 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:23:51.117 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.117 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:51.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=ppfahn-42 2024-03-04T19:23:51.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-42/ 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-42/ 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-42/ 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.118 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:03:05 GMT 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:/ppfahn-42/ 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:tU1VwWnzM0JT5PGbQgquKhABkX0= 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:03:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tU1VwWnzM0JT5PGbQgquKhABkX0=', 'Content-Length': '0'} 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000086fe4b65a61911d9-0065e60cd9-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=ppfahn-43 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-43/ 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-43/ 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-43/ 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:03:05 GMT 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests:/ppfahn-43/ 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:xvEzjdVNPdXqcIFOTddK6DPfETQ= 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:03:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:xvEzjdVNPdXqcIFOTddK6DPfETQ=', 'Content-Length': '0'} 2024-03-04T19:23:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000fbd19d050df4c6e1-0065e60cd9-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:03:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=ppfahn-44 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-44/ 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-44/ 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-44/ 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.121 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:03:05 GMT 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:/ppfahn-44/ 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:FCyRPDq0sFMq2EdDx79FKUHKEjI= 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:03:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FCyRPDq0sFMq2EdDx79FKUHKEjI=', 'Content-Length': '0'} 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000035f7572cab1bf4a9-0065e60cd9-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:03:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709575386.010513_362.1","last_update":"2024-03-04T18:03:06.010513Z"}] 2024-03-04T19:23:51.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:06.290+0000 7f0e97fd5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:06.290+0000 7f0e97fd5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:06Z"} 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:06Z"} 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.937123_282.1 2024-03-04T19:23:51.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:11.426+0000 7f3db5652980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:11.426+0000 7f3db5652980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.132048_294.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.132048Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:11Z"} 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.132048_294.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.132048Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:11Z"} 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.132048_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.132048Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.132048_294.1'} 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.132048_294.1 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:16.566+0000 7f1cfd5bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:16.566+0000 7f1cfd5bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:16Z"} 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:16Z"} 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:21.690+0000 7f872aae3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:21.690+0000 7f872aae3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.132048_294.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.132048Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:21Z"} 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.132048_294.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.132048Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:21Z"} 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.132048_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.132048Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.132048_294.1'} 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.132048_294.1 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:26.818+0000 7f96db7de980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:26.818+0000 7f96db7de980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:26Z"} 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:26Z"} 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.937123_282.1 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:31.946+0000 7fc3c0f2d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:31.946+0000 7fc3c0f2d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:31Z"} 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:31Z"} 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:37.070+0000 7f1e180d9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:37.074+0000 7f1e180d9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:37Z"} 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:37Z"} 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.937123_282.1 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:42.218+0000 7fda27909980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:42.218+0000 7fda27909980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:42Z"} 2024-03-04T19:23:51.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:42Z"} 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:47.346+0000 7f5ff9024980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:47.346+0000 7f5ff9024980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:47Z"} 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:47Z"} 2024-03-04T19:23:51.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:52.469+0000 7f11891ee980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:52.469+0000 7f11891ee980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:52Z"} 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:52Z"} 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:57.593+0000 7f07d8961980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:03:57.593+0000 7f07d8961980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:57Z"} 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:57Z"} 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:02.717+0000 7f20f0e26980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:02.721+0000 7f20f0e26980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:02Z"} 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:02Z"} 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.133 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:07.865+0000 7fd0a1ecf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:07.869+0000 7fd0a1ecf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:07Z"} 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:07Z"} 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:13.005+0000 7f29be0d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:13.005+0000 7f29be0d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:13Z"} 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:13Z"} 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.937123_282.1 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:18.137+0000 7f6d5cae4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:18.137+0000 7f6d5cae4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:18Z"} 2024-03-04T19:23:51.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:18Z"} 2024-03-04T19:23:51.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T19:23:51.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.937123_282.1 2024-03-04T19:23:51.136 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.136 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:23.265+0000 7fa531d8a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:23.265+0000 7fa531d8a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:23Z"} 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:23Z"} 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:28.389+0000 7f8ddaf90980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:28.389+0000 7f8ddaf90980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:28Z"} 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:28Z"} 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.861873_275.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.861873Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.861873_275.1'} 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.861873_275.1 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:33.517+0000 7fa47f75a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:33.517+0000 7fa47f75a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:33Z"} 2024-03-04T19:23:51.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:33Z"} 2024-03-04T19:23:51.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.139 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.139 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:38.641+0000 7fed7b362980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:38.641+0000 7fed7b362980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:38Z"} 2024-03-04T19:23:51.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:38Z"} 2024-03-04T19:23:51.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.140 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.140 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:43.768+0000 7f7c2ae7b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:43.768+0000 7f7c2ae7b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:43Z"} 2024-03-04T19:23:51.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:43Z"} 2024-03-04T19:23:51.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:48.896+0000 7f93f04dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:48.896+0000 7f93f04dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:48Z"} 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:48Z"} 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:54.024+0000 7f8653971980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:54.024+0000 7f8653971980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:54Z"} 2024-03-04T19:23:51.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:54Z"} 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:59.192+0000 7f64b0f5b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:04:59.192+0000 7f64b0f5b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:59Z"} 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:59Z"} 2024-03-04T19:23:51.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.144 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.144 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:04.320+0000 7ff55ce32980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:04.320+0000 7ff55ce32980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:04Z"} 2024-03-04T19:23:51.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:04Z"} 2024-03-04T19:23:51.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:09.444+0000 7f8a21902980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:09.448+0000 7f8a21902980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:09Z"} 2024-03-04T19:23:51.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:09Z"} 2024-03-04T19:23:51.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.146 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.146 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:14.568+0000 7f44c2a08980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:14.568+0000 7f44c2a08980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:14Z"} 2024-03-04T19:23:51.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:14Z"} 2024-03-04T19:23:51.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:19.692+0000 7f4260ea6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:19.692+0000 7f4260ea6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:19Z"} 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:19Z"} 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:24.816+0000 7f4a8c315980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:24.816+0000 7f4a8c315980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:24Z"} 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:24Z"} 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:29.944+0000 7fe942232980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:29.948+0000 7fe942232980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:29Z"} 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:29Z"} 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.150 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:35.072+0000 7f0243cc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:35.072+0000 7f0243cc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:35Z"} 2024-03-04T19:23:51.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:35Z"} 2024-03-04T19:23:51.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.150 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:40.199+0000 7ff5cfa82980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:40.199+0000 7ff5cfa82980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:40Z"} 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:40Z"} 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:45.323+0000 7fb71872a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:45.323+0000 7fb71872a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:45Z"} 2024-03-04T19:23:51.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:45Z"} 2024-03-04T19:23:51.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:50.447+0000 7f15b9bfb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:50.447+0000 7f15b9bfb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:50Z"} 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:50Z"} 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:55.575+0000 7f4c92d00980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:05:55.583+0000 7f4c92d00980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:55Z"} 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:55Z"} 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:00.715+0000 7f5de1b70980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:00.715+0000 7f5de1b70980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:00Z"} 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:00Z"} 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:05.843+0000 7fbecaf25980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:05.847+0000 7fbecaf25980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:05Z"} 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:05Z"} 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:10.975+0000 7fa56e18b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:10.975+0000 7fa56e18b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:10Z"} 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:10Z"} 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:16.102+0000 7f77a5427980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:16.102+0000 7f77a5427980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:16Z"} 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:16Z"} 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:21.234+0000 7feddf602980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:21.234+0000 7feddf602980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:21Z"} 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:21Z"} 2024-03-04T19:23:51.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:26.382+0000 7fc35ae40980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:26.382+0000 7fc35ae40980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:26Z"} 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:26Z"} 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:31.506+0000 7f20e6d10980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:31.510+0000 7f20e6d10980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:31Z"} 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:31Z"} 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:36.638+0000 7efdd3821980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:36.638+0000 7efdd3821980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:36Z"} 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:36Z"} 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.861873_275.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.861873Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.861873_275.1'} 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.861873_275.1 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.161 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:41.766+0000 7fe9fe1a8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:41.766+0000 7fe9fe1a8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:41Z"} 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:41Z"} 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:46.914+0000 7ff549a3e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:46.914+0000 7ff549a3e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:46Z"} 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:46Z"} 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:52.050+0000 7ffbe6fe8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:52.050+0000 7ffbe6fe8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:52Z"} 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:52Z"} 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:57.177+0000 7f2db4fea980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:06:57.181+0000 7f2db4fea980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:57Z"} 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:57Z"} 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:02.305+0000 7fc9bf976980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:02.309+0000 7fc9bf976980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.081961_289.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.081961Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:02Z"} 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.081961_289.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.081961Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:02Z"} 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.081961_289.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.081961Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.081961_289.1'} 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.081961_289.1 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:07.449+0000 7fbf361af980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:07.449+0000 7fbf361af980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:07Z"} 2024-03-04T19:23:51.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:07Z"} 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:12.577+0000 7f0c1130c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:12.577+0000 7f0c1130c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:12Z"} 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:12Z"} 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:17.709+0000 7f85b10ce980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:17.709+0000 7f85b10ce980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.088577_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.088577Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:17Z"} 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.088577_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.088577Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:17Z"} 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.088577_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.088577Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.088577_290.1'} 2024-03-04T19:23:51.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.088577_290.1 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:22.837+0000 7f5971d60980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:22.837+0000 7f5971d60980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.088577_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.088577Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:22Z"} 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.088577_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.088577Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:22Z"} 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.088577_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.088577Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.088577_290.1'} 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.088577_290.1 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:27.957+0000 7fe6ac772980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:27.957+0000 7fe6ac772980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:27Z"} 2024-03-04T19:23:51.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:27Z"} 2024-03-04T19:23:51.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.170 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.170 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:33.085+0000 7f01d218c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:33.085+0000 7f01d218c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:33Z"} 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:33Z"} 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi153.stdout: INFO: 2024-03-04T18:07:36.769+0000 7fa53962a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:38.216+0000 7f93b7b1a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:38.216+0000 7f93b7b1a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.014763_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.014763Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:38Z"} 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.014763_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.014763Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:38Z"} 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.014763_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.014763Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.014763_287.1'} 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.014763_287.1 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:43.344+0000 7f733cc40980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:43.348+0000 7f733cc40980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:43Z"} 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:43Z"} 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:48.472+0000 7f3d67cef980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:48.476+0000 7f3d67cef980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:48Z"} 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:48Z"} 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.861873_275.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.861873Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.861873_275.1'} 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.861873_275.1 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:53.600+0000 7f78d66d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:53.600+0000 7f78d66d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:53Z"} 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:53Z"} 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:58.728+0000 7f8912488980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:07:58.732+0000 7f8912488980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:58Z"} 2024-03-04T19:23:51.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:58Z"} 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:03.856+0000 7f2a0c393980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:03.860+0000 7f2a0c393980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:03Z"} 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.984537_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.984537Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:03Z"} 2024-03-04T19:23:51.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.984537_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.984537Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.984537_285.1'} 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575074.984537_285.1 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:08.992+0000 7f793548a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:08.992+0000 7f793548a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.141267_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.141267Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:08Z"} 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.141267_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.141267Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:08Z"} 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.141267_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.141267Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575386.010513_362.1'} 2024-03-04T19:23:51.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.141267_296.1'} 2024-03-04T19:23:51.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575386.010513_362.1 target=1_1709575075.141267_296.1 2024-03-04T19:23:51.178 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.178 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.178 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:51.178 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_datalog_autotrim 2024-03-04T19:23:51.178 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.178 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:51.178 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1021, in test_datalog_autotrim 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=ppfahn-45 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-45/ 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-45/ 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-45/ 2024-03-04T19:23:51.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:08:14 GMT 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests:/ppfahn-45/ 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:suSXJ/VJKMFY+GtJ+WlgMhEpxjI= 2024-03-04T19:23:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:suSXJ/VJKMFY+GtJ+WlgMhEpxjI=', 'Content-Length': '0'} 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000fcf4ea21f29ee96e-0065e60e0e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=ppfahn-46 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-46/ 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-46/ 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-46/ 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.181 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:08:14 GMT 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests:/ppfahn-46/ 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:FbmbJuALSYhgLTcEmvtG7mvM7fY= 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FbmbJuALSYhgLTcEmvtG7mvM7fY=', 'Content-Length': '0'} 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000c5937fc317071284-0065e60e0e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=ppfahn-47 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-47/ 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-47/ 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-47/ 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:08:14 GMT 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:/ppfahn-47/ 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:XEjhPCgV+B80W3he7f9eeXVZVtU= 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XEjhPCgV+B80W3he7f9eeXVZVtU=', 'Content-Length': '0'} 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000659ded98b03d680e-0065e60e0e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-45/ 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-45/ 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-45/ 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:08:14 GMT 2024-03-04T19:23:51.184 INFO:tasks.rgw_multisite_tests:/ppfahn-45/ 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:XaJ8BxCu51i63Enp7PsthffGI2A= 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XaJ8BxCu51i63Enp7PsthffGI2A=', 'Content-Length': '0'} 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b263d024a4802d9-0065e60e0e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-45/key 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-45/key 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-45/key 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.185 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:08:14 GMT 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:/ppfahn-45/key 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:Qv3fj3rGqiE+lvX2L856dIni0Ec= 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Qv3fj3rGqiE+lvX2L856dIni0Ec='} 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072efd20f71d2651e-0065e60e0e-4349-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-46/ 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-46/ 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-46/ 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:08:14 GMT 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests:/ppfahn-46/ 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:FxD4ssHhqUiQT2doSGQNgnXkELo= 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:FxD4ssHhqUiQT2doSGQNgnXkELo=', 'Content-Length': '0'} 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000261a348c6c77c07e-0065e60e0e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-46/key 2024-03-04T19:23:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-46/key 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-46/key 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:08:14 GMT 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:/ppfahn-46/key 2024-03-04T19:23:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:ex3lukmqPCx5dG5ZNyNgXh9fb5I= 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ex3lukmqPCx5dG5ZNyNgXh9fb5I='} 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e96d1a884e4426b-0065e60e0e-4306-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-47/ 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-47/ 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-47/ 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.189 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:08:14 GMT 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:/ppfahn-47/ 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:VRlljhMkpkcRxkVkZXnOGLF/alE= 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VRlljhMkpkcRxkVkZXnOGLF/alE=', 'Content-Length': '0'} 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063dce749e837266b-0065e60e0e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-47/key 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-47/key 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-47/key 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:08:14 GMT 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:/ppfahn-47/key 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:hBSNuy382yAt/lM9W3iz8nwMb5c= 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:08:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hBSNuy382yAt/lM9W3iz8nwMb5c='} 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016c9eaa4e9709bb8-0065e60e0e-4356-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709575694.259906_393.1","last_update":"2024-03-04T18:08:14.259906Z"}] 2024-03-04T19:23:51.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:14.656+0000 7fadc9c0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:14.656+0000 7fadc9c0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:14Z"} 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:14Z"} 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi153.stdout: INFO: 2024-03-04T18:08:16.780+0000 7fa54263c640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:19.812+0000 7f435591c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:19.812+0000 7f435591c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:19Z"} 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:19Z"} 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.937123_282.1 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi153.stdout: INFO: 2024-03-04T18:08:24.851+0000 7fa541e3b640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:24.947+0000 7f37e6260980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:24.947+0000 7f37e6260980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:24Z"} 2024-03-04T19:23:51.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:24Z"} 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:30.071+0000 7f2bd1c88980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:30.071+0000 7f2bd1c88980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:30Z"} 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:30Z"} 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.196 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.196 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:35.199+0000 7f12182de980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:35.203+0000 7f12182de980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:35Z"} 2024-03-04T19:23:51.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:35Z"} 2024-03-04T19:23:51.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.937123_282.1 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:40.331+0000 7f4d1021c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:40.331+0000 7f4d1021c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:40Z"} 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:40Z"} 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:45.459+0000 7f71c8a66980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:45.459+0000 7f71c8a66980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:45Z"} 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:45Z"} 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.198 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:50.591+0000 7ff7f8511980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:50.591+0000 7ff7f8511980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:50Z"} 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:50Z"} 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:55.715+0000 7f5a33be4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:08:55.719+0000 7f5a33be4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:55Z"} 2024-03-04T19:23:51.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:55Z"} 2024-03-04T19:23:51.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.200 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.200 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:00.847+0000 7fe4f351e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:00.847+0000 7fe4f351e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:00Z"} 2024-03-04T19:23:51.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:00Z"} 2024-03-04T19:23:51.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.201 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.201 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:05.975+0000 7f71bb39e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:05.975+0000 7f71bb39e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:05Z"} 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:05Z"} 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:11.102+0000 7fbe796ef980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:11.102+0000 7fbe796ef980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:11Z"} 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:11Z"} 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:16.230+0000 7f534205f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:16.234+0000 7f534205f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:16Z"} 2024-03-04T19:23:51.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:16Z"} 2024-03-04T19:23:51.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T19:23:51.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.937123_282.1 2024-03-04T19:23:51.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:21.358+0000 7fd0f249d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:21.362+0000 7fd0f249d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:21Z"} 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:21Z"} 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.388355_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.388355Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.388355_309.1'} 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575075.388355_309.1 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:26.486+0000 7fc8ebf3a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:26.486+0000 7fc8ebf3a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:26Z"} 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:26Z"} 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:31.614+0000 7f37afb83980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:31.614+0000 7f37afb83980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:31Z"} 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:31Z"} 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:36.742+0000 7f1a81922980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:36.742+0000 7f1a81922980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:36Z"} 2024-03-04T19:23:51.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:36Z"} 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:41.870+0000 7fe559db8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:41.870+0000 7fe559db8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:41Z"} 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:41Z"} 2024-03-04T19:23:51.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:46.998+0000 7fcb223f5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:46.998+0000 7fcb223f5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.014763_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.014763Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:47Z"} 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.014763_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.014763Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:47Z"} 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.014763_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.014763Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.014763_287.1'} 2024-03-04T19:23:51.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575075.014763_287.1 2024-03-04T19:23:51.210 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.210 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:52.126+0000 7fa9dc957980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:52.126+0000 7fa9dc957980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:52Z"} 2024-03-04T19:23:51.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:52Z"} 2024-03-04T19:23:51.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:57.254+0000 7f30c4aa7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:09:57.258+0000 7f30c4aa7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:57Z"} 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:57Z"} 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.212 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.212 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:02.381+0000 7fa50023c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:02.381+0000 7fa50023c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.973154_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.973154Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:02Z"} 2024-03-04T19:23:51.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.973154_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.973154Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:02Z"} 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.973154_284.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.973154Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.973154_284.1'} 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.973154_284.1 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:07.509+0000 7f2136e62980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:07.509+0000 7f2136e62980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.973154_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.973154Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:07Z"} 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.973154_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.973154Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:07Z"} 2024-03-04T19:23:51.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.973154_284.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.973154Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.973154_284.1'} 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.973154_284.1 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:12.637+0000 7faed5d6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:12.637+0000 7faed5d6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:12Z"} 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:12Z"} 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:17.765+0000 7f6a06ca5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:17.765+0000 7f6a06ca5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:17Z"} 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:17Z"} 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.216 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.216 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:22.893+0000 7fb7c4184980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:22.893+0000 7fb7c4184980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:22Z"} 2024-03-04T19:23:51.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:22Z"} 2024-03-04T19:23:51.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:28.021+0000 7f69e7ff2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:28.021+0000 7f69e7ff2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:28Z"} 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:28Z"} 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.217 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:33.149+0000 7f6833b7f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:33.149+0000 7f6833b7f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:33Z"} 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:33Z"} 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:38.277+0000 7f1b6b560980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:38.277+0000 7f1b6b560980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:38Z"} 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:38Z"} 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.937123_282.1 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:43.401+0000 7fea97301980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:43.401+0000 7fea97301980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:43Z"} 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:43Z"} 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:48.529+0000 7f4ee293d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:48.529+0000 7f4ee293d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:48Z"} 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:48Z"} 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:53.672+0000 7fd4e8192980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:53.672+0000 7fd4e8192980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:53Z"} 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:53Z"} 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.937123_282.1 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:58.800+0000 7ff8e005b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:10:58.800+0000 7ff8e005b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:58Z"} 2024-03-04T19:23:51.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:58Z"} 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:03.928+0000 7f196296a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:03.928+0000 7f196296a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:03Z"} 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:03Z"} 2024-03-04T19:23:51.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:09.056+0000 7ff5892cc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:09.056+0000 7ff5892cc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:09Z"} 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:09Z"} 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:14.184+0000 7fb4404cf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:14.188+0000 7fb4404cf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:14Z"} 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.937123_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.937123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:14Z"} 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.937123_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.937123Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.937123_282.1'} 2024-03-04T19:23:51.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.937123_282.1 2024-03-04T19:23:51.226 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.226 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:19.312+0000 7f7779413980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:19.312+0000 7f7779413980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:19Z"} 2024-03-04T19:23:51.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:19Z"} 2024-03-04T19:23:51.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:24.444+0000 7f2b6d780980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:24.444+0000 7f2b6d780980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:24Z"} 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.861873_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.861873Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:24Z"} 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.861873_275.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.861873Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.861873_275.1'} 2024-03-04T19:23:51.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.861873_275.1 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:29.568+0000 7f14272b7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:29.572+0000 7f14272b7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:29Z"} 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:29Z"} 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:34.696+0000 7ffad7826980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:34.696+0000 7ffad7826980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:34Z"} 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:34Z"} 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:39.824+0000 7fb6f6421980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:39.824+0000 7fb6f6421980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:39Z"} 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:39Z"} 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:44.964+0000 7f5d41e27980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:44.964+0000 7f5d41e27980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:44Z"} 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:44Z"} 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:50.091+0000 7fa4750b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:50.095+0000 7fa4750b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:50Z"} 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:50Z"} 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:55.219+0000 7ff48bd75980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:11:55.219+0000 7ff48bd75980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:55Z"} 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:55Z"} 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:00.347+0000 7ff5d419d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:00.351+0000 7ff5d419d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:00Z"} 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.930202_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.930202Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:00Z"} 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.930202_281.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.930202Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.930202_281.1'} 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.930202_281.1 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:05.475+0000 7f5e91167980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:05.475+0000 7f5e91167980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:05Z"} 2024-03-04T19:23:51.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:05Z"} 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:10.603+0000 7ff8d1a77980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:10.603+0000 7ff8d1a77980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:10Z"} 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:10Z"} 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:15.731+0000 7fb741047980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:15.731+0000 7fb741047980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:15Z"} 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:15Z"} 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:20.863+0000 7fea7aa35980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:20.867+0000 7fea7aa35980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:20Z"} 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:20Z"} 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:25.995+0000 7f16fd211980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:25.995+0000 7f16fd211980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:26Z"} 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:26Z"} 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.238 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:31.123+0000 7f42e07e4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:31.127+0000 7f42e07e4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:31Z"} 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:31Z"} 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:36.263+0000 7fbcb68fa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:36.263+0000 7fbcb68fa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:36Z"} 2024-03-04T19:23:51.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:36Z"} 2024-03-04T19:23:51.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:41.391+0000 7f711f02c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:41.391+0000 7f711f02c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:41Z"} 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:41Z"} 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi153.stdout: INFO: 2024-03-04T18:12:44.582+0000 7fa53962a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.241 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:46.518+0000 7f3bf34b3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:46.518+0000 7f3bf34b3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:46Z"} 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:46Z"} 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:51.646+0000 7f3d76c62980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:51.646+0000 7f3d76c62980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:51Z"} 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:51Z"} 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:56.774+0000 7f491cf8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:12:56.774+0000 7f491cf8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:56Z"} 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:56Z"} 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:13:01.902+0000 7fa49161a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:13:01.902+0000 7fa49161a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:01Z"} 2024-03-04T19:23:51.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:01Z"} 2024-03-04T19:23:51.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.245 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.245 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:13:07.030+0000 7fe6b08ac980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:13:07.034+0000 7fe6b08ac980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:07Z"} 2024-03-04T19:23:51.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:07Z"} 2024-03-04T19:23:51.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.247 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.247 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:13:12.166+0000 7fc0bf4a3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:13:12.166+0000 7fc0bf4a3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:12Z"} 2024-03-04T19:23:51.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.965270_283.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.965270Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:12Z"} 2024-03-04T19:23:51.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.965270_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.965270Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.965270_283.1'} 2024-03-04T19:23:51.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.965270_283.1 2024-03-04T19:23:51.248 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.248 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:13:17.330+0000 7f5374665980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:13:17.330+0000 7f5374665980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:17Z"} 2024-03-04T19:23:51.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575074.919962_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:54.919962Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:17Z"} 2024-03-04T19:23:51.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575074.919962_279.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:54.919962Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.259906_393.1'} 2024-03-04T19:23:51.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575074.919962_279.1'} 2024-03-04T19:23:51.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709575694.259906_393.1 target=1_1709575074.919962_279.1 2024-03-04T19:23:51.249 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.249 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.249 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:51.249 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_zone_redirect 2024-03-04T19:23:51.249 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.250 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:51.250 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:51.250 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:51.250 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1064, in test_multi_zone_redirect 2024-03-04T19:23:51.250 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:51.250 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:51.250 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:51.250 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:23:51.250 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:23:51.250 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.250 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:51.250 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T19:23:51.250 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"id":"185c722f-6c40-48e4-8855-bf605b5498c7","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"1THAB257QDW2GNAH","secret_key":"QoUA0kEYOw3e9ZPsidmlofuWIy510GDX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","notif_pool":"test-zone2.rgw.log:notif"} 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: Sending period to new master zone d96636ac-054d-40e3-90f3-e3912da784b2 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"id":"f6a09d88-afdd-40f4-a159-dda085c373df","epoch":4,"predecessor_uuid":"aa7942d8-051d-40ca-b86f-359c9c0301cc","sync_status":[],"period_map":{"id":"f6a09d88-afdd-40f4-a159-dda085c373df","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","zones":[{"id":"185c722f-6c40-48e4-8855-bf605b5498c7","name":"test-zone2","endpoints":["http://smithi153.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"c146b41a-1ed1-469f-afe6-2dddc3577925","name":"test-zone3","endpoints":["http://smithi136.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"d96636ac-054d-40e3-90f3-e3912da784b2","name":"test-zone1","endpoints":["http://smithi136.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"185c722f-6c40-48e4-8855-bf605b5498c7","val":1962731440},{"key":"c146b41a-1ed1-469f-afe6-2dddc3577925","val":3710957901},{"key":"d96636ac-054d-40e3-90f3-e3912da784b2","val":2093114680}]},"master_zonegroup":"test-zonegroup","master_zone":"d96636ac-054d-40e3-90f3-e3912da784b2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"fcff5afe-947d-4c2e-8449-fe8f8ae33dc5","realm_epoch":2} 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: Set sync_from_all flag on zone test-zone2 to false 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=ppfahn-48 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-48/ 2024-03-04T19:23:51.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-48/ 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-48/ 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:14:22 GMT 2024-03-04T19:23:51.252 INFO:tasks.rgw_multisite_tests:/ppfahn-48/ 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:eGGUzqY5elGWUKQZm6Zd1kL2BKQ= 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:14:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eGGUzqY5elGWUKQZm6Zd1kL2BKQ=', 'Content-Length': '0'} 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000fccba63d48f35bb7-0065e60f7e-5302-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:14:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-48/testredirect 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-48/testredirect 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-48/testredirect 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:14:22 GMT 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:/ppfahn-48/testredirect 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oe9XnGFVLFRLI+s4jED2Q2uxQyk= 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:14:22 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oe9XnGFVLFRLI+s4jED2Q2uxQyk='} 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a204acfd39fd5a3-0065e60f7e-5302-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:14:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709576062.712249_431.1","last_update":"2024-03-04T18:14:22.712249Z"}] 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:22.981+0000 7fc75bbea980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:22.985+0000 7fc75bbea980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:22Z"} 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:22Z"} 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T19:23:51.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T19:23:51.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:28.113+0000 7f99b03af980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:28.113+0000 7f99b03af980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:28Z"} 2024-03-04T19:23:51.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:28Z"} 2024-03-04T19:23:51.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:33.241+0000 7f8ce1aa9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:33.241+0000 7f8ce1aa9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:33Z"} 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:33Z"} 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T19:23:51.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:38.373+0000 7f518ac4c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:38.377+0000 7f518ac4c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:38Z"} 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:38Z"} 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:43.504+0000 7f434df46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:43.504+0000 7f434df46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:43Z"} 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:43Z"} 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:48.636+0000 7f4b6f078980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:48.636+0000 7f4b6f078980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:48Z"} 2024-03-04T19:23:51.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:48Z"} 2024-03-04T19:23:51.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T19:23:51.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.380518_307.1 2024-03-04T19:23:51.260 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.260 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:53.764+0000 7fee696d2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:53.764+0000 7fee696d2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:53Z"} 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:53Z"} 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.380518_307.1 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:14:54.019+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:14:54.119+0000 7f5c190f6640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:14:54.163+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:14:54.163+0000 7f5c29116640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:58.888+0000 7fa58dd0f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:14:58.892+0000 7fa58dd0f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:58Z"} 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:58Z"} 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:04.016+0000 7f0ebeded980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:04.016+0000 7f0ebeded980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:04Z"} 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:04Z"} 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:09.144+0000 7f0205354980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:09.144+0000 7f0205354980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.417189_311.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.417189Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:09Z"} 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.417189_311.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.417189Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:09Z"} 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.417189_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.417189Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.417189_311.1'} 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.417189_311.1 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:14.272+0000 7f8b934ec980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:14.276+0000 7f8b934ec980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.837105_354.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.837105Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:14Z"} 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.837105_354.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.837105Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:14Z"} 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575385.837105_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:03:05.837105Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575385.837105_354.1'} 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575385.837105_354.1 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:19.404+0000 7f43df131980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:19.408+0000 7f43df131980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:19Z"} 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:19Z"} 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:15:24.139+0000 7fae3197f640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:24.532+0000 7fcd93736980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:24.532+0000 7fcd93736980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:24Z"} 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:24Z"} 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:29.660+0000 7fc328164980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:29.660+0000 7fc328164980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:29Z"} 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:29Z"} 2024-03-04T19:23:51.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:34.787+0000 7f612edfd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:34.787+0000 7f612edfd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:34Z"} 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:34Z"} 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:39.919+0000 7fc900758980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:39.919+0000 7fc900758980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:39Z"} 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:39Z"} 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:45.043+0000 7f2a8b950980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:45.047+0000 7f2a8b950980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:45Z"} 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:45Z"} 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:50.179+0000 7f2306e61980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:50.183+0000 7f2306e61980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:50Z"} 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:50Z"} 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:55.331+0000 7f62ff4b4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:15:55.331+0000 7f62ff4b4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:55Z"} 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:55Z"} 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:00.459+0000 7f35ac25f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:00.463+0000 7f35ac25f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:00Z"} 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:00Z"} 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:05.595+0000 7f01efabd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:05.599+0000 7f01efabd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:05Z"} 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:05Z"} 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:10.727+0000 7f20a7dc0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:10.731+0000 7f20a7dc0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:10Z"} 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:10Z"} 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:15.863+0000 7f9e2db7b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:15.863+0000 7f9e2db7b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:15Z"} 2024-03-04T19:23:51.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:15Z"} 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:20.999+0000 7f756c47f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:20.999+0000 7f756c47f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:21Z"} 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:21Z"} 2024-03-04T19:23:51.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:26.127+0000 7fa8c0b26980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:26.127+0000 7fa8c0b26980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:26Z"} 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:26Z"} 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:31.258+0000 7f8dcf7e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:31.258+0000 7f8dcf7e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:31Z"} 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:31Z"} 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T19:23:51.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:36.398+0000 7f1914182980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:36.402+0000 7f1914182980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:36Z"} 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:36Z"} 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:41.526+0000 7facefb90980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:41.530+0000 7facefb90980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:41Z"} 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:41Z"} 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:46.658+0000 7f2d5c88d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:46.658+0000 7f2d5c88d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:46Z"} 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:46Z"} 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:51.790+0000 7fb13caa3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:51.794+0000 7fb13caa3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:51Z"} 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:51Z"} 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:56.922+0000 7f61b1306980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:16:56.922+0000 7f61b1306980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:56Z"} 2024-03-04T19:23:51.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:56Z"} 2024-03-04T19:23:51.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:02.046+0000 7f40f20dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:02.046+0000 7f40f20dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:02Z"} 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:02Z"} 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:07.174+0000 7f570aa8e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:07.174+0000 7f570aa8e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:07Z"} 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:07Z"} 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:12.322+0000 7f9359bc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:12.322+0000 7f9359bc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:12Z"} 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:12Z"} 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:17.454+0000 7f5e2cda7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:17.454+0000 7f5e2cda7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:17Z"} 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:17Z"} 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.495723_318.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.495723Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.495723_318.1'} 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.495723_318.1 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:22.581+0000 7ff6af76d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:22.585+0000 7ff6af76d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:22Z"} 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:22Z"} 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:27.733+0000 7fc9d3e8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:27.733+0000 7fc9d3e8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:27Z"} 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:27Z"} 2024-03-04T19:23:51.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:32.861+0000 7f4cafd4d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:32.865+0000 7f4cafd4d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:32Z"} 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:32Z"} 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:37.993+0000 7f4f24a9d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:37.993+0000 7f4f24a9d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:37Z"} 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:37Z"} 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:43.125+0000 7fdc78614980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:43.125+0000 7fdc78614980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:43Z"} 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:43Z"} 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.380518_307.1 2024-03-04T19:23:51.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:48.257+0000 7f223d44c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:48.257+0000 7f223d44c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.480321_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.480321Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:48Z"} 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.480321_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.480321Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:48Z"} 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.480321_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.480321Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.480321_315.1'} 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.480321_315.1 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:53.385+0000 7fa1a56a0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:53.385+0000 7fa1a56a0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:53Z"} 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:53Z"} 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.380518_307.1 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:58.513+0000 7f9e99093980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:17:58.517+0000 7f9e99093980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.417189_311.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.417189Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:58Z"} 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.417189_311.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.417189Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:58Z"} 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.417189_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.417189Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.417189_311.1'} 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.417189_311.1 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:03.645+0000 7f2d9fc28980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:03.645+0000 7f2d9fc28980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:03Z"} 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:03Z"} 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:08.777+0000 7f3b41f4d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:08.777+0000 7f3b41f4d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:08Z"} 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:08Z"} 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:13.909+0000 7f5d54a94980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:13.909+0000 7f5d54a94980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:13Z"} 2024-03-04T19:23:51.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:13Z"} 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:19.036+0000 7fa4dd524980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:19.036+0000 7fa4dd524980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:19Z"} 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:19Z"} 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:24.172+0000 7f32dedee980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:24.172+0000 7f32dedee980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:24Z"} 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:24Z"} 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.299 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.299 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:29.308+0000 7f373f0f7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:29.308+0000 7f373f0f7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:29Z"} 2024-03-04T19:23:51.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:29Z"} 2024-03-04T19:23:51.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:34.436+0000 7fa2be5fb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:34.436+0000 7fa2be5fb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:34Z"} 2024-03-04T19:23:51.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:34Z"} 2024-03-04T19:23:51.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:39.564+0000 7f1077fab980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:39.568+0000 7f1077fab980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:39Z"} 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:39Z"} 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.301 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:44.708+0000 7f0a1b60e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:44.712+0000 7f0a1b60e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:44Z"} 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:44Z"} 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:49.868+0000 7f7dcbc4b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:49.868+0000 7f7dcbc4b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:49Z"} 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:49Z"} 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:54.996+0000 7f47a5703980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:18:54.996+0000 7f47a5703980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:55Z"} 2024-03-04T19:23:51.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:55Z"} 2024-03-04T19:23:51.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:00.140+0000 7f57ab06e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:00.140+0000 7f57ab06e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:00Z"} 2024-03-04T19:23:51.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:00Z"} 2024-03-04T19:23:51.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:05.268+0000 7f3e343c6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:05.268+0000 7f3e343c6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:05Z"} 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:05Z"} 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:10.403+0000 7fc0e8649980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:10.403+0000 7fc0e8649980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:10Z"} 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:10Z"} 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:15.539+0000 7f3f4a1ad980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:15.543+0000 7f3f4a1ad980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:15Z"} 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:15Z"} 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:20.671+0000 7fddb27bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:20.671+0000 7fddb27bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:20Z"} 2024-03-04T19:23:51.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:20Z"} 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:25.803+0000 7ff13f5d4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:25.803+0000 7ff13f5d4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:25Z"} 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:25Z"} 2024-03-04T19:23:51.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.310 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.310 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.310 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:51.310 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_zonegroup_remove 2024-03-04T19:23:51.310 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.310 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1093, in test_zonegroup_remove 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709576062.712249_431.1","last_update":"2024-03-04T18:14:22.712249Z"}] 2024-03-04T19:23:51.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:31.071+0000 7f32e4fdf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:31.071+0000 7f32e4fdf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:31Z"} 2024-03-04T19:23:51.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:31Z"} 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:36.203+0000 7ff91ac11980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:36.203+0000 7ff91ac11980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:36Z"} 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:36Z"} 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:41.335+0000 7fc40d495980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:41.335+0000 7fc40d495980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:41Z"} 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:41Z"} 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:46.459+0000 7fd828820980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:46.459+0000 7fd828820980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:46Z"} 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:46Z"} 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.316 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.316 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:51.595+0000 7f0070924980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:51.595+0000 7f0070924980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:51Z"} 2024-03-04T19:23:51.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:51Z"} 2024-03-04T19:23:51.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:19:54.033+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:19:54.173+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:19:54.181+0000 7f5c190f6640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:56.723+0000 7f1745879980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:19:56.723+0000 7f1745879980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:56Z"} 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:56Z"} 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:01.851+0000 7f119966d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:01.851+0000 7f119966d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:01Z"} 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:01Z"} 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:06.978+0000 7fb700f1b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:06.982+0000 7fb700f1b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:06Z"} 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:06Z"} 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:12.110+0000 7f85e6c5d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:12.110+0000 7f85e6c5d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:12Z"} 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:12Z"} 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:17.242+0000 7faecbb66980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:17.246+0000 7faecbb66980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:17Z"} 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.217214_299.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.217214Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:17Z"} 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.217214_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.217214Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.217214_299.1'} 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.217214_299.1 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:22.374+0000 7f15cddc4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:22.374+0000 7f15cddc4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:22Z"} 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:22Z"} 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:20:24.164+0000 7fae3197f640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:27.502+0000 7f3660310980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:27.506+0000 7f3660310980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:27Z"} 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:27Z"} 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:32.630+0000 7f3c30ceb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:32.634+0000 7f3c30ceb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:32Z"} 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:32Z"} 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:37.766+0000 7f554c248980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:37.766+0000 7f554c248980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:37Z"} 2024-03-04T19:23:51.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:37Z"} 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:42.894+0000 7fba8e06a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:42.898+0000 7fba8e06a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:42Z"} 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:42Z"} 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:48.030+0000 7f938ba84980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:48.034+0000 7f938ba84980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:48Z"} 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:48Z"} 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:53.166+0000 7f39f191e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:53.166+0000 7f39f191e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:53Z"} 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:53Z"} 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.327 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:58.313+0000 7f20b08ab980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:20:58.313+0000 7f20b08ab980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:58Z"} 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:58Z"} 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:03.445+0000 7f283c79b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:03.445+0000 7f283c79b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:03Z"} 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:03Z"} 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:08.573+0000 7f618005f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:08.577+0000 7f618005f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:08Z"} 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:08Z"} 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.380518_307.1 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:13.701+0000 7fc3b6b0f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:13.705+0000 7fc3b6b0f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:13Z"} 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:13Z"} 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:18.833+0000 7fca02968980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:18.837+0000 7fca02968980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:18Z"} 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:18Z"} 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:23.965+0000 7f4f60cb2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:23.965+0000 7f4f60cb2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:23Z"} 2024-03-04T19:23:51.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:23Z"} 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:29.105+0000 7f2672746980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:29.105+0000 7f2672746980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:29Z"} 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:29Z"} 2024-03-04T19:23:51.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:34.257+0000 7f45a90eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:34.257+0000 7f45a90eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:34Z"} 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:34Z"} 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:39.397+0000 7f3f2c45c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:39.401+0000 7f3f2c45c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:39Z"} 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:39Z"} 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:44.537+0000 7eff90032980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:44.537+0000 7eff90032980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:44Z"} 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:44Z"} 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:49.669+0000 7fdda0a1b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:49.669+0000 7fdda0a1b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:49Z"} 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:49Z"} 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:54.816+0000 7fb955c32980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:54.816+0000 7fb955c32980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:54Z"} 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:54Z"} 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:59.952+0000 7f39f73be980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:21:59.952+0000 7f39f73be980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:59Z"} 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:59Z"} 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:05.088+0000 7fb98c5ea980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:05.088+0000 7fb98c5ea980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:05Z"} 2024-03-04T19:23:51.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:05Z"} 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:10.244+0000 7f7bd1e48980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:10.244+0000 7f7bd1e48980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:10Z"} 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:10Z"} 2024-03-04T19:23:51.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:15.380+0000 7f5fe2d39980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:15.384+0000 7f5fe2d39980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:15Z"} 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:15Z"} 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:20.520+0000 7f286453b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:20.524+0000 7f286453b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:20Z"} 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:20Z"} 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:25.656+0000 7f3305d4b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:25.656+0000 7f3305d4b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:25Z"} 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:25Z"} 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575076.582500_321.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:56.582500Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575076.582500_321.1'} 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575076.582500_321.1 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.343 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:30.804+0000 7fdcc9bb2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:30.804+0000 7fdcc9bb2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:30Z"} 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:30Z"} 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.523626_319.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.523626Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.523626_319.1'} 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.523626_319.1 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:35.936+0000 7fb85887e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:35.936+0000 7fb85887e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:35Z"} 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:35Z"} 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:41.088+0000 7f75b2333980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:41.088+0000 7f75b2333980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:41Z"} 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:41Z"} 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:46.248+0000 7f2d037ec980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:46.252+0000 7f2d037ec980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:46Z"} 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:46Z"} 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:51.399+0000 7fb4a1b01980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:51.399+0000 7fb4a1b01980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:51Z"} 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:51Z"} 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:56.535+0000 7f338a0a0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:22:56.535+0000 7f338a0a0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:56Z"} 2024-03-04T19:23:51.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:56Z"} 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:01.683+0000 7fa2aac42980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:01.687+0000 7fa2aac42980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:01Z"} 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:01Z"} 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:06.831+0000 7f83a203a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:06.831+0000 7f83a203a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:06Z"} 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:06Z"} 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:11.983+0000 7f35edc51980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:11.983+0000 7f35edc51980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:11Z"} 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:11Z"} 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.495723_318.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.495723Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.495723_318.1'} 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.495723_318.1 2024-03-04T19:23:51.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:17.135+0000 7f3824fc1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:17.135+0000 7f3824fc1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:17Z"} 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:17Z"} 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:22.283+0000 7f9077926980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:22.283+0000 7f9077926980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:22Z"} 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:22Z"} 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575076.582500_321.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:56.582500Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575076.582500_321.1'} 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575076.582500_321.1 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:27.443+0000 7f2444e6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:27.447+0000 7f2444e6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:27Z"} 2024-03-04T19:23:51.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:27Z"} 2024-03-04T19:23:51.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:32.587+0000 7f527c8be980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:32.587+0000 7f527c8be980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:32Z"} 2024-03-04T19:23:51.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:32Z"} 2024-03-04T19:23:51.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:37.735+0000 7f55f3245980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:37.735+0000 7f55f3245980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:37Z"} 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:37Z"} 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:42.875+0000 7fe0e7f5b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:42.875+0000 7fe0e7f5b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:42Z"} 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:42Z"} 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:48.006+0000 7fcf68d7c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:48.006+0000 7fcf68d7c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:48Z"} 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:48Z"} 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:53.158+0000 7f8b63e20980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:53.158+0000 7f8b63e20980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:53Z"} 2024-03-04T19:23:51.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:53Z"} 2024-03-04T19:23:51.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.359 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.359 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:58.290+0000 7fd188ddf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:23:58.290+0000 7fd188ddf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:58Z"} 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:58Z"} 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:03.430+0000 7f39770c3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:03.430+0000 7f39770c3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:03Z"} 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:03Z"} 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:08.570+0000 7faa35589980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:08.570+0000 7faa35589980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:08Z"} 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:08Z"} 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:13.742+0000 7f3d93a64980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:13.746+0000 7f3d93a64980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:13Z"} 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:13Z"} 2024-03-04T19:23:51.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:18.882+0000 7f269a7b2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:18.886+0000 7f269a7b2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:18Z"} 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:18Z"} 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:24.034+0000 7f798835f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:24.038+0000 7f798835f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:24Z"} 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:24Z"} 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:29.178+0000 7fbcf266f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:29.182+0000 7fbcf266f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:29Z"} 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:29Z"} 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:34.342+0000 7f05aebc9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:34.342+0000 7f05aebc9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:34Z"} 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.125374_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.125374Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:34Z"} 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.125374_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.125374Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576062.712249_431.1'} 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.125374_293.1'} 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576062.712249_431.1 target=1_1709575075.125374_293.1 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.366 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:51.367 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable 2024-03-04T19:23:51.367 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.367 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:51.367 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:51.367 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:51.367 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1176, in test_bucket_sync_disable 2024-03-04T19:23:51.367 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:51.367 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:51.367 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=ppfahn-55 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-55/ 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-55/ 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-55/ 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:24:45 GMT 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests:/ppfahn-55/ 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:841diROdo0OV8CZWOUWyS4tBtGM= 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:841diROdo0OV8CZWOUWyS4tBtGM=', 'Content-Length': '0'} 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ad25d504e8f3e13d-0065e611ed-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=ppfahn-56 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-56/ 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-56/ 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-56/ 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:24:45 GMT 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests:/ppfahn-56/ 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:ivtCSb6jxy9e16f8a5SQBgSW9ms= 2024-03-04T19:23:51.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ivtCSb6jxy9e16f8a5SQBgSW9ms=', 'Content-Length': '0'} 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000d8c6ba875ae8b083-0065e611ed-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=ppfahn-57 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-57/ 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-57/ 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-57/ 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.371 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:24:45 GMT 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:/ppfahn-57/ 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:JzQQyDpOHH013bf4V1Jlm5eSOg0= 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:24:45 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JzQQyDpOHH013bf4V1Jlm5eSOg0=', 'Content-Length': '0'} 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000af3c2b93f1b415b7-0065e611ed-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709576685.881266_517.1","last_update":"2024-03-04T18:24:45.881266Z"}] 2024-03-04T19:23:51.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:46.325+0000 7f4a55473980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:46.325+0000 7f4a55473980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:46Z"} 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:46Z"} 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:51.537+0000 7f50cefab980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:51.537+0000 7f50cefab980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:51Z"} 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:51Z"} 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:56.669+0000 7f685c190980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:24:56.669+0000 7f685c190980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:56Z"} 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:56Z"} 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:01.853+0000 7f38fd94e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:01.853+0000 7f38fd94e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:01Z"} 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:01Z"} 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:07.013+0000 7f0a3fff7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:07.017+0000 7f0a3fff7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:07Z"} 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:07Z"} 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:12.181+0000 7fcf962d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:12.181+0000 7fcf962d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:12Z"} 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:12Z"} 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:17.333+0000 7f4b5c370980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:17.333+0000 7f4b5c370980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:17Z"} 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:17Z"} 2024-03-04T19:23:51.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.292440_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.292440Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.292440_304.1'} 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.292440_304.1 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:22.481+0000 7fa7f18c3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:22.481+0000 7fa7f18c3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.828001_352.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.828001Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:22Z"} 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.828001_352.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.828001Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:22Z"} 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575385.828001_352.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:03:05.828001Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575385.828001_352.1'} 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575385.828001_352.1 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:27.625+0000 7f6e7bdce980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:27.629+0000 7f6e7bdce980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:27Z"} 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:27Z"} 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:32.777+0000 7ff2c0e82980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:32.777+0000 7ff2c0e82980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:32Z"} 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:32Z"} 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:37.925+0000 7f84d576a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:37.925+0000 7f84d576a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:37Z"} 2024-03-04T19:23:51.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:37Z"} 2024-03-04T19:23:51.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:43.073+0000 7f94f6593980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:43.077+0000 7f94f6593980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:43Z"} 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:43Z"} 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:48.216+0000 7f86cda90980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:48.216+0000 7f86cda90980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:48Z"} 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:48Z"} 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:53.352+0000 7fb5c10c4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:53.352+0000 7fb5c10c4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:53Z"} 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:53Z"} 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:58.492+0000 7ff6adba3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:25:58.492+0000 7ff6adba3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:58Z"} 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:58Z"} 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:03.616+0000 7f404a4f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:03.616+0000 7f404a4f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:03Z"} 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:03Z"} 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:08.748+0000 7f5b5a08f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:08.752+0000 7f5b5a08f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:08Z"} 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:08Z"} 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:13.900+0000 7faeb9fbb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:13.904+0000 7faeb9fbb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:13Z"} 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:13Z"} 2024-03-04T19:23:51.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:26:13.940+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:26:14.148+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:19.036+0000 7f5a9224c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:19.036+0000 7f5a9224c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:19Z"} 2024-03-04T19:23:51.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:19Z"} 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.422483_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.422483Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.422483_312.1'} 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.422483_312.1 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:24.176+0000 7f68d0680980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:24.176+0000 7f68d0680980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:24Z"} 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:24Z"} 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:29.328+0000 7f3c337e5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:29.328+0000 7f3c337e5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:29Z"} 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:29Z"} 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:34.504+0000 7f2d29a8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:34.504+0000 7f2d29a8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:34Z"} 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:34Z"} 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:39.668+0000 7f5bcb103980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:39.668+0000 7f5bcb103980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:39Z"} 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:39Z"} 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:26:44.115+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:44.800+0000 7f8c69c39980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:44.800+0000 7f8c69c39980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:44Z"} 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:44Z"} 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:49.939+0000 7f3b6b5ab980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:49.939+0000 7f3b6b5ab980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:49Z"} 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:49Z"} 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:55.071+0000 7fef41422980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:26:55.075+0000 7fef41422980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:55Z"} 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:55Z"} 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:00.203+0000 7f4aaecfe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:00.203+0000 7f4aaecfe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:00Z"} 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:00Z"} 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:05.351+0000 7fa3ad5b8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:05.351+0000 7fa3ad5b8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:05Z"} 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:05Z"} 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:10.495+0000 7ff2bd40b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:10.495+0000 7ff2bd40b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:10Z"} 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:10Z"} 2024-03-04T19:23:51.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:15.627+0000 7f5ba9624980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:15.631+0000 7f5ba9624980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:15Z"} 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:15Z"} 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.400 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.400 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:20.775+0000 7fe3bd2f9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:20.775+0000 7fe3bd2f9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:20Z"} 2024-03-04T19:23:51.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:20Z"} 2024-03-04T19:23:51.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.315397_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.315397Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.315397_306.1'} 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.315397_306.1 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:25.915+0000 7f209feea980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:25.919+0000 7f209feea980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:25Z"} 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:25Z"} 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:31.055+0000 7f806043d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:31.055+0000 7f806043d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:31Z"} 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:31Z"} 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:36.195+0000 7f5fff8b8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:36.195+0000 7f5fff8b8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:36Z"} 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:36Z"} 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:41.395+0000 7fd4ee95f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:41.395+0000 7fd4ee95f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:41Z"} 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:41Z"} 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:46.539+0000 7f0dd7115980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:46.543+0000 7f0dd7115980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:46Z"} 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:46Z"} 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:51.686+0000 7f2ac121a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:51.686+0000 7f2ac121a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:51Z"} 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:51Z"} 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:56.822+0000 7f9678996980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:27:56.822+0000 7f9678996980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:56Z"} 2024-03-04T19:23:51.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:56Z"} 2024-03-04T19:23:51.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:01.962+0000 7f2e51b3c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:01.966+0000 7f2e51b3c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.820809_351.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.820809Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:01Z"} 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.820809_351.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.820809Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:01Z"} 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575385.820809_351.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:03:05.820809Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575385.820809_351.1'} 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575385.820809_351.1 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:07.106+0000 7f2180113980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:07.106+0000 7f2180113980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.051072_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.051072Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:07Z"} 2024-03-04T19:23:51.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.051072_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.051072Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:07Z"} 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.051072_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.051072Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.051072_382.1'} 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575694.051072_382.1 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:12.250+0000 7fd21b6cc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:12.250+0000 7fd21b6cc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.051072_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.051072Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:12Z"} 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.051072_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.051072Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:12Z"} 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.051072_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.051072Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.051072_382.1'} 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575694.051072_382.1 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:17.398+0000 7f669c096980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:17.398+0000 7f669c096980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.051072_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.051072Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:17Z"} 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.051072_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.051072Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:17Z"} 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.051072_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.051072Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.051072_382.1'} 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575694.051072_382.1 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:22.558+0000 7fb3ff426980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:22.558+0000 7fb3ff426980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:22Z"} 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:22Z"} 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.412 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:27.710+0000 7f7b8b42d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:27.710+0000 7f7b8b42d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:27Z"} 2024-03-04T19:23:51.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:27Z"} 2024-03-04T19:23:51.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.412 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:32.870+0000 7f31889e4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:32.874+0000 7f31889e4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:32Z"} 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:32Z"} 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:38.046+0000 7f1325625980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:38.046+0000 7f1325625980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:38Z"} 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:38Z"} 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:43.174+0000 7f28d9b4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:43.178+0000 7f28d9b4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:43Z"} 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:43Z"} 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:48.338+0000 7fc3aa735980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:48.342+0000 7fc3aa735980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:48Z"} 2024-03-04T19:23:51.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:48Z"} 2024-03-04T19:23:51.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:53.489+0000 7f1b99f29980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:53.489+0000 7f1b99f29980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:53Z"} 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:53Z"} 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:58.637+0000 7f018a62b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:28:58.637+0000 7f018a62b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:58Z"} 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:58Z"} 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:03.797+0000 7f91410ba980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:03.801+0000 7f91410ba980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:03Z"} 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:03Z"} 2024-03-04T19:23:51.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:08.965+0000 7fcdd359d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:08.965+0000 7fcdd359d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:08Z"} 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:08Z"} 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.422483_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.422483Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.422483_312.1'} 2024-03-04T19:23:51.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.422483_312.1 2024-03-04T19:23:51.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:14.109+0000 7fa5106bf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:14.113+0000 7fa5106bf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:14Z"} 2024-03-04T19:23:51.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:14Z"} 2024-03-04T19:23:51.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.422483_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.422483Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.422483_312.1'} 2024-03-04T19:23:51.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.422483_312.1 2024-03-04T19:23:51.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:19.249+0000 7fdb3b52d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:19.249+0000 7fdb3b52d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:19Z"} 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:19Z"} 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:24.401+0000 7fac98cd1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:24.401+0000 7fac98cd1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:24Z"} 2024-03-04T19:23:51.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:24Z"} 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:29.553+0000 7fc2b5578980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:29.553+0000 7fc2b5578980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:29Z"} 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:29Z"} 2024-03-04T19:23:51.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:34.693+0000 7fc5cbdb4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:34.693+0000 7fc5cbdb4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:34Z"} 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:34Z"} 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:39.845+0000 7fed6d8e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:39.849+0000 7fed6d8e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:39Z"} 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:39Z"} 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:44.985+0000 7f1a20e67980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:44.985+0000 7f1a20e67980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:44Z"} 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:44Z"} 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:50.133+0000 7f8ddc347980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:50.137+0000 7f8ddc347980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:50Z"} 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:50Z"} 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576685.881266_517.1'} 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576685.881266_517.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.427 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.428 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:51.428 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2024-03-04T19:23:51.428 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.428 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:51.428 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:51.428 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:51.428 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1200, in test_bucket_sync_enable_right_after_disable 2024-03-04T19:23:51.428 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=ppfahn-58 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-58/ 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-58/ 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-58/ 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests:/ppfahn-58/ 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:wcaZ6dQexmkmcG19krQyzPo8OfY= 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:wcaZ6dQexmkmcG19krQyzPo8OfY=', 'Content-Length': '0'} 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e09eba8f275cd2df-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=ppfahn-59 2024-03-04T19:23:51.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-59/ 2024-03-04T19:23:51.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-59/ 2024-03-04T19:23:51.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-59/ 2024-03-04T19:23:51.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.431 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.431 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests:/ppfahn-59/ 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:McfwhZdwN7HNRa8Bmfr5dyWCiaU= 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:McfwhZdwN7HNRa8Bmfr5dyWCiaU=', 'Content-Length': '0'} 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000578f8edf459d0b84-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=ppfahn-60 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-60/ 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-60/ 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-60/ 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests:/ppfahn-60/ 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:hGh8SprXApuusEMdFji7IxTp5hQ= 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:hGh8SprXApuusEMdFji7IxTp5hQ=', 'Content-Length': '0'} 2024-03-04T19:23:51.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000752292a78d0bdb4f-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-58/ 2024-03-04T19:23:51.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-58/ 2024-03-04T19:23:51.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-58/ 2024-03-04T19:23:51.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.434 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.434 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests:/ppfahn-58/ 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8= 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8=', 'Content-Length': '0'} 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000579e4df355a7835b-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-58/obj1 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-58/obj1 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-58/obj1 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:/ppfahn-58/obj1 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:CTGuT2/bp4NGJey/YytDSnoZTJw= 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CTGuT2/bp4NGJey/YytDSnoZTJw='} 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a34638bd2157d24c-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-58/ 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-58/ 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-58/ 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.437 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:/ppfahn-58/ 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8= 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8=', 'Content-Length': '0'} 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe43d72cb6bc10f4-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-58/obj2 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-58/obj2 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-58/obj2 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:/ppfahn-58/obj2 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:0rOaXR+qvgqyG04qCJkcH/W8C1Q= 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:0rOaXR+qvgqyG04qCJkcH/W8C1Q='} 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e83ef1b357e3f57-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-58/ 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-58/ 2024-03-04T19:23:51.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-58/ 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests:/ppfahn-58/ 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.440 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8= 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8=', 'Content-Length': '0'} 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000734492fae90aac08-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-58/obj3 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-58/obj3 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-58/obj3 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.441 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:/ppfahn-58/obj3 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:pPyhmg+WAjaG+NdAg+e9c21353I= 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:pPyhmg+WAjaG+NdAg+e9c21353I='} 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004bb01be47f2f51c8-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-58/ 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-58/ 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-58/ 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests:/ppfahn-58/ 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8= 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lvwK0skejPAZJLUal189afxydJ8=', 'Content-Length': '0'} 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d067a75d48a7997-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-58/obj4 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-58/obj4 2024-03-04T19:23:51.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-58/obj4 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:/ppfahn-58/obj4 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.444 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:oVdRLw8AEXXSPNLG4xvnUxU0P2g= 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:oVdRLw8AEXXSPNLG4xvnUxU0P2g='} 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c467f606f8f60cf9-0065e61323-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-59/ 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-59/ 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-59/ 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.445 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests:/ppfahn-59/ 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU= 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU=', 'Content-Length': '0'} 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5c8861c089ad5ed-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-59/obj1 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-59/obj1 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-59/obj1 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:/ppfahn-59/obj1 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:GPwi37L+gmpO0VTuylbeo3yP6C0= 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GPwi37L+gmpO0VTuylbeo3yP6C0='} 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7b2468713a2e5c2-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-59/ 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-59/ 2024-03-04T19:23:51.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-59/ 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests:/ppfahn-59/ 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.448 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU= 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU=', 'Content-Length': '0'} 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002500251ec509ff90-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-59/obj2 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-59/obj2 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-59/obj2 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.449 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:/ppfahn-59/obj2 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:qaRzw6/fQk1cYNlpiq0foT9KrfY= 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:qaRzw6/fQk1cYNlpiq0foT9KrfY='} 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004515fe7b3995ce03-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-59/ 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-59/ 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-59/ 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests:/ppfahn-59/ 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU= 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU=', 'Content-Length': '0'} 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092744c9ad8d3d792-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-59/obj3 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-59/obj3 2024-03-04T19:23:51.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-59/obj3 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:/ppfahn-59/obj3 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.452 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:5Vh6PCkKZRlvPRGVa3sKRIGKi/I= 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5Vh6PCkKZRlvPRGVa3sKRIGKi/I='} 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb865bc975d278e1-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-59/ 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-59/ 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-59/ 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.453 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:/ppfahn-59/ 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU= 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:je8qgfl97J/+SWw77TcnlIcMjVU=', 'Content-Length': '0'} 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000717c007c42199623-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-59/obj4 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-59/obj4 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-59/obj4 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:/ppfahn-59/obj4 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8opWzLzevpYNVafadPjZoIEA/MI= 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8opWzLzevpYNVafadPjZoIEA/MI='} 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e028d41bb076493-0065e61323-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-60/ 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-60/ 2024-03-04T19:23:51.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-60/ 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests:/ppfahn-60/ 2024-03-04T19:23:51.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio= 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio=', 'Content-Length': '0'} 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d723d8bd88490c71-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-60/obj1 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-60/obj1 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-60/obj1 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.457 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:/ppfahn-60/obj1 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:GLhHy8wwY6CA7o1Xokle4yk81uE= 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:GLhHy8wwY6CA7o1Xokle4yk81uE='} 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005daa37c12dbd891b-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-60/ 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-60/ 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-60/ 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests:/ppfahn-60/ 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio= 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio=', 'Content-Length': '0'} 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004be8b08183969e64-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-60/obj2 2024-03-04T19:23:51.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-60/obj2 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-60/obj2 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:/ppfahn-60/obj2 2024-03-04T19:23:51.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:uden4NgtxYaHWZ0ecOZtkMFFO48= 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uden4NgtxYaHWZ0ecOZtkMFFO48='} 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc6663e77abbb991-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-60/ 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-60/ 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-60/ 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:/ppfahn-60/ 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio= 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio=', 'Content-Length': '0'} 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e382c2bb2d75b68e-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-60/obj3 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-60/obj3 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-60/obj3 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:/ppfahn-60/obj3 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:lPuIlXurJf303NWLY5mVhhp1eL4= 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:lPuIlXurJf303NWLY5mVhhp1eL4='} 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001971693ee3a404f5-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-60/ 2024-03-04T19:23:51.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-60/ 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-60/ 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.464 INFO:tasks.rgw_multisite_tests:/ppfahn-60/ 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio= 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:6jE9phHIzxS47j9dHl1RAvIhXio=', 'Content-Length': '0'} 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d25ea4c53dd4c59-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-60/obj4 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-60/obj4 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-60/obj4 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:29:55 GMT 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:/ppfahn-60/obj4 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:g7MzNMdRCENrBO4fTsERu/L71Cc= 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:29:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:g7MzNMdRCENrBO4fTsERu/L71Cc='} 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004295622cb06c7141-0065e61323-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:29:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709576995.430314_531.1","last_update":"2024-03-04T18:29:55.430314Z"}] 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:56.220+0000 7f9bb5077980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:29:56.220+0000 7f9bb5077980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:56Z"} 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:56Z"} 2024-03-04T19:23:51.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:01.376+0000 7fa04225c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:01.380+0000 7fa04225c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:01Z"} 2024-03-04T19:23:51.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:01Z"} 2024-03-04T19:23:51.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:30:04.113+0000 7f5c190f6640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:06.520+0000 7f881b671980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:06.524+0000 7f881b671980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.690390_325.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.690390Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:06Z"} 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.690390_325.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.690390Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:06Z"} 2024-03-04T19:23:51.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.690390_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.690390Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.690390_325.1'} 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575077.690390_325.1 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:11.672+0000 7fc494f6d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:11.672+0000 7fc494f6d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:11Z"} 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:11Z"} 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:16.820+0000 7f64c014a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:16.824+0000 7f64c014a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:16Z"} 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:16Z"} 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:21.968+0000 7f2ffc3a6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:21.972+0000 7f2ffc3a6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:21Z"} 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:21Z"} 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:27.120+0000 7fc6839f4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:27.120+0000 7fc6839f4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:27Z"} 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:27Z"} 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:32.276+0000 7f6d8ff96980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:32.276+0000 7f6d8ff96980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.486968_316.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.486968Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:32Z"} 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.486968_316.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.486968Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:32Z"} 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.486968_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.486968Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.486968_316.1'} 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.486968_316.1 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:37.440+0000 7fd09b4e7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:37.444+0000 7fd09b4e7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:37Z"} 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:37Z"} 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:42.588+0000 7f97e1577980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:42.592+0000 7f97e1577980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:42Z"} 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:42Z"} 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:47.740+0000 7f1e1014f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:47.740+0000 7f1e1014f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:47Z"} 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:47Z"} 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:52.908+0000 7f314b36c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:52.908+0000 7f314b36c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:52Z"} 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:52Z"} 2024-03-04T19:23:51.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:58.056+0000 7f533a7c2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:30:58.056+0000 7f533a7c2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:58Z"} 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:58Z"} 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.523626_319.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.523626Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.523626_319.1'} 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.523626_319.1 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:03.207+0000 7ff3bdb51980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:03.207+0000 7ff3bdb51980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:03Z"} 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:03Z"} 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.480 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.480 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:08.383+0000 7f23e7917980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:08.387+0000 7f23e7917980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:08Z"} 2024-03-04T19:23:51.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:08Z"} 2024-03-04T19:23:51.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:13.523+0000 7fa896f52980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:13.523+0000 7fa896f52980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:13Z"} 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:13Z"} 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:18.659+0000 7fb6b004f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:18.663+0000 7fb6b004f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:18Z"} 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:18Z"} 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:23.799+0000 7f3a9d3b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:23.799+0000 7f3a9d3b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:23Z"} 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:23Z"} 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:31:23.951+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.483 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:28.959+0000 7f7c77c6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:28.959+0000 7f7c77c6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:28Z"} 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:28Z"} 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:34.107+0000 7f8b33ed6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:34.111+0000 7f8b33ed6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:34Z"} 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:34Z"} 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:31:35.375+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:39.259+0000 7f5574fde980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:39.259+0000 7f5574fde980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:39Z"} 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:39Z"} 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.380518_307.1 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:44.411+0000 7faa15d99980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:44.411+0000 7faa15d99980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:44Z"} 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:44Z"} 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:49.555+0000 7ff7aab1b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:49.559+0000 7ff7aab1b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:49Z"} 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:49Z"} 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:54.711+0000 7f18f6b0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:54.711+0000 7f18f6b0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:54Z"} 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:54Z"} 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:59.859+0000 7f4a51244980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:31:59.859+0000 7f4a51244980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:59Z"} 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:59Z"} 2024-03-04T19:23:51.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:05.010+0000 7f5a80faf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:05.010+0000 7f5a80faf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:05Z"} 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:05Z"} 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:10.170+0000 7fc41f4e3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:10.170+0000 7fc41f4e3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:10Z"} 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:10Z"} 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:15.310+0000 7fdc20fbb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:15.310+0000 7fdc20fbb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:15Z"} 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:15Z"} 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.493 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:20.446+0000 7f5519727980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:20.446+0000 7f5519727980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:20Z"} 2024-03-04T19:23:51.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:20Z"} 2024-03-04T19:23:51.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.493 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:32:24.142+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:25.594+0000 7f570f1ac980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:25.598+0000 7f570f1ac980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:25Z"} 2024-03-04T19:23:51.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:25Z"} 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:30.742+0000 7f448c9d5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:30.742+0000 7f448c9d5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:30Z"} 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:30Z"} 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:35.878+0000 7f45f890d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:35.878+0000 7f45f890d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.489493_317.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.489493Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:35Z"} 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.489493_317.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.489493Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:35Z"} 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.489493_317.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.489493Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.489493_317.1'} 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.489493_317.1 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:41.030+0000 7f1dcf18c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:41.030+0000 7f1dcf18c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:41Z"} 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:41Z"} 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:46.166+0000 7fc0195f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:46.166+0000 7fc0195f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:46Z"} 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:46Z"} 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:51.306+0000 7f8f00ab0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:51.306+0000 7f8f00ab0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:51Z"} 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:51Z"} 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:56.454+0000 7fdedefa2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:32:56.454+0000 7fdedefa2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:56Z"} 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:56Z"} 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:01.598+0000 7fac4e753980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:01.602+0000 7fac4e753980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:01Z"} 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:01Z"} 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:06.757+0000 7f7651954980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:06.761+0000 7f7651954980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:06Z"} 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:06Z"} 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:11.925+0000 7f5d75d0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:11.925+0000 7f5d75d0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:11Z"} 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:11Z"} 2024-03-04T19:23:51.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:17.073+0000 7fb3c2d59980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:17.077+0000 7fb3c2d59980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:17Z"} 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:17Z"} 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:22.209+0000 7f61781e9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:22.209+0000 7f61781e9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:22Z"} 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:22Z"} 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:27.357+0000 7fe095ffa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:27.357+0000 7fe095ffa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:27Z"} 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:27Z"} 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.505 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:32.509+0000 7fee7556e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:32.513+0000 7fee7556e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:32Z"} 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:32Z"} 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.380518_307.1 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:37.653+0000 7f9836333980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:37.653+0000 7f9836333980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:37Z"} 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:37Z"} 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:42.785+0000 7fd4b0120980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:42.789+0000 7fd4b0120980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:42Z"} 2024-03-04T19:23:51.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:42Z"} 2024-03-04T19:23:51.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.508 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.508 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:47.941+0000 7f565a499980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:47.941+0000 7f565a499980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:47Z"} 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:47Z"} 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:53.089+0000 7f325dcc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:53.093+0000 7f325dcc7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:53Z"} 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:53Z"} 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:58.241+0000 7f86cf5b6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:33:58.245+0000 7f86cf5b6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:58Z"} 2024-03-04T19:23:51.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:58Z"} 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:03.381+0000 7f30f9b93980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:03.381+0000 7f30f9b93980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:03Z"} 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:03Z"} 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.512 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.512 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:08.517+0000 7f4808985980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:08.521+0000 7f4808985980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:08Z"} 2024-03-04T19:23:51.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:08Z"} 2024-03-04T19:23:51.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:13.660+0000 7f86da25c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:13.660+0000 7f86da25c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:13Z"} 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:13Z"} 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:18.796+0000 7fc0a5403980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:18.796+0000 7fc0a5403980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:18Z"} 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:18Z"} 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:23.940+0000 7f2c6527d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:23.940+0000 7f2c6527d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:23Z"} 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.422483_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.422483Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:23Z"} 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.422483_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.422483Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.422483_312.1'} 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.422483_312.1 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:29.092+0000 7f487825e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:29.092+0000 7f487825e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:29Z"} 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:29Z"} 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:34.240+0000 7fe6e55fb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:34.240+0000 7fe6e55fb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:34Z"} 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:34Z"} 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:39.388+0000 7fda685bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:39.388+0000 7fda685bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:39Z"} 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:39Z"} 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:44.536+0000 7f181fb6b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:44.536+0000 7f181fb6b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:44Z"} 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:44Z"} 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:49.684+0000 7f1d5bff2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:49.688+0000 7f1d5bff2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:49Z"} 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:49Z"} 2024-03-04T19:23:51.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:54.832+0000 7fcd2c44f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:54.832+0000 7fcd2c44f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:54Z"} 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:54Z"} 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:59.968+0000 7f44f2b17980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:34:59.968+0000 7f44f2b17980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:59Z"} 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:59Z"} 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709576995.430314_531.1'} 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709576995.430314_531.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.522 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.522 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.522 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:51.522 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable_enable 2024-03-04T19:23:51.522 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.522 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:51.522 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1235, in test_bucket_sync_disable_enable 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=ppfahn-61 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-61/ 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-61/ 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-61/ 2024-03-04T19:23:51.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests:/ppfahn-61/ 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:tD/RyF0dJuJj1eNNLOsAX0LkwuA= 2024-03-04T19:23:51.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tD/RyF0dJuJj1eNNLOsAX0LkwuA=', 'Content-Length': '0'} 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f60fffd7d08f4af8-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=ppfahn-62 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-62/ 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-62/ 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-62/ 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.525 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests:/ppfahn-62/ 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:T0J3F37cTJywCyWUCbcGWiexdKc= 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:T0J3F37cTJywCyWUCbcGWiexdKc=', 'Content-Length': '0'} 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b57036635e476a8f-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=ppfahn-63 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-63/ 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-63/ 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-63/ 2024-03-04T19:23:51.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests:/ppfahn-63/ 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:2oEw/0OQ+aJdyjuB7vCLnhaKiJQ= 2024-03-04T19:23:51.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2oEw/0OQ+aJdyjuB7vCLnhaKiJQ=', 'Content-Length': '0'} 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e6d9c3ed2a38a86f-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-61/ 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-61/ 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-61/ 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.528 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.529 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.529 INFO:tasks.rgw_multisite_tests:/ppfahn-61/ 2024-03-04T19:23:51.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.529 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg= 2024-03-04T19:23:51.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg=', 'Content-Length': '0'} 2024-03-04T19:23:51.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a70995c43080d9a-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-61/obj1 2024-03-04T19:23:51.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-61/obj1 2024-03-04T19:23:51.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-61/obj1 2024-03-04T19:23:51.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:/ppfahn-61/obj1 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:5B/s7WLinUK4gtsQzce/3IOX96Y= 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:5B/s7WLinUK4gtsQzce/3IOX96Y='} 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9c0f6fa902cf358-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-61/ 2024-03-04T19:23:51.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-61/ 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-61/ 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests:/ppfahn-61/ 2024-03-04T19:23:51.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg= 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg=', 'Content-Length': '0'} 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005498287dd0b58bca-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-61/obj2 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-61/obj2 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-61/obj2 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.532 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:/ppfahn-61/obj2 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:cGewrvDG/wTE1IgAJMmnMVzp8Jc= 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cGewrvDG/wTE1IgAJMmnMVzp8Jc='} 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e607e4328837802-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-61/ 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-61/ 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-61/ 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests:/ppfahn-61/ 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg= 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg=', 'Content-Length': '0'} 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddf5a637dd142859-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-61/obj3 2024-03-04T19:23:51.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-61/obj3 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-61/obj3 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:/ppfahn-61/obj3 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.535 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:+ZY+E/mQhgGqxRdHFnx85Hru0Lk= 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:+ZY+E/mQhgGqxRdHFnx85Hru0Lk='} 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082f7c65ca622fe1e-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-61/ 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-61/ 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-61/ 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.536 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests:/ppfahn-61/ 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg= 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:HcGIAEywhdp7fccGWEDB+03tCLg=', 'Content-Length': '0'} 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000757a3fd48401f1ab-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-61/obj4 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-61/obj4 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-61/obj4 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:/ppfahn-61/obj4 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:F7RNz9yUvfFyKBuTjJ/If3NgTvo= 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:F7RNz9yUvfFyKBuTjJ/If3NgTvo='} 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d03606d6d5216535-0065e61459-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-62/ 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-62/ 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-62/ 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.539 INFO:tasks.rgw_multisite_tests:/ppfahn-62/ 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E= 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E=', 'Content-Length': '0'} 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005504c68284f78706-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-62/obj1 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-62/obj1 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-62/obj1 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:/ppfahn-62/obj1 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:jxl+ksOvIfHxATiwJcuuB0mArG0= 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:jxl+ksOvIfHxATiwJcuuB0mArG0='} 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d848fde8df0b0eb1-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-62/ 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-62/ 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-62/ 2024-03-04T19:23:51.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests:/ppfahn-62/ 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E= 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E=', 'Content-Length': '0'} 2024-03-04T19:23:51.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000853acf679b243e9e-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-62/obj2 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-62/obj2 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-62/obj2 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.543 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:/ppfahn-62/obj2 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:uER3QCLEEtxGDUd+h+xxSNZJxsM= 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:uER3QCLEEtxGDUd+h+xxSNZJxsM='} 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f3e3c71f0ec03843-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-62/ 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-62/ 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-62/ 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests:/ppfahn-62/ 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E= 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E=', 'Content-Length': '0'} 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1f7ddb5c81e02f1-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-62/obj3 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-62/obj3 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-62/obj3 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:/ppfahn-62/obj3 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.546 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:n4PzhrSkDO7n6zF87Hl8pWggfsQ= 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:n4PzhrSkDO7n6zF87Hl8pWggfsQ='} 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cabb989e1956581c-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-62/ 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-62/ 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-62/ 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.547 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.548 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.548 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.548 INFO:tasks.rgw_multisite_tests:/ppfahn-62/ 2024-03-04T19:23:51.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.548 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E= 2024-03-04T19:23:51.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MN75fPJ6X39C5XhAI0jiaClOR8E=', 'Content-Length': '0'} 2024-03-04T19:23:51.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004a1c8e9a093d665-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-62/obj4 2024-03-04T19:23:51.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-62/obj4 2024-03-04T19:23:51.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-62/obj4 2024-03-04T19:23:51.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:/ppfahn-62/obj4 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:dKx4IUsHXpNmx9XaNicHKSCST8Y= 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:dKx4IUsHXpNmx9XaNicHKSCST8Y='} 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003db527dd09c4a11f-0065e61459-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-63/ 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-63/ 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-63/ 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.550 INFO:tasks.rgw_multisite_tests:/ppfahn-63/ 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I= 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I=', 'Content-Length': '0'} 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057e535deb2d41037-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-63/obj1 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-63/obj1 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-63/obj1 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.552 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.552 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.552 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.552 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.552 INFO:tasks.rgw_multisite_tests:/ppfahn-63/obj1 2024-03-04T19:23:51.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.552 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:tmesgIAPlTt1mjVvhh1B2yK/HBU= 2024-03-04T19:23:51.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:tmesgIAPlTt1mjVvhh1B2yK/HBU='} 2024-03-04T19:23:51.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f5cbc958550adc6-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-63/ 2024-03-04T19:23:51.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-63/ 2024-03-04T19:23:51.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-63/ 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests:/ppfahn-63/ 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.553 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I= 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I=', 'Content-Length': '0'} 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7222a2aab779e97-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-63/obj2 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-63/obj2 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-63/obj2 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.554 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:/ppfahn-63/obj2 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:7oUzQ4UXMTaBlQD7f6uhyBs5eC0= 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:7oUzQ4UXMTaBlQD7f6uhyBs5eC0='} 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6a426bbcb482ee0-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-63/ 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-63/ 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-63/ 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests:/ppfahn-63/ 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I= 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I=', 'Content-Length': '0'} 2024-03-04T19:23:51.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000096cb4bc59e8bda8-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-63/obj3 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-63/obj3 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-63/obj3 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.557 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.558 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.558 INFO:tasks.rgw_multisite_tests:/ppfahn-63/obj3 2024-03-04T19:23:51.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.558 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:3PCJjuh3VNHJLTKXdRue6aWzVaw= 2024-03-04T19:23:51.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:3PCJjuh3VNHJLTKXdRue6aWzVaw='} 2024-03-04T19:23:51.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000ea510c2b34d40e-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-63/ 2024-03-04T19:23:51.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-63/ 2024-03-04T19:23:51.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-63/ 2024-03-04T19:23:51.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests:/ppfahn-63/ 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I= 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:srtUWHzPGqOifqEjm198ShTZM1I=', 'Content-Length': '0'} 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b513b4c3ed355395-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-63/obj4 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-63/obj4 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-63/obj4 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.560 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:35:05 GMT 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:/ppfahn-63/obj4 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:q0hzhMo+JlZYrTnQuKwpyzeHQIs= 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:35:05 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:q0hzhMo+JlZYrTnQuKwpyzeHQIs='} 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ab0e1111a4f1bb7-0065e61459-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:35:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709577305.271001_560.1","last_update":"2024-03-04T18:35:05.271001Z"}] 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:06.096+0000 7f187e28f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:06.096+0000 7f187e28f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:06Z"} 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:06Z"} 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:35:06.586+0000 7f5c2310a640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T19:23:51.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:11.228+0000 7f19dfd49980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:11.232+0000 7f19dfd49980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:11Z"} 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:11Z"} 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:35:14.126+0000 7f5c190f6640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.563 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:35:14.126+0000 7f5c2390b640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T19:23:51.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:16.383+0000 7f7c1bca2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:16.383+0000 7f7c1bca2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:16Z"} 2024-03-04T19:23:51.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:16Z"} 2024-03-04T19:23:51.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:21.531+0000 7f77e6944980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:21.531+0000 7f77e6944980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:21Z"} 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:21Z"} 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.565 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:26.683+0000 7f4544104980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:26.683+0000 7f4544104980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:26Z"} 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:26Z"} 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.315397_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.315397Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.315397_306.1'} 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.315397_306.1 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:31.823+0000 7fa0d9648980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:31.827+0000 7fa0d9648980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:31Z"} 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:31Z"} 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:36.967+0000 7fda06434980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:36.967+0000 7fda06434980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.815737_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.815737Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:36Z"} 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.815737_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.815737Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:36Z"} 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.815737_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.815737Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.815737_334.1'} 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575077.815737_334.1 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:42.111+0000 7f6654249980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:42.111+0000 7f6654249980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:42Z"} 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:42Z"} 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:47.259+0000 7f8a561dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:47.259+0000 7f8a561dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:47Z"} 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:47Z"} 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:52.415+0000 7f516f530980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:52.419+0000 7f516f530980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:52Z"} 2024-03-04T19:23:51.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:52Z"} 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:57.555+0000 7f53d78ad980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:35:57.555+0000 7f53d78ad980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:57Z"} 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:57Z"} 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:02.695+0000 7fd318a54980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:02.695+0000 7fd318a54980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:02Z"} 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:02Z"} 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:07.831+0000 7f376fe99980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:07.831+0000 7f376fe99980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:07Z"} 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:07Z"} 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:12.967+0000 7f83aaabd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:12.967+0000 7f83aaabd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:12Z"} 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:12Z"} 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.523626_319.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.523626Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.523626_319.1'} 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.523626_319.1 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.574 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:18.111+0000 7f0c9674b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:18.111+0000 7f0c9674b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:18Z"} 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:18Z"} 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:23.254+0000 7fa4e1455980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:23.254+0000 7fa4e1455980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.489493_317.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.489493Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:23Z"} 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.489493_317.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.489493Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:23Z"} 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.489493_317.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.489493Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.489493_317.1'} 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.489493_317.1 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:36:26.312+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:28.390+0000 7ff47019b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:28.390+0000 7ff47019b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:28Z"} 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:28Z"} 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:33.542+0000 7f79d6c24980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:33.542+0000 7f79d6c24980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:33Z"} 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:33Z"} 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:38.698+0000 7f62fb4ae980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:38.698+0000 7f62fb4ae980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:38Z"} 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:38Z"} 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:43.850+0000 7f68b4574980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:43.850+0000 7f68b4574980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:43Z"} 2024-03-04T19:23:51.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:43Z"} 2024-03-04T19:23:51.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.580 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.580 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:48.994+0000 7fdd546da980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:48.998+0000 7fdd546da980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.486968_316.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.486968Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:49Z"} 2024-03-04T19:23:51.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.486968_316.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.486968Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:49Z"} 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.486968_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.486968Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.486968_316.1'} 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.486968_316.1 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:54.138+0000 7fc8493fd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:54.138+0000 7fc8493fd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:54Z"} 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:54Z"} 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:59.282+0000 7f469afbf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:36:59.282+0000 7f469afbf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:59Z"} 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:59Z"} 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:04.434+0000 7fb54f8d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:04.438+0000 7fb54f8d8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:04Z"} 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:04Z"} 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:09.590+0000 7f1d639c1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:09.590+0000 7f1d639c1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:09Z"} 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:09Z"} 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:14.742+0000 7efd90a35980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:14.742+0000 7efd90a35980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:14Z"} 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:14Z"} 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:19.893+0000 7f439007d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:19.893+0000 7f439007d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:19Z"} 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:19Z"} 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:37:24.163+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:25.033+0000 7fe73934e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:25.037+0000 7fe73934e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:25Z"} 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:25Z"} 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:30.169+0000 7f7f8cfcb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:30.169+0000 7f7f8cfcb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:30Z"} 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:30Z"} 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:35.305+0000 7f9722560980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:35.305+0000 7f9722560980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:35Z"} 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:35Z"} 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:40.441+0000 7f44a8266980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:40.441+0000 7f44a8266980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:40Z"} 2024-03-04T19:23:51.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:40Z"} 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:45.589+0000 7f098bbf2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:45.593+0000 7f098bbf2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:45Z"} 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:45Z"} 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:50.749+0000 7ffad88e0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:50.749+0000 7ffad88e0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:50Z"} 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:50Z"} 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:55.905+0000 7f9383542980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:37:55.909+0000 7f9383542980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:55Z"} 2024-03-04T19:23:51.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:55Z"} 2024-03-04T19:23:51.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:01.053+0000 7fbb3ea18980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:01.057+0000 7fbb3ea18980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:01Z"} 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:01Z"} 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:06.201+0000 7f12d57d5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:06.205+0000 7f12d57d5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:06Z"} 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:06Z"} 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:11.369+0000 7f1d9b30c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:11.373+0000 7f1d9b30c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:11Z"} 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:11Z"} 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:16.524+0000 7f2a58587980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:16.524+0000 7f2a58587980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:16Z"} 2024-03-04T19:23:51.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:16Z"} 2024-03-04T19:23:51.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.596 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.596 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:21.680+0000 7f003839c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:21.680+0000 7f003839c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:21Z"} 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:21Z"} 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:26.848+0000 7ff311305980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:26.852+0000 7ff311305980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:26Z"} 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:26Z"} 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:32.056+0000 7f1b49276980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:32.060+0000 7f1b49276980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:32Z"} 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:32Z"} 2024-03-04T19:23:51.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:37.200+0000 7f9c8b1dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:37.204+0000 7f9c8b1dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:37Z"} 2024-03-04T19:23:51.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:37Z"} 2024-03-04T19:23:51.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:42.356+0000 7f4e4e0b2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:42.356+0000 7f4e4e0b2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:42Z"} 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:42Z"} 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575076.582500_321.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:56.582500Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575076.582500_321.1'} 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575076.582500_321.1 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:47.508+0000 7fb3c4ad9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:47.508+0000 7fb3c4ad9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:47Z"} 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:47Z"} 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:52.640+0000 7f26466c7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:52.640+0000 7f26466c7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:52Z"} 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:52Z"} 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.388355_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.388355Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.388355_309.1'} 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.388355_309.1 2024-03-04T19:23:51.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:57.788+0000 7f9f496e7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:38:57.788+0000 7f9f496e7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:57Z"} 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:57Z"} 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:02.932+0000 7fda8db68980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:02.936+0000 7fda8db68980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:02Z"} 2024-03-04T19:23:51.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:02Z"} 2024-03-04T19:23:51.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.604 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.604 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:08.068+0000 7fe5f5b57980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:08.068+0000 7fe5f5b57980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:08Z"} 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:08Z"} 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:13.199+0000 7f55ad198980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:13.199+0000 7f55ad198980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:13Z"} 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:13Z"} 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:18.391+0000 7fb89c867980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:18.391+0000 7fb89c867980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:18Z"} 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:18Z"} 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:23.527+0000 7f7095bf8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:23.527+0000 7f7095bf8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:23Z"} 2024-03-04T19:23:51.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:23Z"} 2024-03-04T19:23:51.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:28.663+0000 7f0b7d0de980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:28.663+0000 7f0b7d0de980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:28Z"} 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:28Z"} 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:33.803+0000 7f7851513980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:33.803+0000 7f7851513980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:33Z"} 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:33Z"} 2024-03-04T19:23:51.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:38.979+0000 7f3a1f1a3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:38.983+0000 7f3a1f1a3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:38Z"} 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:38Z"} 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:44.127+0000 7f8c5df09980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:44.131+0000 7f8c5df09980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:44Z"} 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:44Z"} 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:49.279+0000 7fb19ac68980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:49.279+0000 7fb19ac68980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.393343_310.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.393343Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:49Z"} 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.393343_310.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.393343Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:49Z"} 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.393343_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.393343Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.393343_310.1'} 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.393343_310.1 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.612 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:54.447+0000 7f36bc21e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:54.447+0000 7f36bc21e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:54Z"} 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:54Z"} 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:59.591+0000 7fb3aa45d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:39:59.595+0000 7fb3aa45d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:59Z"} 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:59Z"} 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:04.746+0000 7fcfe65dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:04.746+0000 7fcfe65dc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:04Z"} 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:04Z"} 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:09.902+0000 7f831af5f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:09.902+0000 7f831af5f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:09Z"} 2024-03-04T19:23:51.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:09Z"} 2024-03-04T19:23:51.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577305.271001_560.1'} 2024-03-04T19:23:51.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577305.271001_560.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.616 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.616 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.616 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:51.616 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multipart_object_sync 2024-03-04T19:23:51.616 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.617 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:51.617 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:51.617 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:51.617 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1279, in test_multipart_object_sync 2024-03-04T19:23:51.617 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:51.617 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:51.617 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:51.617 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:23:51.617 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:23:51.617 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.617 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=ppfahn-64 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/ 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/ 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-64/ 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.618 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:40:14 GMT 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:/ppfahn-64/ 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:Ms8x8N00fAzKrfDrajNRldFAfSQ= 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:40:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:Ms8x8N00fAzKrfDrajNRldFAfSQ=', 'Content-Length': '0'} 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000004b3193439ab16fbc-0065e6158e-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:40:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=ppfahn-65 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-65/ 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-65/ 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-65/ 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi153.front.sepia.ceph.com:8000 2024-03-04T19:23:51.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:40:14 GMT 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests:/ppfahn-65/ 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:ldi8I+ILnrtT98YyNek8Rhcmeyw= 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:40:14 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ldi8I+ILnrtT98YyNek8Rhcmeyw=', 'Content-Length': '0'} 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f660999ea68c93d1-0065e6158e-7099-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:40:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=ppfahn-66 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-66/ 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-66/ 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-66/ 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8001 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.621 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:40:15 GMT 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:/ppfahn-66/ 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:eWLsa8IjL79wy0xwd8B26FT6NJ8= 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:40:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eWLsa8IjL79wy0xwd8B26FT6NJ8=', 'Content-Length': '0'} 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f79a72231b10b73a-0065e6158f-5379-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:40:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART?uploads 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART?uploads 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-64/MULTIPART?uploads 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:40:15 GMT 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests:/ppfahn-64/MULTIPART?uploads 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:zmjPkLaBWjRzg9Um/qfMgk3TXig= 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:40:15 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:zmjPkLaBWjRzg9Um/qfMgk3TXig=', 'Content-Length': '0'} 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000040a980df6df2a661-0065e6158f-5376-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '249'), ('Date', 'Mon, 04 Mar 2024 18:40:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'ppfahn-64MULTIPART2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb' 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=1 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=1 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=1 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:ebKBBg0ze5srhMzzkK3PdA== 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.624 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:40:17 GMT 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:/ppfahn-64/MULTIPART?partNumber=1&uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:2MDtzWDPf9tYwmpH14vxOh/vYUI= 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:40:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:2MDtzWDPf9tYwmpH14vxOh/vYUI='} 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0d22a7e28828b93-0065e61591-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:40:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=2 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=2 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=2 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:dIQ6OrGTo4m87YmUAtmdXw== 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:40:17 GMT 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:/ppfahn-64/MULTIPART?partNumber=2&uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:TxG+MiKSesPFiucNo2NKPH0la0A= 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:40:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:TxG+MiKSesPFiucNo2NKPH0la0A='} 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077353b7b32d526b8-0065e61591-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:40:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=3 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=3 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=3 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:e4RW4edMN49Fhh9TYZ51tg== 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:40:17 GMT 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:/ppfahn-64/MULTIPART?partNumber=3&uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T19:23:51.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:XBXsXck1g9Aba5AX2524jmGuwZk= 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:40:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:XBXsXck1g9Aba5AX2524jmGuwZk='} 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f771588707147bf8-0065e61591-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:40:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=4 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=4 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb&partNumber=4 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:l+jDi4yPcIlVvEiQwDQZTQ== 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:40:17 GMT 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:/ppfahn-64/MULTIPART?partNumber=4&uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:gAskm4AqdIlcMf5TtHbUgsmLfQY= 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:40:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gAskm4AqdIlcMf5TtHbUgsmLfQY='} 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5b13278920ef77a-0065e61591-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:40:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T19:23:51.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests:GET 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:40:17 GMT 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests:/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.630 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:cJCjF6S/tZOQMehP7njNux535uM= 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:40:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:cJCjF6S/tZOQMehP7njNux535uM=', 'Content-Length': '0'} 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070e8e105ce78d9cb-0065e61591-5376-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:40:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests: 1 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests: "79b281060d337b9b2b84ccf390adcf74" 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.631 INFO:tasks.rgw_multisite_tests: 2 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests: "74843a3ab193a389bced899402d99d5f" 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests: 3 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests: "7b8456e1e74c378f45861f53619e75b6" 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests: 4 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests: "97e8c38b8c8f708955bc4890c034194d" 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'Content-Type': 'text/xml'} 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:text/xml 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:40:17 GMT 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:/ppfahn-64/MULTIPART?uploadId=2~iqfxDcMxMknW0DOWI6-iyjZkJ-AIrKb 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:VbHiKdhZZGRkE1SjfyGv8wzBUbg= 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:40:17 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:VbHiKdhZZGRkE1SjfyGv8wzBUbg=', 'Content-Length': '460'} 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000008c920bbd5e3bb902-0065e61591-5376-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '325'), ('Date', 'Mon, 04 Mar 2024 18:40:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: http://smithi136.front.sepia.ceph.com:8000/ppfahn-64/MULTIPARTppfahn-64MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709577615.212032_585.1","last_update":"2024-03-04T18:40:15.212032Z"}] 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:18.318+0000 7f91f8f98980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:18.322+0000 7f91f8f98980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:18Z"} 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:18Z"} 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:23.474+0000 7fc1e1423980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:23.474+0000 7fc1e1423980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:23Z"} 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:23Z"} 2024-03-04T19:23:51.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:28.622+0000 7f2e82167980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:28.622+0000 7f2e82167980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.830517_353.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.830517Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:28Z"} 2024-03-04T19:23:51.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575385.830517_353.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:03:05.830517Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:28Z"} 2024-03-04T19:23:51.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575385.830517_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:03:05.830517Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575385.830517_353.1'} 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575385.830517_353.1 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:33.762+0000 7fea734da980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:33.762+0000 7fea734da980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:33Z"} 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:33Z"} 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:40:34.258+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:40:34.258+0000 7f5c2d91f640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:38.898+0000 7fd1e85a1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:38.902+0000 7fd1e85a1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:38Z"} 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:38Z"} 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:44.054+0000 7fd35bb39980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:44.054+0000 7fd35bb39980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:44Z"} 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:44Z"} 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:49.210+0000 7f7228689980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:49.214+0000 7f7228689980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:49Z"} 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:49Z"} 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:54.370+0000 7f3f51338980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:54.370+0000 7f3f51338980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:54Z"} 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.189999_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.189999Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:54Z"} 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.189999_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.189999Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.189999_297.1'} 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.189999_297.1 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:59.521+0000 7f533b517980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:40:59.521+0000 7f533b517980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:59Z"} 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:59Z"} 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.380518_307.1 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:04.681+0000 7f75b3bfe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:04.681+0000 7f75b3bfe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:04Z"} 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:04Z"} 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:09.837+0000 7f3b311e8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:09.841+0000 7f3b311e8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:09Z"} 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:09Z"} 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:14.985+0000 7fd128359980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:14.985+0000 7fd128359980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:14Z"} 2024-03-04T19:23:51.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:14Z"} 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:20.137+0000 7f11686b8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:20.137+0000 7f11686b8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:20Z"} 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:20Z"} 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:25.289+0000 7f7b48c0a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:25.289+0000 7f7b48c0a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:25Z"} 2024-03-04T19:23:51.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:25Z"} 2024-03-04T19:23:51.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:41:26.349+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:30.441+0000 7f0311c0e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:30.441+0000 7f0311c0e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:30Z"} 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:30Z"} 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:35.577+0000 7f10aa5da980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:35.581+0000 7f10aa5da980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:35Z"} 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:35Z"} 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:40.729+0000 7f2456e52980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:40.729+0000 7f2456e52980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:40Z"} 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:40Z"} 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.315397_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.315397Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.315397_306.1'} 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.315397_306.1 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:45.869+0000 7f48c9fda980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:45.873+0000 7f48c9fda980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:45Z"} 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:45Z"} 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:51.005+0000 7f25923ef980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:51.009+0000 7f25923ef980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:51Z"} 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:51Z"} 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:56.136+0000 7f48203eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:41:56.140+0000 7f48203eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:56Z"} 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:56Z"} 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:01.284+0000 7f13a2f84980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:01.284+0000 7f13a2f84980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:01Z"} 2024-03-04T19:23:51.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:01Z"} 2024-03-04T19:23:51.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.653 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.653 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:06.436+0000 7f23dc261980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:06.436+0000 7f23dc261980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.735016_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.735016Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:06Z"} 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.735016_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.735016Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:06Z"} 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.735016_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.735016Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.735016_329.1'} 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575077.735016_329.1 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:11.580+0000 7fcd83604980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:11.580+0000 7fcd83604980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:11Z"} 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:11Z"} 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:16.736+0000 7f1dca4e8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:16.744+0000 7f1dca4e8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:16Z"} 2024-03-04T19:23:51.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:16Z"} 2024-03-04T19:23:51.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:21.900+0000 7fc049239980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:21.900+0000 7fc049239980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:21Z"} 2024-03-04T19:23:51.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:21Z"} 2024-03-04T19:23:51.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.658 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:27.052+0000 7fce09840980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:27.052+0000 7fce09840980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:27Z"} 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:27Z"} 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:32.184+0000 7f8bea794980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:32.184+0000 7f8bea794980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:32Z"} 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:32Z"} 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:37.324+0000 7f30f277b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:37.324+0000 7f30f277b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:37Z"} 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:37Z"} 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:42.464+0000 7fcf0b38a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:42.468+0000 7fcf0b38a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:42Z"} 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:42Z"} 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:47.607+0000 7f05776bb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:47.611+0000 7f05776bb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:47Z"} 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:47Z"} 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:42:48.988+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:52.779+0000 7f578e37e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:52.779+0000 7f578e37e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:52Z"} 2024-03-04T19:23:51.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:52Z"} 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.292440_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.292440Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.292440_304.1'} 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.292440_304.1 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:57.919+0000 7f274a045980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:42:57.923+0000 7f274a045980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:57Z"} 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:57Z"} 2024-03-04T19:23:51.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:03.063+0000 7f67402e9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:03.063+0000 7f67402e9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:03Z"} 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:03Z"} 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:08.199+0000 7f543519f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:08.199+0000 7f543519f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:08Z"} 2024-03-04T19:23:51.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:08Z"} 2024-03-04T19:23:51.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:13.363+0000 7f4602649980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:13.363+0000 7f4602649980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:13Z"} 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:13Z"} 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:18.519+0000 7fdb46bb8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:18.519+0000 7fdb46bb8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:18Z"} 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:18Z"} 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:23.679+0000 7f5f35fea980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:23.679+0000 7f5f35fea980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:23Z"} 2024-03-04T19:23:51.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:23Z"} 2024-03-04T19:23:51.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:28.859+0000 7f661d4c8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:28.863+0000 7f661d4c8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:28Z"} 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:28Z"} 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:34.003+0000 7f04cd550980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:34.003+0000 7f04cd550980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.489493_317.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.489493Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:34Z"} 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.489493_317.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.489493Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:34Z"} 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.489493_317.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.489493Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.489493_317.1'} 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.489493_317.1 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:39.154+0000 7f7884a5c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:39.158+0000 7f7884a5c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:39Z"} 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:39Z"} 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:44.314+0000 7f197b5c4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:44.318+0000 7f197b5c4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:44Z"} 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:44Z"} 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:49.490+0000 7fd00287a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:49.490+0000 7fd00287a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:49Z"} 2024-03-04T19:23:51.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:49Z"} 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:54.638+0000 7fb914f85980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:54.642+0000 7fb914f85980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:54Z"} 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:54Z"} 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:59.786+0000 7f19c68bf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:43:59.786+0000 7f19c68bf980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:59Z"} 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:59Z"} 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:04.950+0000 7f55ab4fa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:04.950+0000 7f55ab4fa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:04Z"} 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:04Z"} 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:10.086+0000 7f03e2f5f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:10.086+0000 7f03e2f5f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:10Z"} 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:10Z"} 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:15.226+0000 7f068cfd0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:15.226+0000 7f068cfd0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:15Z"} 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:15Z"} 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:20.382+0000 7fb96c3e3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:20.382+0000 7fb96c3e3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:20Z"} 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:20Z"} 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:25.530+0000 7f78bbe31980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:25.530+0000 7f78bbe31980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:25Z"} 2024-03-04T19:23:51.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:25Z"} 2024-03-04T19:23:51.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.292440_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.292440Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.292440_304.1'} 2024-03-04T19:23:51.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.292440_304.1 2024-03-04T19:23:51.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:30.678+0000 7f6d6af33980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:30.678+0000 7f6d6af33980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:30Z"} 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:30Z"} 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:35.821+0000 7f39573dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:35.821+0000 7f39573dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:35Z"} 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:35Z"} 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:40.985+0000 7f9f3adfc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:40.985+0000 7f9f3adfc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:40Z"} 2024-03-04T19:23:51.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:40Z"} 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:46.125+0000 7fd991bdc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:46.125+0000 7fd991bdc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:46Z"} 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:46Z"} 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:51.257+0000 7f2a5475d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:51.257+0000 7f2a5475d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:51Z"} 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:51Z"} 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:56.425+0000 7f9434e28980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:44:56.425+0000 7f9434e28980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:56Z"} 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:56Z"} 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.685 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:01.581+0000 7f547a394980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:01.585+0000 7f547a394980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:01Z"} 2024-03-04T19:23:51.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:01Z"} 2024-03-04T19:23:51.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:06.721+0000 7fe5369b3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:06.721+0000 7fe5369b3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:06Z"} 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.523626_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.523626Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:06Z"} 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.523626_319.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.523626Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.523626_319.1'} 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.523626_319.1 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:11.881+0000 7f0dd0411980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:11.881+0000 7f0dd0411980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:11Z"} 2024-03-04T19:23:51.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:11Z"} 2024-03-04T19:23:51.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.688 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.688 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:17.025+0000 7f8e4b86b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:17.025+0000 7f8e4b86b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:17Z"} 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:17Z"} 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:22.185+0000 7fc0d9e04980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:22.185+0000 7fc0d9e04980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:22Z"} 2024-03-04T19:23:51.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:22Z"} 2024-03-04T19:23:51.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577615.212032_585.1'} 2024-03-04T19:23:51.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577615.212032_585.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.690 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.690 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.690 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:51.690 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_encrypted_object_sync 2024-03-04T19:23:51.691 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.691 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:51.691 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:51.691 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:51.691 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1316, in test_encrypted_object_sync 2024-03-04T19:23:51.691 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:51.691 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:51.691 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=ppfahn-67 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-67/ 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-67/ 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-67/ 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:45:27 GMT 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests:/ppfahn-67/ 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:bGT6qDvkxaZV22cXL+tO+WF6TMQ= 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:45:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:bGT6qDvkxaZV22cXL+tO+WF6TMQ=', 'Content-Length': '0'} 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000730c0352672deb99-0065e616c7-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-67/testobj-sse-c 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-67/testobj-sse-c 2024-03-04T19:23:51.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-67/testobj-sse-c 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:45:27 GMT 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-algorithm:AES256 2024-03-04T19:23:51.694 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:/ppfahn-67/testobj-sse-c 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:JHgcBG/xAsffu0O8L15wlk3g+XU= 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:45:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:JHgcBG/xAsffu0O8L15wlk3g+XU='} 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000c36442f349410c26-0065e616c7-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-67/testobj-sse-kms 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-67/testobj-sse-kms 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-67/testobj-sse-kms 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:45:27 GMT 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption:aws:kms 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-aws-kms-key-id:testkey-1 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:/ppfahn-67/testobj-sse-kms 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:NkVUiAyZrR6sGEKRd5C9Z4M/47U= 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:45:27 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:NkVUiAyZrR6sGEKRd5C9Z4M/47U='} 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6cHBmYWhuLTY3L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000d24b4df8d2ae5d2c-0065e616c7-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:45:27 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.696 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709577927.265995_602.1","last_update":"2024-03-04T18:45:27.265995Z"}] 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:27.673+0000 7f1c23bb2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:27.677+0000 7f1c23bb2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:27Z"} 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:27Z"} 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:32.840+0000 7fbf81cb3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:32.840+0000 7fbf81cb3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:32Z"} 2024-03-04T19:23:51.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:32Z"} 2024-03-04T19:23:51.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.699 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:45:35.728+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.699 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:45:35.728+0000 7f5c1f903640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T19:23:51.699 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:45:35.732+0000 7f5c21907640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T19:23:51.699 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.699 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:37.992+0000 7f7579c2c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:37.992+0000 7f7579c2c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:37Z"} 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:37Z"} 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:43.132+0000 7fdae5ce1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:43.132+0000 7fdae5ce1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:43Z"} 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:43Z"} 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:48.280+0000 7f0381018980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:48.280+0000 7f0381018980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:48Z"} 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:48Z"} 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:53.436+0000 7f5754291980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:53.440+0000 7f5754291980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:53Z"} 2024-03-04T19:23:51.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:53Z"} 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:58.588+0000 7f2d14ef2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:45:58.588+0000 7f2d14ef2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:58Z"} 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:58Z"} 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:03.736+0000 7f12b6b3f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:03.736+0000 7f12b6b3f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:03Z"} 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:03Z"} 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:08.880+0000 7f8663f11980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:08.880+0000 7f8663f11980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:08Z"} 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:08Z"} 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:14.064+0000 7f537ebe7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:14.064+0000 7f537ebe7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:14Z"} 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:14Z"} 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.706 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:19.212+0000 7f87a34b2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:19.212+0000 7f87a34b2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:19Z"} 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:19Z"} 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:24.364+0000 7f5c9e39e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:24.364+0000 7f5c9e39e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:24Z"} 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:24Z"} 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:46:26.403+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:29.507+0000 7f616f76a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:29.511+0000 7f616f76a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:29Z"} 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:29Z"} 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:34.663+0000 7fd551a15980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:34.667+0000 7fd551a15980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:34Z"} 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:34Z"} 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:39.807+0000 7fefad610980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:39.811+0000 7fefad610980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.735016_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.735016Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:39Z"} 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.735016_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.735016Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:39Z"} 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.735016_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.735016Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.735016_329.1'} 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575077.735016_329.1 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:44.967+0000 7fef304ac980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:44.967+0000 7fef304ac980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:44Z"} 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:44Z"} 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:50.107+0000 7f4b3b6c9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:50.107+0000 7f4b3b6c9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:50Z"} 2024-03-04T19:23:51.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:50Z"} 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:55.247+0000 7f5eb8a0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:46:55.247+0000 7f5eb8a0d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:55Z"} 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:55Z"} 2024-03-04T19:23:51.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:00.387+0000 7ff8666cb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:00.387+0000 7ff8666cb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:00Z"} 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:00Z"} 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:05.523+0000 7ff394ba6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:05.527+0000 7ff394ba6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:05Z"} 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:05Z"} 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:10.703+0000 7f82f0e63980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:10.703+0000 7f82f0e63980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:10Z"} 2024-03-04T19:23:51.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:10Z"} 2024-03-04T19:23:51.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:15.843+0000 7f855a856980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:15.847+0000 7f855a856980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:15Z"} 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:15Z"} 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:21.019+0000 7f2773d33980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:21.019+0000 7f2773d33980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.393343_310.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.393343Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:21Z"} 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.393343_310.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.393343Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:21Z"} 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.393343_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.393343Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.393343_310.1'} 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.393343_310.1 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:26.174+0000 7f44a1979980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:26.174+0000 7f44a1979980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:26Z"} 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:26Z"} 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:31.426+0000 7fb339f8d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:31.426+0000 7fb339f8d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:31Z"} 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:31Z"} 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:36.582+0000 7fdf49993980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:36.582+0000 7fdf49993980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:36Z"} 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:36Z"} 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:41.718+0000 7ff10bf4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:41.718+0000 7ff10bf4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:41Z"} 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:41Z"} 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:46.862+0000 7fe8fb535980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:46.862+0000 7fe8fb535980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:46Z"} 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:46Z"} 2024-03-04T19:23:51.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:52.042+0000 7efd0d606980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:52.046+0000 7efd0d606980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:52Z"} 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:52Z"} 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:57.190+0000 7ff4ee3b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:47:57.190+0000 7ff4ee3b9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:57Z"} 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:57Z"} 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:02.342+0000 7f78f4fce980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:02.342+0000 7f78f4fce980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:02Z"} 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:02Z"} 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:07.502+0000 7f6fda780980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:07.502+0000 7f6fda780980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:07Z"} 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:07Z"} 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:48:09.120+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.726 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:12.662+0000 7faee5f88980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:12.662+0000 7faee5f88980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:12Z"} 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:12Z"} 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:17.814+0000 7ffaa374b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:17.814+0000 7ffaa374b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:17Z"} 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:17Z"} 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:22.957+0000 7fb671420980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:22.957+0000 7fb671420980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:22Z"} 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:22Z"} 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:28.097+0000 7f9d5e281980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:28.101+0000 7f9d5e281980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:28Z"} 2024-03-04T19:23:51.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:28Z"} 2024-03-04T19:23:51.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.730 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.730 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:33.233+0000 7fb992f4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:33.233+0000 7fb992f4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:33Z"} 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:33Z"} 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:38.377+0000 7fea846e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:38.377+0000 7fea846e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:38Z"} 2024-03-04T19:23:51.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:38Z"} 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:43.509+0000 7fc20a1d7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:43.509+0000 7fc20a1d7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:43Z"} 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:43Z"} 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:48.649+0000 7f3017362980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:48.649+0000 7f3017362980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:48Z"} 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:48Z"} 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:53.789+0000 7f0f01ed9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:53.789+0000 7f0f01ed9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:53Z"} 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:53Z"} 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.734 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:58.945+0000 7f72e604f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:48:58.945+0000 7f72e604f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:58Z"} 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:58Z"} 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:04.077+0000 7f3e83033980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:04.077+0000 7f3e83033980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:04Z"} 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:04Z"} 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:09.221+0000 7fe6b5046980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:09.221+0000 7fe6b5046980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:09Z"} 2024-03-04T19:23:51.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:09Z"} 2024-03-04T19:23:51.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.737 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.737 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:14.373+0000 7fd7c8d9f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:14.377+0000 7fd7c8d9f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:14Z"} 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:14Z"} 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:19.525+0000 7f1c843b1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:19.529+0000 7f1c843b1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:19Z"} 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:19Z"} 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:24.668+0000 7f8eb01f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:24.672+0000 7f8eb01f1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:24Z"} 2024-03-04T19:23:51.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:24Z"} 2024-03-04T19:23:51.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:29.840+0000 7f9643ccb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:29.840+0000 7f9643ccb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:29Z"} 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:29Z"} 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:34.992+0000 7f7725c1d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:34.992+0000 7f7725c1d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:34Z"} 2024-03-04T19:23:51.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:34Z"} 2024-03-04T19:23:51.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.742 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.742 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:40.136+0000 7f30d780b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:40.136+0000 7f30d780b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:40Z"} 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:40Z"} 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:45.288+0000 7f8bcd7bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:45.288+0000 7f8bcd7bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.424470_313.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.424470Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:45Z"} 2024-03-04T19:23:51.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.424470_313.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.424470Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:45Z"} 2024-03-04T19:23:51.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.424470_313.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.424470Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.424470_313.1'} 2024-03-04T19:23:51.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.424470_313.1 2024-03-04T19:23:51.744 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.744 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:50.440+0000 7fd2675c2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:50.440+0000 7fd2675c2980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:50Z"} 2024-03-04T19:23:51.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:50Z"} 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:55.576+0000 7feff27af980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:49:55.576+0000 7feff27af980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:55Z"} 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:55Z"} 2024-03-04T19:23:51.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:00.712+0000 7f8024d37980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:00.712+0000 7f8024d37980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:00Z"} 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:00Z"} 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:05.864+0000 7f4cb7acd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:05.868+0000 7f4cb7acd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:05Z"} 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:05Z"} 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.747 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:11.012+0000 7f789091b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:11.016+0000 7f789091b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:11Z"} 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:11Z"} 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:16.164+0000 7f9dbc73d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:16.164+0000 7f9dbc73d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:16Z"} 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:16Z"} 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:21.319+0000 7f41bcf95980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:21.319+0000 7f41bcf95980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:21Z"} 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:21Z"} 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:26.487+0000 7fb8a467d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:26.487+0000 7fb8a467d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:26Z"} 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:26Z"} 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:31.627+0000 7f731fca3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:31.627+0000 7f731fca3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:31Z"} 2024-03-04T19:23:51.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:31Z"} 2024-03-04T19:23:51.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709577927.265995_602.1'} 2024-03-04T19:23:51.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709577927.265995_602.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.752 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.752 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.752 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:51.752 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_index_log_trim 2024-03-04T19:23:51.752 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.753 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:51.753 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:51.753 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:51.753 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1346, in test_bucket_index_log_trim 2024-03-04T19:23:51.753 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2024-03-04T19:23:51.753 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1341, in make_test_bucket 2024-03-04T19:23:51.753 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:51.753 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:51.753 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:51.753 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:23:51.753 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:23:51.753 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=ppfahn-68 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-68/ 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-68/ 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-68/ 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.755 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.755 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.755 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.755 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:50:36 GMT 2024-03-04T19:23:51.755 INFO:tasks.rgw_multisite_tests:/ppfahn-68/ 2024-03-04T19:23:51.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.755 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:mwb6PUqwiT/z7X7aNOHMfkP5Fz4= 2024-03-04T19:23:51.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:mwb6PUqwiT/z7X7aNOHMfkP5Fz4=', 'Content-Length': '0'} 2024-03-04T19:23:51.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000c1fc30bf1fdb2bad-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-68/ 2024-03-04T19:23:51.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-68/ 2024-03-04T19:23:51.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-68/ 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:50:36 GMT 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests:/ppfahn-68/ 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.756 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM= 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM=', 'Content-Length': '0'} 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004598b08dc07cbf62-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-68/a 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-68/a 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-68/a 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.757 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:50:36 GMT 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:/ppfahn-68/a 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:d0taOZ22dCQWa/fDwGf8feucCwY= 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:d0taOZ22dCQWa/fDwGf8feucCwY='} 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005af2cf190a2b0c1c-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-68/ 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-68/ 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-68/ 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:50:36 GMT 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests:/ppfahn-68/ 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM= 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM=', 'Content-Length': '0'} 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a79932c357452e77-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-68/b 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-68/b 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-68/b 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.760 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:50:36 GMT 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:/ppfahn-68/b 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:vn7WgJ2H+/v0ikR51ydC8eDdNWg= 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:vn7WgJ2H+/v0ikR51ydC8eDdNWg='} 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002865ed5d55ede09d-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-68/ 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-68/ 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-68/ 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:50:36 GMT 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests:/ppfahn-68/ 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM= 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM=', 'Content-Length': '0'} 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a10c3ca9d09d611-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-68/c 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-68/c 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-68/c 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:50:36 GMT 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:/ppfahn-68/c 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:CAFmecd/Si+5RKT/OdcVfv4Gz8w= 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:CAFmecd/Si+5RKT/OdcVfv4Gz8w='} 2024-03-04T19:23:51.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001fe82e2188b91e35-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-68/ 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-68/ 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-68/ 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.764 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:50:36 GMT 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:/ppfahn-68/ 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM= 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:MSXvYTnDmiB1Rg8fZJrSl2YIoRM=', 'Content-Length': '0'} 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aaca58dd69f9c4e7-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-68/d 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-68/d 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-68/d 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:50:36 GMT 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:/ppfahn-68/d 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:gMIz82FDScZg4Ol5/AyHd4Pc5PY= 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:50:36 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:gMIz82FDScZg4Ol5/AyHd4Pc5PY='} 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d8546a99bb18633-0065e617fc-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:50:36.937+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:50:36.941+0000 7f5c2310a640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:50:36.941+0000 7f5c2a919640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709578236.710082_611.1","last_update":"2024-03-04T18:50:36.710082Z"}] 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:37.267+0000 7fda6d96a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:37.267+0000 7fda6d96a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:37Z"} 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:37Z"} 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:42.447+0000 7febbe2c3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:42.451+0000 7febbe2c3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:42Z"} 2024-03-04T19:23:51.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:42Z"} 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:47.591+0000 7f30b7545980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:47.595+0000 7f30b7545980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:47Z"} 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:47Z"} 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:52.799+0000 7fd97be9f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:52.799+0000 7fd97be9f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:52Z"} 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:52Z"} 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:57.939+0000 7fd625dfa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:50:57.939+0000 7fd625dfa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:57Z"} 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:57Z"} 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:03.083+0000 7f7e43c6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:03.083+0000 7f7e43c6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:03Z"} 2024-03-04T19:23:51.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:03Z"} 2024-03-04T19:23:51.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.772 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.772 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:08.227+0000 7fa635182980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:08.227+0000 7fa635182980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:08Z"} 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:08Z"} 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:13.375+0000 7fa39ba51980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:13.375+0000 7fa39ba51980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:13Z"} 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:13Z"} 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:18.518+0000 7fbbecd25980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:18.522+0000 7fbbecd25980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.480321_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.480321Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:18Z"} 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.480321_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.480321Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:18Z"} 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.480321_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.480321Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.480321_315.1'} 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.480321_315.1 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:23.682+0000 7f8721281980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:23.682+0000 7f8721281980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:23Z"} 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:23Z"} 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:51:26.484+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:28.818+0000 7fa3ac5fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:28.818+0000 7fa3ac5fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:28Z"} 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:28Z"} 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:33.966+0000 7fb404344980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:33.970+0000 7fb404344980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:33Z"} 2024-03-04T19:23:51.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:33Z"} 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:39.114+0000 7fa3cb345980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:39.118+0000 7fa3cb345980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:39Z"} 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:39Z"} 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:44.266+0000 7f959aecc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:44.266+0000 7f959aecc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:44Z"} 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:44Z"} 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:49.410+0000 7fe0b5afb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:49.410+0000 7fe0b5afb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:49Z"} 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:49Z"} 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.780 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:54.546+0000 7f07e0764980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:54.550+0000 7f07e0764980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:54Z"} 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:54Z"} 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:59.686+0000 7f41e4ada980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:51:59.690+0000 7f41e4ada980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:59Z"} 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:59Z"} 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.380518_307.1 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:04.830+0000 7ff1ee252980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:04.834+0000 7ff1ee252980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:04Z"} 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:04Z"} 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:09.974+0000 7f4b5c8c8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:09.974+0000 7f4b5c8c8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:09Z"} 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:09Z"} 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:15.110+0000 7f5b3c613980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:15.110+0000 7f5b3c613980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:15Z"} 2024-03-04T19:23:51.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:15Z"} 2024-03-04T19:23:51.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:20.261+0000 7f79eeede980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:20.261+0000 7f79eeede980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:20Z"} 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:20Z"} 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:25.409+0000 7f60c82d5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:25.413+0000 7f60c82d5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:25Z"} 2024-03-04T19:23:51.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:25Z"} 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:30.557+0000 7f49106fd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:30.557+0000 7f49106fd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.815737_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.815737Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:30Z"} 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.815737_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.815737Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:30Z"} 2024-03-04T19:23:51.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.815737_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.815737Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.815737_334.1'} 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575077.815737_334.1 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:35.689+0000 7fb5c7c6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:35.689+0000 7fb5c7c6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:35Z"} 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:35Z"} 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:40.837+0000 7fbf48b4d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:40.841+0000 7fbf48b4d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:40Z"} 2024-03-04T19:23:51.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:40Z"} 2024-03-04T19:23:51.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.790 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.790 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:45.985+0000 7f4d87e15980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:45.985+0000 7f4d87e15980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:45Z"} 2024-03-04T19:23:51.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:45Z"} 2024-03-04T19:23:51.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:51.129+0000 7f5cb7e61980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:51.129+0000 7f5cb7e61980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:51Z"} 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:51Z"} 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:56.281+0000 7f14a410b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:52:56.281+0000 7f14a410b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:56Z"} 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:56Z"} 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:01.441+0000 7fa7e9e8e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:01.441+0000 7fa7e9e8e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:01Z"} 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:01Z"} 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.793 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:06.613+0000 7f9bcc5aa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:06.613+0000 7f9bcc5aa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:06Z"} 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:06Z"} 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:53:10.034+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.794 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:11.797+0000 7f32e812a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:11.797+0000 7f32e812a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:11Z"} 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:11Z"} 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:16.933+0000 7f4da96b5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:16.933+0000 7f4da96b5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:16Z"} 2024-03-04T19:23:51.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:16Z"} 2024-03-04T19:23:51.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.796 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.796 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:22.144+0000 7f4863d9e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:22.144+0000 7f4863d9e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:22Z"} 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:22Z"} 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:27.316+0000 7f00f925d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:27.316+0000 7f00f925d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:27Z"} 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:27Z"} 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:32.464+0000 7f7c01e8a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:32.464+0000 7f7c01e8a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:32Z"} 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:32Z"} 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:37.612+0000 7fbfee031980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:37.616+0000 7fbfee031980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:37Z"} 2024-03-04T19:23:51.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:37Z"} 2024-03-04T19:23:51.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.800 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.800 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:42.748+0000 7fb6abe7a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:42.748+0000 7fb6abe7a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:42Z"} 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:42Z"} 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.388355_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.388355Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.388355_309.1'} 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.388355_309.1 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:47.884+0000 7f589dd2c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:47.888+0000 7f589dd2c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:47Z"} 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:47Z"} 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:53.032+0000 7f696c812980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:53.032+0000 7f696c812980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:53Z"} 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:53Z"} 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:58.184+0000 7f7257aec980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:53:58.184+0000 7f7257aec980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:58Z"} 2024-03-04T19:23:51.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:58Z"} 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:03.336+0000 7f3339372980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:03.336+0000 7f3339372980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:03Z"} 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:03Z"} 2024-03-04T19:23:51.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:08.472+0000 7fbc80ee7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:08.472+0000 7fbc80ee7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:08Z"} 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:08Z"} 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:13.604+0000 7f7f46070980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:13.604+0000 7f7f46070980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:13Z"} 2024-03-04T19:23:51.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:13Z"} 2024-03-04T19:23:51.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:18.751+0000 7f615268a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:18.755+0000 7f615268a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:18Z"} 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:18Z"} 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.808 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.808 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:23.899+0000 7f5c2169e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:23.899+0000 7f5c2169e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:23Z"} 2024-03-04T19:23:51.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:23Z"} 2024-03-04T19:23:51.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.315397_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.315397Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.315397_306.1'} 2024-03-04T19:23:51.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.315397_306.1 2024-03-04T19:23:51.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:29.087+0000 7f827d637980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:29.091+0000 7f827d637980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:29Z"} 2024-03-04T19:23:51.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:29Z"} 2024-03-04T19:23:51.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:34.231+0000 7fe0da3d7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:34.235+0000 7fe0da3d7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:34Z"} 2024-03-04T19:23:51.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:34Z"} 2024-03-04T19:23:51.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:39.375+0000 7fbcdc130980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:39.379+0000 7fbcdc130980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:39Z"} 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:39Z"} 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:44.539+0000 7fba83e16980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:44.539+0000 7fba83e16980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:44Z"} 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:44Z"} 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:49.683+0000 7f6006596980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:49.683+0000 7f6006596980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:49Z"} 2024-03-04T19:23:51.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:49Z"} 2024-03-04T19:23:51.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:54.835+0000 7f8b68baa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:54.839+0000 7f8b68baa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:54Z"} 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:54Z"} 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:59.983+0000 7f8576e3f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:54:59.983+0000 7f8576e3f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:59Z"} 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:59Z"} 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.495723_318.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.495723Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.495723_318.1'} 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.495723_318.1 2024-03-04T19:23:51.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:05.159+0000 7f23814dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:05.159+0000 7f23814dd980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:05Z"} 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.495723_318.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.495723Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:05Z"} 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.495723_318.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.495723Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.495723_318.1'} 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.495723_318.1 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:10.303+0000 7fb95eef0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:10.307+0000 7fb95eef0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:10Z"} 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:10Z"} 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:15.467+0000 7fdce1a22980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:15.467+0000 7fdce1a22980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:15Z"} 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:15Z"} 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:20.610+0000 7ff12c91d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:20.610+0000 7ff12c91d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:20Z"} 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:20Z"} 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:25.766+0000 7f5547c6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:25.766+0000 7f5547c6c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:25Z"} 2024-03-04T19:23:51.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:25Z"} 2024-03-04T19:23:51.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.820 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.820 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:30.906+0000 7f73cf444980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:30.906+0000 7f73cf444980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:30Z"} 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:30Z"} 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:36.066+0000 7f958812f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:36.066+0000 7f958812f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:36Z"} 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:36Z"} 2024-03-04T19:23:51.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.822 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.822 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:41.226+0000 7f5b9fff7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:41.226+0000 7f5b9fff7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:41Z"} 2024-03-04T19:23:51.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:41Z"} 2024-03-04T19:23:51.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578236.710082_611.1'} 2024-03-04T19:23:51.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578236.710082_611.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.823 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.823 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.823 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:51.823 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2024-03-04T19:23:51.823 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.824 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:51.824 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:51.824 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:51.824 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1400, in test_bucket_reshard_index_log_trim 2024-03-04T19:23:51.824 INFO:tasks.rgw_multisite_tests: test_bucket = make_test_bucket() 2024-03-04T19:23:51.824 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1395, in make_test_bucket 2024-03-04T19:23:51.824 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:51.824 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:51.824 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:51.824 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:23:51.824 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:23:51.824 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.824 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=ppfahn-69 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-69/ 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-69/ 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-69/ 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.825 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:55:46 GMT 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:/ppfahn-69/ 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:8WQKpOZYRNx3NiC8PqYWg6cvHMk= 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:8WQKpOZYRNx3NiC8PqYWg6cvHMk=', 'Content-Length': '0'} 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e2cbec3a68d7b188-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-69/ 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-69/ 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-69/ 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:55:46 GMT 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests:/ppfahn-69/ 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw= 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw=', 'Content-Length': '0'} 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038f809046450a688-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-69/a 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-69/a 2024-03-04T19:23:51.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-69/a 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:55:46 GMT 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:/ppfahn-69/a 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.828 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:9bcaI/Chx84lFhFBIk4WB6QGQ+U= 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:9bcaI/Chx84lFhFBIk4WB6QGQ+U='} 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000364316d504145386-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-69/ 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-69/ 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-69/ 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.829 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:55:46 GMT 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:/ppfahn-69/ 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw= 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw=', 'Content-Length': '0'} 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f91bcdd13c80db0-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-69/b 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-69/b 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-69/b 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:55:46 GMT 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:/ppfahn-69/b 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:ITJCtU7GlSkhhj4Fq/nClFfNlOk= 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ITJCtU7GlSkhhj4Fq/nClFfNlOk='} 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b63e02fce9771c41-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-69/ 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-69/ 2024-03-04T19:23:51.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-69/ 2024-03-04T19:23:51.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.832 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.832 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.832 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.832 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:55:46 GMT 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:/ppfahn-69/ 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw= 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw=', 'Content-Length': '0'} 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a0e7b32313b32f2-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-69/c 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-69/c 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-69/c 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:55:46 GMT 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:/ppfahn-69/c 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:ptd7ofyGw0FFbHOXwMjSH9j9G+M= 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:ptd7ofyGw0FFbHOXwMjSH9j9G+M='} 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c4e53722359d069-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-69/ 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-69/ 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-69/ 2024-03-04T19:23:51.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:55:46 GMT 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests:/ppfahn-69/ 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw= 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:eAjtY4H+6M8xVWyL6l2w8YXHRuw=', 'Content-Length': '0'} 2024-03-04T19:23:51.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f97bffdc91c99d47-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-69/d 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-69/d 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-69/d 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:23:51.836 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:55:46 GMT 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:/ppfahn-69/d 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:y1rK82aw6Y8Dm+aRJNhUZZxmi/0= 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 18:55:46 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:y1rK82aw6Y8Dm+aRJNhUZZxmi/0='} 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034a6e775a0b0c1c4-0065e61932-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 18:55:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709578546.310901_628.1","last_update":"2024-03-04T18:55:46.310901Z"}] 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:46.830+0000 7f88504aa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:46.834+0000 7f88504aa980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:46Z"} 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:46Z"} 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.838 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:55:48.870+0000 7f5c178f3640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi136.stdout: INFO: 2024-03-04T18:55:48.870+0000 7f5c22108640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:51.982+0000 7fe81348a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:51.982+0000 7fe81348a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:51Z"} 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:51Z"} 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.840 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:57.134+0000 7f8b4b5bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:55:57.134+0000 7f8b4b5bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:57Z"} 2024-03-04T19:23:51.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:57Z"} 2024-03-04T19:23:51.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T19:23:51.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.380518_307.1 2024-03-04T19:23:51.840 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:02.278+0000 7f3173b7a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:02.278+0000 7f3173b7a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:02Z"} 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:02Z"} 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:07.430+0000 7fcb1d0b7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:07.434+0000 7fcb1d0b7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:07Z"} 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:07Z"} 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:12.570+0000 7fb4b5e60980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:12.570+0000 7fb4b5e60980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:12Z"} 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:12Z"} 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:17.710+0000 7ff5ba602980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:17.710+0000 7ff5ba602980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:17Z"} 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:17Z"} 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:22.845+0000 7f0fd1af8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:22.849+0000 7f0fd1af8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:22Z"} 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:22Z"} 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:56:26.526+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:28.005+0000 7f15ce02e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:28.009+0000 7f15ce02e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:28Z"} 2024-03-04T19:23:51.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:28Z"} 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:33.141+0000 7fc91f107980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:33.141+0000 7fc91f107980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:33Z"} 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:33Z"} 2024-03-04T19:23:51.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:38.301+0000 7f13588b3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:38.301+0000 7f13588b3980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:38Z"} 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:38Z"} 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:43.449+0000 7f48ea65f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:43.449+0000 7f48ea65f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:43Z"} 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:43Z"} 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.848 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:48.589+0000 7f05345b0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:48.589+0000 7f05345b0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:48Z"} 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:48Z"} 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:53.749+0000 7f1c9fcdb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:53.749+0000 7f1c9fcdb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:53Z"} 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:53Z"} 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:58.897+0000 7f663f958980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:56:58.897+0000 7f663f958980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:58Z"} 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:58Z"} 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:04.033+0000 7f0407a04980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:04.037+0000 7f0407a04980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:04Z"} 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:04Z"} 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:09.185+0000 7f2fd0c87980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:09.185+0000 7f2fd0c87980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:09Z"} 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:09Z"} 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:14.341+0000 7fec65906980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:14.341+0000 7fec65906980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:14Z"} 2024-03-04T19:23:51.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:14Z"} 2024-03-04T19:23:51.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.854 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.854 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:19.489+0000 7fcb9c778980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:19.493+0000 7fcb9c778980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:19Z"} 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:19Z"} 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:24.640+0000 7f8dc8b32980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:24.640+0000 7f8dc8b32980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:24Z"} 2024-03-04T19:23:51.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:24Z"} 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:29.788+0000 7f56f4427980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:29.792+0000 7f56f4427980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:29Z"} 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:29Z"} 2024-03-04T19:23:51.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:34.948+0000 7fb8e7f91980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:34.948+0000 7fb8e7f91980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:34Z"} 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:34Z"} 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:40.080+0000 7f578cf3a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:40.080+0000 7f578cf3a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:40Z"} 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:40Z"} 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.860 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:45.236+0000 7f7cecada980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:45.236+0000 7f7cecada980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:45Z"} 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:45Z"} 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.315397_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.315397Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.315397_306.1'} 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.315397_306.1 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:50.412+0000 7f5eb344a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:50.416+0000 7f5eb344a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:50Z"} 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:50Z"} 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:55.576+0000 7f712a9f8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:57:55.580+0000 7f712a9f8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:55Z"} 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:55Z"} 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:00.724+0000 7ff37ebba980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:00.724+0000 7ff37ebba980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:00Z"} 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:00Z"} 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:05.876+0000 7f9c5bf9a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:05.880+0000 7f9c5bf9a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:05Z"} 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:05Z"} 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T18:58:10.099+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:11.020+0000 7f150462d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:11.020+0000 7f150462d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:11Z"} 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:11Z"} 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575076.582500_321.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:56.582500Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575076.582500_321.1'} 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575076.582500_321.1 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:16.156+0000 7f4d9ef9c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:16.156+0000 7f4d9ef9c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:16Z"} 2024-03-04T19:23:51.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:16Z"} 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575076.582500_321.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:56.582500Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575076.582500_321.1'} 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575076.582500_321.1 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:21.311+0000 7fb223213980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:21.311+0000 7fb223213980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:21Z"} 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:21Z"} 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:26.471+0000 7f070b3a1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:26.471+0000 7f070b3a1980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:26Z"} 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:26Z"} 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:31.655+0000 7f7a05446980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:31.659+0000 7f7a05446980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:31Z"} 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:31Z"} 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.869 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:36.807+0000 7fdde78e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:36.811+0000 7fdde78e6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:36Z"} 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:36Z"} 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:41.959+0000 7f8583c38980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:41.959+0000 7f8583c38980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:41Z"} 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:41Z"} 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.380518_307.1 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:47.091+0000 7fed6ed36980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:47.095+0000 7fed6ed36980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:47Z"} 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:47Z"} 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:52.255+0000 7fb4c7b4c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:52.255+0000 7fb4c7b4c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:52Z"} 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:52Z"} 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:57.407+0000 7f7da822b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:58:57.407+0000 7f7da822b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:57Z"} 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575076.582500_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:56.582500Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:57Z"} 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575076.582500_321.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:56.582500Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575076.582500_321.1'} 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575076.582500_321.1 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:02.547+0000 7f1851dad980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:02.547+0000 7f1851dad980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:02Z"} 2024-03-04T19:23:51.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:02Z"} 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:07.683+0000 7f93aed9e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:07.683+0000 7f93aed9e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:07Z"} 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:07Z"} 2024-03-04T19:23:51.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:12.838+0000 7fcd52f5f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:12.842+0000 7fcd52f5f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:12Z"} 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:12Z"} 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:17.978+0000 7f2d2790d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:17.982+0000 7f2d2790d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:17Z"} 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:17Z"} 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:23.134+0000 7f041d10c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:23.134+0000 7f041d10c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:23Z"} 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:23Z"} 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:28.286+0000 7f8ec0c4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:28.286+0000 7f8ec0c4a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:28Z"} 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:28Z"} 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.879 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:33.418+0000 7fdd6a625980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:33.422+0000 7fdd6a625980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:33Z"} 2024-03-04T19:23:51.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:33Z"} 2024-03-04T19:23:51.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.880 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.880 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:38.550+0000 7fc4422d4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:38.554+0000 7fc4422d4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:38Z"} 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:38Z"} 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:43.702+0000 7f87d0f21980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:43.702+0000 7f87d0f21980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:43Z"} 2024-03-04T19:23:51.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:43Z"} 2024-03-04T19:23:51.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.882 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.882 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:48.846+0000 7f86843bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:48.850+0000 7f86843bc980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.120434_389.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.120434Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:48Z"} 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.120434_389.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.120434Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:48Z"} 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.120434_389.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.120434Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575694.120434_389.1'} 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575694.120434_389.1 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:53.982+0000 7ff660139980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:53.982+0000 7ff660139980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:53Z"} 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:53Z"} 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:59.121+0000 7f45c4da6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T18:59:59.121+0000 7f45c4da6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:59Z"} 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:59Z"} 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:04.269+0000 7f022ffff980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:04.273+0000 7f022ffff980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:04Z"} 2024-03-04T19:23:51.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:04Z"} 2024-03-04T19:23:51.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:09.425+0000 7fdce3dd0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:09.425+0000 7fdce3dd0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:09Z"} 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:09Z"} 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:14.653+0000 7fb6d6e17980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:14.657+0000 7fb6d6e17980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:14Z"} 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:14Z"} 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:19.801+0000 7f44b3c7c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:19.801+0000 7f44b3c7c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.480321_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.480321Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:19Z"} 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.480321_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.480321Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:19Z"} 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.480321_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.480321Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.480321_315.1'} 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.480321_315.1 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:24.945+0000 7eff047b6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:24.945+0000 7eff047b6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:24Z"} 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:24Z"} 2024-03-04T19:23:51.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.890 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.890 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:30.085+0000 7fbb58ff8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:30.085+0000 7fbb58ff8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:30Z"} 2024-03-04T19:23:51.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:30Z"} 2024-03-04T19:23:51.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:35.233+0000 7fd7f92c0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:35.233+0000 7fd7f92c0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:35Z"} 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:35Z"} 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:40.369+0000 7fe9861c0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:40.369+0000 7fe9861c0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:40Z"} 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.310269_305.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.310269Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:40Z"} 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.310269_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.310269Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.310269_305.1'} 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.310269_305.1 2024-03-04T19:23:51.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:45.505+0000 7fbb4f8fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:45.505+0000 7fbb4f8fe980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.393343_310.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.393343Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:45Z"} 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.393343_310.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.393343Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:45Z"} 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.393343_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.393343Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.393343_310.1'} 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.393343_310.1 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:50.648+0000 7f139e8d0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:50.652+0000 7f139e8d0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:50Z"} 2024-03-04T19:23:51.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.292440_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.292440Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:50Z"} 2024-03-04T19:23:51.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.292440_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.292440Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578546.310901_628.1'} 2024-03-04T19:23:51.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.292440_304.1'} 2024-03-04T19:23:51.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578546.310901_628.1 target=1_1709575075.292440_304.1 2024-03-04T19:23:51.894 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.894 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.894 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:23:51.895 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_incremental 2024-03-04T19:23:51.895 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.895 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:23:51.895 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:23:51.895 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:23:51.895 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 1477, in test_bucket_reshard_incremental 2024-03-04T19:23:51.895 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:23:51.895 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:23:51.895 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:23:51.896 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:23:51.896 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:23:51.896 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.896 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:23:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/ppfahn-70/ 2024-03-04T19:23:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/ppfahn-70/ 2024-03-04T19:23:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:23:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /ppfahn-70/ 2024-03-04T19:23:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:23:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:23:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi136.front.sepia.ceph.com:8000 2024-03-04T19:23:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:23:51.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 19:00:55 GMT 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests:/ppfahn-70/ 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests:AWS HBI3JS1W1675M40GMG1A:BRTDvVQKsyjyU9Ul61r6hN4/kKM= 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 19:00:55 GMT', 'Authorization': 'AWS HBI3JS1W1675M40GMG1A:BRTDvVQKsyjyU9Ul61r6hN4/kKM=', 'Content-Length': '0'} 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ff544f1af2dd86d7-0065e61a67-5376-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 19:00:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=ppfahn-70 2024-03-04T19:23:51.897 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stderr: INFO: No --period given, using current period=f6a09d88-afdd-40f4-a159-dda085c373df 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi136.stdout: INFO: [{"marker":"1_1709573646.723863_31.1","last_update":"2024-03-04T17:34:06.723863Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1709573789.409125_113.1","last_update":"2024-03-04T17:36:29.409125Z"},{"marker":"1_1709578855.731020_637.1","last_update":"2024-03-04T19:00:55.731020Z"}] 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:56.268+0000 7f51c66b4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:00:56.268+0000 7f51c66b4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:56Z"} 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:56Z"} 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:01.468+0000 7fddf282e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:01.468+0000 7fddf282e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:01Z"} 2024-03-04T19:23:51.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:01Z"} 2024-03-04T19:23:51.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.900 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.900 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:06.616+0000 7fbeaa422980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:06.616+0000 7fbeaa422980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:06Z"} 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:06Z"} 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:11.776+0000 7ff7169de980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:11.776+0000 7ff7169de980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:11Z"} 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:11Z"} 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:16.904+0000 7fa3294c6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:16.908+0000 7fa3294c6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:16Z"} 2024-03-04T19:23:51.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:16Z"} 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:22.056+0000 7f8693ace980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:22.056+0000 7f8693ace980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:22Z"} 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:22Z"} 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:27.204+0000 7f8924b55980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:27.204+0000 7f8924b55980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:27Z"} 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:27Z"} 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:32.348+0000 7fac3bce9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:32.352+0000 7fac3bce9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.424470_313.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.424470Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:32Z"} 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.424470_313.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.424470Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:32Z"} 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.424470_313.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.424470Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.424470_313.1'} 2024-03-04T19:23:51.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.424470_313.1 2024-03-04T19:23:51.906 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.906 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:37.496+0000 7fcc54234980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:37.496+0000 7fcc54234980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:37Z"} 2024-03-04T19:23:51.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:37Z"} 2024-03-04T19:23:51.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:42.631+0000 7f79c8a17980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:42.631+0000 7f79c8a17980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:42Z"} 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.380518_307.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.380518Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:42Z"} 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.380518_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.380518Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.380518_307.1'} 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.380518_307.1 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T19:01:46.451+0000 7fae33182640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:47.791+0000 7f216e20c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:47.795+0000 7f216e20c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:47Z"} 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:47Z"} 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:52.943+0000 7f025bf6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:52.943+0000 7f025bf6f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:52Z"} 2024-03-04T19:23:51.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:52Z"} 2024-03-04T19:23:51.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:58.095+0000 7f00ef21f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:01:58.099+0000 7f00ef21f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:58Z"} 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:58Z"} 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:03.235+0000 7faa1f9d9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:03.235+0000 7faa1f9d9980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:03Z"} 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:03Z"} 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:08.371+0000 7f3da1236980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:08.375+0000 7f3da1236980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:08Z"} 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:08Z"} 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:13.523+0000 7f76e66f8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:13.523+0000 7f76e66f8980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:13Z"} 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:13Z"} 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:18.659+0000 7f571409d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:18.663+0000 7f571409d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:18Z"} 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:18Z"} 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:23.815+0000 7f4700f0e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:23.815+0000 7f4700f0e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:23Z"} 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:23Z"} 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:28.955+0000 7f1de5c93980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:28.959+0000 7f1de5c93980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:28Z"} 2024-03-04T19:23:51.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:28Z"} 2024-03-04T19:23:51.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.916 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.916 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:34.110+0000 7f7f7769b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:34.110+0000 7f7f7769b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:34Z"} 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:34Z"} 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:39.262+0000 7fa41f153980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:39.262+0000 7fa41f153980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:39Z"} 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:39Z"} 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:44.422+0000 7f10876eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:44.422+0000 7f10876eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:44Z"} 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:44Z"} 2024-03-04T19:23:51.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:49.558+0000 7f470925b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:49.558+0000 7f470925b980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:49Z"} 2024-03-04T19:23:51.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:49Z"} 2024-03-04T19:23:51.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.920 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.920 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:54.714+0000 7f2345c46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:54.718+0000 7f2345c46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:54Z"} 2024-03-04T19:23:51.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:54Z"} 2024-03-04T19:23:51.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:59.870+0000 7f9e0330f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:02:59.870+0000 7f9e0330f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:59Z"} 2024-03-04T19:23:51.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:59Z"} 2024-03-04T19:23:51.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:05.014+0000 7f05ff090980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:05.018+0000 7f05ff090980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:05Z"} 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:05Z"} 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:10.154+0000 7fc70aa46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:10.158+0000 7fc70aa46980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:10Z"} 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:10Z"} 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:15.346+0000 7fc6b7856980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:15.350+0000 7fc6b7856980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:15Z"} 2024-03-04T19:23:51.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:15Z"} 2024-03-04T19:23:51.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.925 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.925 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:20.502+0000 7f5e5c55a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:20.502+0000 7f5e5c55a980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:20Z"} 2024-03-04T19:23:51.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:20Z"} 2024-03-04T19:23:51.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:25.666+0000 7f135a33c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:25.670+0000 7f135a33c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:25Z"} 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:25Z"} 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi136.stdout: INFO: 2024-03-04T19:03:28.904+0000 7fae32180640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:23:51.926 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:30.817+0000 7fa56d028980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:30.817+0000 7fa56d028980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:30Z"} 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:30Z"} 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:35.961+0000 7fdde5fba980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:35.961+0000 7fdde5fba980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:35Z"} 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.388355_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.388355Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:35Z"} 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.388355_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.388355Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.388355_309.1'} 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.388355_309.1 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:41.109+0000 7efce25eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:41.109+0000 7efce25eb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:41Z"} 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:41Z"} 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:46.253+0000 7f62a49c7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:46.257+0000 7f62a49c7980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:46Z"} 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:46Z"} 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:51.397+0000 7f6fe0661980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:51.401+0000 7f6fe0661980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:51Z"} 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:51Z"} 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:56.513+0000 7fa4f67fc640 0 ms_deliver_dispatch: unhandled message 0x7fa60004de00 osd_map(42..42 src has 1..42) v4 from mon.0 v2:172.21.15.153:3300/0 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:56.557+0000 7fa61337d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:03:56.557+0000 7fa61337d980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:56Z"} 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:56Z"} 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:01.713+0000 7fe4cbfc6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:01.717+0000 7fe4cbfc6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.735016_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.735016Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:01Z"} 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.735016_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.735016Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:01Z"} 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.735016_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.735016Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.735016_329.1'} 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575077.735016_329.1 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:06.861+0000 7f7e191b5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:06.861+0000 7f7e191b5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:06Z"} 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:06Z"} 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:11.997+0000 7fae1261f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:12.001+0000 7fae1261f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:12Z"} 2024-03-04T19:23:51.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:12Z"} 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:17.153+0000 7f19bbe8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:17.153+0000 7f19bbe8f980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:17Z"} 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:17Z"} 2024-03-04T19:23:51.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:22.304+0000 7f5368f54980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:22.304+0000 7f5368f54980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:22Z"} 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:22Z"} 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:27.452+0000 7f14053a4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:27.452+0000 7f14053a4980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:27Z"} 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.315397_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.315397Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:27Z"} 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.315397_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.315397Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.315397_306.1'} 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.315397_306.1 2024-03-04T19:23:51.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:32.600+0000 7f4175f12980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:32.600+0000 7f4175f12980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:32Z"} 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:32Z"} 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:37.748+0000 7f6d45928980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:37.748+0000 7f6d45928980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:37Z"} 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:37Z"} 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:42.908+0000 7f23fb946980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:42.908+0000 7f23fb946980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:42Z"} 2024-03-04T19:23:51.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:42Z"} 2024-03-04T19:23:51.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.940 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.940 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:48.068+0000 7f66ef762980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:48.068+0000 7f66ef762980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:48Z"} 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:48Z"} 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:53.208+0000 7f2590434980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:53.208+0000 7f2590434980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:53Z"} 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:53Z"} 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:58.424+0000 7f2f2d4e5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:04:58.424+0000 7f2f2d4e5980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:58Z"} 2024-03-04T19:23:51.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:58Z"} 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:03.564+0000 7f3e2cd4e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:03.564+0000 7f3e2cd4e980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:03Z"} 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:03Z"} 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:08.720+0000 7fad00536980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:08.720+0000 7fad00536980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:08Z"} 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:08Z"} 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:13.871+0000 7f19f45d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:13.871+0000 7f19f45d6980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:13Z"} 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:13Z"} 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:19.023+0000 7fabb0cc0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:19.027+0000 7fabb0cc0980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:19Z"} 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:19Z"} 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:24.163+0000 7f0f1b155980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:24.163+0000 7f0f1b155980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:24Z"} 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:24Z"} 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:29.311+0000 7f25c5493980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:29.315+0000 7f25c5493980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:29Z"} 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:29Z"} 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:34.459+0000 7fb6341fb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:34.463+0000 7fb6341fb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.650819_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.650819Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:34Z"} 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575077.650819_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:57.650819Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:34Z"} 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575077.650819_323.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:57.650819Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575077.650819_323.1'} 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575077.650819_323.1 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:39.623+0000 7fd8b5012980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:39.623+0000 7fd8b5012980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:39Z"} 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:39Z"} 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:44.771+0000 7f28e5cef980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:44.771+0000 7f28e5cef980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:44Z"} 2024-03-04T19:23:51.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:44Z"} 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:49.907+0000 7f0e1870c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:49.907+0000 7f0e1870c980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.386285_308.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.386285Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:49Z"} 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.386285_308.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.386285Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:49Z"} 2024-03-04T19:23:51.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.386285_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.386285Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.386285_308.1'} 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.386285_308.1 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:55.059+0000 7f5a60afb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:05:55.059+0000 7f5a60afb980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:55Z"} 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.271482_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.271482Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:55Z"} 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.271482_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.271482Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.271482_301.1'} 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.271482_301.1 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:00.198+0000 7f3a32952980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stderr: INFO: 2024-03-04T19:06:00.198+0000 7f3a32952980 20 RGW-SYNC:meta: read sync status 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi153.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:00Z"} 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f6a09d88-afdd-40f4-a159-dda085c373df","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709573646.723863_31.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:34:06.723863Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709573789.409125_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:36:29.409125Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575075.288119_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T17:57:55.288119Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:00Z"} 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709573646.723863_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:34:06.723863Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709573789.409125_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:36:29.409125Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575075.288119_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T17:57:55.288119Z', 'realm_epoch': 2}}] 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709578855.731020_637.1'} 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709573646.723863_31.1', 1: '', 2: '1_1709573789.409125_113.1', 3: '1_1709575075.288119_303.1'} 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1709578855.731020_637.1 target=1_1709575075.288119_303.1 2024-03-04T19:23:51.953 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:23:51.954 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.954 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:23:51.954 INFO:tasks.rgw_multisite_tests:Ran 63 tests in 6465.030s 2024-03-04T19:23:51.954 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:23:51.954 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=15, errors=19, failures=14) 2024-03-04T19:23:51.954 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-03-04T19:23:52.519 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=704cc80372d14a69bcde036f98eb956b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-03-04T19:23:52.520 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2024-03-04T19:23:52.530 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2024-03-04T19:23:52.577 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-03-04T19:23:52.626 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-03-04T19:23:52.626 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:23:52.627 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:23:52.628 ERROR:teuthology.run_tasks:Manager failed: rgw Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" 2024-03-04T19:23:52.636 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-04T19:23:52.680 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-03-04T19:23:52.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:23:52.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:23:55.740 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:24:04.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:24:04.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:24:07.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:24:19.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:24:19.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:24:22.876 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:24:25.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:24:25.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:24:25.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:24:25.951 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:24:25.951 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi153 2024-03-04T19:24:25.965 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-04T19:24:25.965 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-04T19:24:25.966 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2024-03-04T19:24:25.966 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:24:25.969 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:24:25.969 ERROR:teuthology.misc:Saw exception from c2.osd.0 Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi153 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi153 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0' 2024-03-04T19:24:25.984 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2024-03-04T19:24:25.984 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:24:25.985 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:24:25.985 ERROR:teuthology.misc:Saw exception from c2.osd.1 Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi153 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi153 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1' 2024-03-04T19:24:25.998 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2024-03-04T19:24:25.998 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:24:25.999 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:24:25.999 ERROR:teuthology.misc:Saw exception from c2.osd.2 Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi153 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi153 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2' 2024-03-04T19:24:26.013 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-04T19:24:26.013 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2024-03-04T19:24:26.014 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:24:26.014 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:24:26.014 ERROR:teuthology.misc:Saw exception from c2.mgr.x Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi153 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi153 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi153 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x' 2024-03-04T19:24:26.031 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-04T19:24:26.031 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2024-03-04T19:24:26.031 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:24:26.032 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:24:26.032 ERROR:teuthology.misc:Saw exception from c2.mon.a Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi153 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi153 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi153 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi153 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a' 2024-03-04T19:24:26.052 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-04T19:24:26.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:24:26.053 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:24:29.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:24:38.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:24:38.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:24:41.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:24:53.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:24:53.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:24:56.252 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:24:59.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-04T19:24:59.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:24:59.324 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-04T19:24:59.324 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:24:59.324 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi153 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1144, in cluster if first_in_ceph_log('\[ERR\]|\[WRN\]|\[SEC\]', File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1141, in first_in_ceph_log stdout = mon0_remote.sh(args) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi153 2024-03-04T19:24:59.345 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-04T19:24:59.354 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-03-04T19:24:59.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:24:59.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:25:02.428 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:25:11.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:25:11.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:25:14.492 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:25:26.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:25:26.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:25:29.564 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:25:32.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:25:32.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:25:32.636 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:25:32.636 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:25:32.636 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:25:32.653 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-04T19:25:32.653 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-04T19:25:32.654 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2024-03-04T19:25:32.654 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:25:32.657 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:25:32.658 ERROR:teuthology.misc:Saw exception from c1.osd.0 Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0' 2024-03-04T19:25:32.674 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2024-03-04T19:25:32.674 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:25:32.675 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:25:32.675 ERROR:teuthology.misc:Saw exception from c1.osd.1 Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1' 2024-03-04T19:25:32.691 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2024-03-04T19:25:32.691 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:25:32.692 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:25:32.692 ERROR:teuthology.misc:Saw exception from c1.osd.2 Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2' 2024-03-04T19:25:32.709 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-04T19:25:32.709 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2024-03-04T19:25:32.709 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:25:32.709 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:25:32.709 ERROR:teuthology.misc:Saw exception from c1.mgr.x Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x' 2024-03-04T19:25:32.728 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-04T19:25:32.729 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2024-03-04T19:25:32.729 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:25:32.729 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:25:32.729 ERROR:teuthology.misc:Saw exception from c1.mon.a Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a' 2024-03-04T19:25:32.752 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-04T19:25:32.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:25:32.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:25:35.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:25:44.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:25:44.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:25:47.868 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:25:59.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:25:59.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:26:02.940 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:26:05.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:26:05.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:26:06.012 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:26:06.012 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:26:06.012 INFO:tasks.ceph:Archiving crash dumps... 2024-03-04T19:26:06.014 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/crash to /home/teuthworker/archive/suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi/7581342/remote/smithi136/crash 2024-03-04T19:26:06.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:26:06.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:26:09.084 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:26:18.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:26:18.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:26:21.148 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:26:33.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:26:33.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:26:36.220 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:26:39.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:26:39.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:26:39.292 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:26:39.292 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:26:39.293 INFO:tasks.ceph:Compressing logs... 2024-03-04T19:26:39.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:26:39.294 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:26:42.364 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:26:51.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:26:51.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:26:54.428 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:27:06.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:27:06.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:27:09.500 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:27:12.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:27:12.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:27:12.572 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:27:12.572 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:27:12.573 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/ceph.py", line 264, in ceph_log ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:27:12.599 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-04T19:27:12.614 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" 2024-03-04T19:27:12.630 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-04T19:27:12.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:27:12.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:27:15.708 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:27:24.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:27:24.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:27:27.772 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:27:39.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:27:39.786 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:27:42.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:27:45.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:27:45.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:27:45.916 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:27:45.916 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:27:45.962 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:27:45.979 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-04T19:27:45.988 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-04T19:27:45.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:27:45.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:27:49.052 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:27:58.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:27:58.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:28:01.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:28:13.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:28:13.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:28:16.189 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:28:19.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:28:19.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:28:19.260 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:28:19.260 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:28:19.261 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:28:19.277 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-04T19:28:19.288 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-04T19:28:19.289 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-04T19:28:19.336 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-04T19:28:19.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-04T19:28:19.435 INFO:teuthology.task.internal:Duration was 7545.863346 seconds 2024-03-04T19:28:19.435 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-04T19:28:19.487 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-04T19:28:19.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:28:19.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:28:22.556 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:28:31.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:28:31.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:28:34.620 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:28:46.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:28:46.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:28:49.724 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:28:52.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:28:52.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:28:52.797 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:28:52.797 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:28:52.799 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:28:52.817 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-04T19:28:52.832 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-04T19:28:52.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:28:52.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:28:55.904 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:29:04.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:29:04.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:29:07.996 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:29:20.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:29:20.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:29:23.068 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:29:26.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:29:26.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:29:26.140 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:29:26.140 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:29:26.140 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:29:26.156 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-04T19:29:26.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:29:26.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:29:29.244 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:29:38.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:29:38.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:29:41.308 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:29:53.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:29:53.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:29:56.380 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:29:59.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:29:59.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:29:59.452 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:29:59.452 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:29:59.453 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:29:59.471 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-04T19:29:59.483 INFO:teuthology.task.internal:Transferring archived files... 2024-03-04T19:29:59.483 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi/7581342/remote/smithi136 2024-03-04T19:29:59.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:29:59.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:30:02.556 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:30:11.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:30:11.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:30:14.621 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:30:26.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:30:26.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:30:29.692 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:30:32.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:30:32.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:30:32.764 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:30:32.764 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:30:32.764 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:30:32.778 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-04T19:30:32.790 INFO:teuthology.task.internal:Not uploading archives. 2024-03-04T19:30:32.790 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-04T19:30:32.829 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-04T19:30:32.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:30:32.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:30:35.900 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:30:44.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:30:44.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:30:47.964 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:30:59.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:30:59.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:31:03.036 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:31:06.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-04T19:31:06.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:31:06.108 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-04T19:31:06.108 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:31:06.109 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_pritha-srivastava_ceph_3aec390336fdfdc8f927818b718cfb0361bd47c8/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi136 2024-03-04T19:31:06.123 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-04T19:31:06.135 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-04T19:31:06.176 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: ConnectionLostError: SSH connection to smithi136 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" 2024-03-04T19:31:06.176 INFO:teuthology.nuke:Checking targets against current locks 2024-03-04T19:31:06.209 DEBUG:teuthology.nuke:shortname: smithi136 2024-03-04T19:31:06.210 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T19:31:06.239 DEBUG:teuthology.nuke:shortname: smithi153 2024-03-04T19:31:06.239 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T19:31:06.259 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi/7581342', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': False, 'locked_since': None, 'locked_by': None, 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM5+Lf8d9jkIUZDTAHuoCGKmJtNNK7KduqKFetIFOBuUOXs7F9wANcTWVBomJoQHbfTA4oYoHd9PFqtZins8FCs='} 2024-03-04T19:31:06.260 ERROR:teuthology.nuke:Could not nuke {'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM5+Lf8d9jkIUZDTAHuoCGKmJtNNK7KduqKFetIFOBuUOXs7F9wANcTWVBomJoQHbfTA4oYoHd9PFqtZins8FCs='} Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/nuke/__init__.py", line 294, in nuke_one nuke_helper(ctx, should_unlock, keep_logs, should_reboot) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/nuke/__init__.py", line 318, in nuke_helper check_lock.check_lock(ctx, None, check_up=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/check_lock.py", line 28, in check_lock assert status['locked'], \ AssertionError: machine smithi153.front.sepia.ceph.com is not locked 2024-03-04T19:31:06.270 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/suriarte-2024-03-04_17:13:34-rgw-wip-rgw-d4n-distro-default-smithi/7581342', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': False, 'locked_since': None, 'locked_by': None, 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAqHjmXINRGS++vEJkLSuQVoCDLD7Q/OTh0RAS4ORRa34gLI/XifPQGnY9NX/bUvqJ8L9VxxHFw5XRlWZkskD6M='} 2024-03-04T19:31:06.271 ERROR:teuthology.nuke:Could not nuke {'smithi136.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAqHjmXINRGS++vEJkLSuQVoCDLD7Q/OTh0RAS4ORRa34gLI/XifPQGnY9NX/bUvqJ8L9VxxHFw5XRlWZkskD6M='} Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/nuke/__init__.py", line 294, in nuke_one nuke_helper(ctx, should_unlock, keep_logs, should_reboot) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/nuke/__init__.py", line 318, in nuke_helper check_lock.check_lock(ctx, None, check_up=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/check_lock.py", line 28, in check_lock assert status['locked'], \ AssertionError: machine smithi136.front.sepia.ceph.com is not locked 2024-03-04T19:31:06.277 ERROR:teuthology.nuke:Could not nuke the following targets: targets: smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAqHjmXINRGS++vEJkLSuQVoCDLD7Q/OTh0RAS4ORRa34gLI/XifPQGnY9NX/bUvqJ8L9VxxHFw5XRlWZkskD6M= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM5+Lf8d9jkIUZDTAHuoCGKmJtNNK7KduqKFetIFOBuUOXs7F9wANcTWVBomJoQHbfTA4oYoHd9PFqtZins8FCs= 2024-03-04T19:31:06.285 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} duration: 7545.863345623016 failure_reason: rgw multisite test failures flavor: default owner: scheduled_suriarte@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=704cc80372d14a69bcde036f98eb956b status: fail success: false 2024-03-04T19:31:06.285 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T19:31:06.360 INFO:teuthology.run:FAIL